08-20-2003, 05:05 PM
|
|
Executive Editor
Join Date: Aug 2006
Posts: 29,160
|
|
Is Windows Mobile 2003 Really Faster?
I had the rare opportunity to run Spb Benchmark on a single iPAQ 3650 running three generations of Microsoft's Mobile operating systems (the Pocket PC 2003 image for the iPAQ 3650 came from Microsoft - it a special testing-only build that won't be released to the public). This gave me the metrics I needed to make this definitely statement: Windows Mobile 2003, the first Pocket PC operating system built atop Windows CE .NET 4.2, is the fastest all-around operating system we've seen so far on these little devices we enjoy using so much. I present to you a series of benchmarks that show you exactly how much faster it really is! The Pocket PC 2002 and 2003 were performed on the same device, while the Pocket PC 2000 tests were performed on a different device, but of the same model.
First, let's look at how the Windows Mobile 2003 iPAQ compares with the baseline index, which is based on a Pocket PC 2000 iPAQ 3650. Take a look:
� Overall, Windows Mobile 2003 is 15% faster
� The CPU performance of the StrongARM 206 Mhz is 25% faster under Windows Mobile 2003 than under Pocket PC 2000
� The file system is slower (0.3%), but only slightly so in the overall score (the new OS is much slower in some key tests you'll see later on in this article
� The graphics subsystem of Windows Mobile 2003 is vastly improved over previous versions - a whopping 40% faster.
� ActiveSync has the most dramatic improvements - an incredible 103% boost in speed over Pocket PC 2000.
Read on for the comparisons between Pocket PC 2000, 2002 and 2003 - I think you'll be surprised at how much of a speed drop we endured with Pocket PC 2002 in some tests, and what a dramatic difference there is in moving to the 2003 OS. Clearly, the resources required to move to the new Windows CE .Net 4.2 core were worth it, even if it meant that we didn't get that many new application-specific features in Windows Mobile 2003.
The screen shots below do not represent every possible test that Spb Benchmark is capable of performing, only the ones I felt were more relevant and showed the most dramatic performance differences. The indexes and averages are based on the entire suit of Spb Benchmark tests.
Figure 2: We lost speed with Pocket PC 2002, but gained it back (and then some) with Pocket PC 2003
Figure 3: Windows Mobile 2003 offers significant CPU enhancements for any ARM processor, without any special XScale tweaks. This is further proof to me that Microsoft made the right decision in not making special optimizations for the XScale processor, and instead focusing on improving the OS for all ARM-compatible processors.
Figure 4: Windows Mobile 2003 gives us greater graphics performance.
Figure 5: As you can see, ActiveSync performance actually got worse with Pocket PC 2002, but we see huge performance gains with Windows Mobile 2003. Testing on the Pocket PC 2002 and 2003 devices was performed using ActiveSync 3.7 on a USB 2.0 port.
Figure 6: The platform index shows us that while Windows Mobile 2003 doesn't offer enormous enhancements over Pocket PC 2000, it does over Pocket PC 2002, which is where most of us are coming from.
Figure 7: Unfortunately, the file system is where the performance losses in Windows Mobile 2003 are most noticeable - this test measures how long it takes the Pocket PC to enumerate (list) a directory with 2000 files. It's not as grim as it looks though - this test is ranked very low in the Spb Benchmark scores, because while the chart would visually indicate a huge difference, it's really the difference between Windows Mobile 2003 taking 0.06 seconds to enumerate 2000 files, and Pocket PC 2002 taking 0.015 seconds. Both are fast. ;-)
Figure 8: Internal database speeds were one of the few things that got faster with Pocket PC 2002, and that trend continued with 2003.
Figure 9: The BitBlt graphics test measures how quickly the Pocket PC can draw an image to the screen using GDI functions. Windows Mobile 2003 shows a tremendous speed boost over previous operating systems, which impacts normal graphics operations like displaying pictures, simple games like Solitaire, etc. Games use GAPI, which this test does not measure.
Figure 10: A Windows Mobile 2003 device is capable of opening a 240 KB Pocket Word document nearly 300% faster than a Pocket PC 2000 device.
Figure 11: Yes, that's right - things got slower with Windows Mobile 2003 when it comes to browsing the Web. Pocket Internet Explorer is still a little pokey when it comes to processing HTML files. One of the reasons that Pocket PC 2000 is much faster than Windows Mobile 2003 in this test is that it's a much simpler browser and supports less HTML functions than Windows Mobile 2003. I feel there's a lot of room for improvement with Pocket Internet Explorer - Windows Mobile 2003 didn't meet my expectations in this regard.
Figure 12: JPEG loading performance is likely improved under 2003.
Figure 13: When it comes to opening a folder with 2000 empty files using File Explorer, Windows Mobile 2003 offers dramatic gains over Pocket PC 2002.
Figure 14: Although you'd think that ZIP compression speed would be solely a function of the CPU, Windows Mobile 2003 gives a solid 128% speed boost over Pocket PC 2002. Cool. 8)
Figure 15: ActiveSync speeds saw the greatest boost - I wonder if this means they finally got rid of the USB over serial driver that hobbled previous ActiveSync speeds? The speed increase in moving data from the desktop to the Pocket PC is so dramatic (139% faster) that this is the first generation of Pocket PC where I don't think people need to have an external memory card reader to load music onto their device - it's finally fast enough to do directly from desktop to Pocket PC. Well done Microsoft!
Figure 16: Although not as dramatic as the speed difference above, the speeds at moving data from the Pocket PC to the desktop PC are also enhanced.
Figure 17: This is the overall index showing how the three iPAQ 3650's each compare to a Dell Axim X5 running Pocket PC 2002, and a new iPAQ 2215 running Windows Mobile 2003. Incredibly, even the blazing iPAQ 2215 still trails the original iPAQ 3650 slightly. What kind of nitro did Compaq put in that device? I find it somewhat sad that, over three years after the fist iPAQ came out, we can't completely trump it with brand new devices.
Figure 18: I continue to have my doubts about how good of a CPU the XScale processor is. The performance dominance of the XScale PXA255 version is obvious when compared to the XScale PXA250 (Dell Axim), but when you consider that the iPAQ 3650 with a score of 1251 is running at 206 Mhz, the score of 1784 for the 400 Mhz XScale PXA255 becomes much less impressive. I hope Intel's next-generation XScale processor offers more impressive performance. I wish I had the Windows Mobile 2003 upgrade for my Dell Axim X5 so I could see exactly how much difference Windows Mobile 2003 makes on a PXA250. Was the PXA250 being held back by the Pocket PC 2002 operating system, or is it the other way around?
Figure 19: As readers of my iPAQ 2215 review will know, the 2215 benchmarked strangely slow in some graphics tests. This chart shows how poorly the 2215 benchmarks against the iPAQ 3650, but it also shows how much the OS alone improves the graphics performance, making the difference in performance of the 2003 iPAQ 3650 (1399) and the iPAQ 2215 (567) even more aappalling- a massive 247% difference! iPAQ 2215 owners take heart - it's been speculated that the Spb Benchmark tests don't tap into the MediaQ chip, so these results shouldn't be taken as a true measure of your device performance. Hopefully future versions of Spb Benchmark will take advantage of the MediaQ chip.
Figure 20: Further reiteration of what a big difference the operating system makes when it comes to ActiveSync speeds. The iPAQ 2215 pulls ahead here (as it should), and the speed of the Axim running 2002 shows that the more modern hardware can account for as much as a 52% speed boost, even with the same OS (2002 Axim vs. 2002 iPAQ).
A big thanks to Spb Software House for making such a powerful tool for us reviewers to use!
|
| |
|
|
|
08-20-2003, 05:39 PM
|
|
Ponderer
Join Date: Jun 2002
Posts: 105
|
|
iPAQ 3650 (2003, 205 Mhz)
A buuhhhh?
Didn't relize they upgraded those models hehe...or maybe thats just a nice feature of spb benchmark utility :-p
Jake-
|
| |
|
|
|
08-20-2003, 05:53 PM
|
|
Moderator
Join Date: Aug 2006
Posts: 6,878
|
|
Quote:
|
Originally Posted by The Half-Ling
iPAQ 3650 (2003, 205 Mhz)
A buuhhhh?
Didn't relize they upgraded those models hehe...or maybe thats just a nice feature of spb benchmark utility
|
Read it again... :wink:
Quote:
|
Originally Posted by Jason Dunn
Pocket PC 2003 image for the iPAQ 3650 came from Microsoft - it a special testing-only build that won't be released to the public
|
Steve
__________________
"My eyes are rolling back in my head so far I can see my grey matter bubbling and frothing from reading this thread....bleh." JD
|
| |
|
|
|
08-20-2003, 05:56 PM
|
|
Intellectual
Join Date: Jul 2003
Posts: 155
|
|
Re: Is Windows Mobile 2003 Really Faster?
Quote:
|
Originally Posted by Jason Dunn
the Pocket PC 2003 image for the iPAQ 3650 came from Microsoft - it a special testing-only build that won't be released to the public
|
Too bad. I sure would enjoy upgrading my iPAQ 3630. :?
At least I have 2002 running on it. It may not be as fast, but I still like the look of the OS.
Thanks for the nice article.
|
| |
|
|
|
08-20-2003, 06:19 PM
|
|
Theorist
Join Date: Jul 2003
Posts: 258
|
|
Nice benchmarks, Jason. I would have liked to get some benchmarks on my e310 in comparison to the rest of the list...  Anyways, one O/T feature request for SPB Benchmark: The devices have the same colors chart to chart. This way, you can tell at a glance who came out on top, on bottom, etc. without having to read the captions every time. (Yes, I really am that lazy that I don't want to read the captions... :wink: ) Anyways, great benchmarks! 
EDIT: Hmmm... was it just me or did this thread disappear for a little while... :wink:
|
| |
|
|
|
08-20-2003, 07:34 PM
|
|
Sage
Join Date: Aug 2006
Posts: 779
|
|
PIE is my biggest disappointment in PPC 2003.
My iPAQ 3630 (PPC 2000) with Bluekite, a Supplynet cable, and a Sprint phone (PRE-1xrtt) seemed nearly as fast as my 5555 does now at home with WiFi and cable modem. And it was much, much faster than the 5555 is with a Supplynet cable and a Sprint 1xrtt phone. Of course Sprint's new method of disconnecting and reconnecting (supposedly transparently) between pages doesn't help matters.
|
| |
|
|
|
08-20-2003, 08:31 PM
|
|
Pontificator
Join Date: Aug 2006
Posts: 1,202
|
|
Great article Jason, thanks for taking all the time to compare these devices. This sure proves what all lot of has have been thinking about performance improvements with 2003.
I guess the one thing that seems to hit me in the face is how much of a fraud the 400 Mhz X-Scale CPU is. It seems we would all be cruising along in the left lane if Intel had just decided to continue developing/improving the original StrongArm CPU's rather than forcing a new technology down our throats.
This bears out when you compare the lower-clocked 1940 with the Samsung chip to the higher clocked 2215 as well.
AND -- what the heck did they do to the graphics system on my beloved 2215? How could HP gone that far back in this area. It seems all we hear about is the multimedia capabilities of the device and how it is meant as a "consumer" device. Someone dropped the ball on the graphics side of things.
Whydidnt
|
| |
|
|
|
08-20-2003, 09:35 PM
|
|
Intellectual
Join Date: Aug 2007
Posts: 238
|
|
How does the 1945 compare? If the StrongARM did so well against the XScale, how does the Samsung chip fare?
|
| |
|
|
|
08-20-2003, 10:00 PM
|
|
Executive Editor
Join Date: Aug 2006
Posts: 29,160
|
|
Quote:
|
Originally Posted by easylife
Nice benchmarks, Jason. I would have liked to get some benchmarks on my e310 in comparison to the rest of the list...
|
You can go to the Spb Benchmark comparison page and compare the e310 we tested with other Pocket PCs in the list. Or you can download Spb Benchmark and test your e310 yourself. :wink:
|
| |
|
|
|
08-20-2003, 10:35 PM
|
|
Pupil
Join Date: Aug 2003
Posts: 16
|
|
[office space]yeah, I'm going to need you to go ahead and send me that wm2003 update for my ipaq [/office space]
seriously, though, HP should really consider releasing an update for the 3600 and 3700 series, as this proves theres really no technological reason behind not offering updates, and especially since this update may even improve the speed of these devices.
edit: and a word about the graphs.... I found them hard to read and compare because the order (and color!!) of the devices changes between each graph. The order and color should stay fixed for each device to make comparing graphs easier.
|
| |
|
|
|
|
|
Linear Mode
