Spb Releases Spb Benchmark [Archive]

Janak Parekh

06-26-2003, 05:45 PM

<div class='os_post_top_link'><a href='http://www.softspb.com/press/pressreleases/2003/jun25.html' target='_blank'>http://www.softspb.com/press/pressr...2003/jun25.html</a><br /><br /></div>Wondering how Jason made all those awesome performance graphs for his <a href="http://www.pocketpcthoughts.com/articles.php?action=expand,13980">iPAQ 2210 review</a>? Wonder no longer. :)<br /><br />"Spb Software House announces the release of Spb Benchmark 1.0, the premier benchmarking tool for measuring Pocket PC performance. Spb Benchmark combines real-world performance and battery tests with synthetic performance tests to give an accurate representation of how well a Pocket PC performs in everyday, realistic tasks. Since the debut of the Pocket PC in April 2000, media and Pocket PC enthusiasts have had no effective way to measure Pocket PC performance. Several simplistic benchmarking tools have emerged, but most have focused on a limited range of synthetic CPU and graphics tests that have little bearing on real-world performance. Battery testing was performed through error-prone manual methods or using battery testing applications that often had unpredictable results. With the launch of Windows Mobile 2003 for Pocket PC, the product has entered a third generation with new high-water marks for performance and reliability. Now, with Spb Benchmark, reviews will have the proper tool to truly get a handle on device performance and speak with authority, armed with hard metrics."<br /><br />Two example performance graphs were given, both of which, incidentally, show the HP 2215 at the top of the heap: <a href="http://www.spbsoftwarehouse.com/press/pressreleases/2003/jun25/graph1.gif">Spb Benchmark Index</a>, which seems to be their combined scoring system, and a <a href="http://www.spbsoftwarehouse.com/press/pressreleases/2003/jun25/graph2.gif">Pocket Word Open</a> graph, where the 2215 is even more dominating.

entropy1980

06-26-2003, 05:51 PM

Still crazy to see that good ole 3650 kicking @$$ and taking names!!! sheesh!!! 8O

pocketpcdude1024

06-26-2003, 05:58 PM

Did I misread, or will this app cost HUNDREDS of $$$ for commercial use and non-reviewers? :idontthinkso:

No, it looks like I didn't:
Spb Benchmark is free for a limited range of users: if you are a member of the media or someone who reviews Pocket PCs for an enthusiast site, we welcome your use of this tool. If you are an end user who wants to learn more about Pocket PC performance using this tool, it is likewise free. If you are a business user who will derive benefit from Spb Benchmark (OEM, ODM, enterprise user evaluating Pocket PC deployment, etc.), Spb Benchmark must be purchased for $495 US.

Luckily, the $500 fee only applies to industry, not end-users or enthusast sites... ;)

Skoobouy

06-26-2003, 06:27 PM

C|Net, take note.

vassiliphilippov

06-26-2003, 06:50 PM

Here is Spb Benchmark Index of all devices we have tested:
http://www.spbsoftwarehouse.com/downloads/full.gif

Vassili Philippov
Spb Software House

vassiliphilippov

06-26-2003, 06:55 PM

We will appreciate if you send information (test results) about Pocket PC devices that are not included in this list. It is also interesting to compare speed of one device with different platform versions (Pocket PC 2000, Pocket PC 2002 and Pocket PC 2003) and with different CPU clock frequency.

Vassili Philippov
Spb Software House

reddeerguy

06-26-2003, 06:57 PM

Question - why does the 3650 perform so well on this benchmark? Is that an accurate representation of its speed?? :?: Also - has anyone checked the old 3800 series on this benchmark?

vassiliphilippov

06-26-2003, 06:59 PM

Question - why does the 3650 perform so well on this benchmark?

It was a fast device :) ARM is faster than XScale 250.

Is that an accurate representation of its speed?

You can read how Spb Benchmark index is calculated at:
http://www.spbsoftwarehouse.com/products/benchmark/index_description.html

You can also read about each test at:
http://www.spbsoftwarehouse.com/products/benchmark/tests.html

Vassili Philippov
Spb Software House

Jimmy Dodd

06-26-2003, 07:08 PM

vassiliphilippov

06-26-2003, 07:12 PM

Vassili: do you have a specific address of where to send results we collect on devices you haven't included in your list?

Please email them to info at softspb.com
Subject: Spb Benchmark results

Thank you in advance!

Vassili Philippov
Spb Software House

Len M.

06-26-2003, 07:17 PM

It would be nice if the file system and memory card tests would post results in KB/s or MB/s instead of time elapsed. What most folks are interested is a number that is higher when better.

Len M.

vassiliphilippov

06-26-2003, 08:08 PM

It would be nice if the file system and memory card tests would post results in KB/s or MB/s instead of time elapsed. What most folks are interested is a number that is higher when better.

They do! :D All tests are measured in both time and speed. Compare graphics use speed. For example:
http://www.spbsoftwarehouse.com/downloads/write.gif

You can get results of other file system tests (Read 1 MB file, Copy 1 MB file, Write 10 KB x 100 files, Read 10 KB x 100 files, Copy 10 KB x 100 files, Directory list of 2000 files) at:
http://www.softspb.com/products/benchmark/compare.asp

Vassili Philippov
Spb Software House

Len M.

06-26-2003, 08:10 PM

It would be nice if the file system and memory card tests would post results in KB/s or MB/s instead of time elapsed. What most folks are interested is a number that is higher when better.

They do so! All tests are measured in both time and speed. Compare graphics use speed. For example: ...

Vassili Philippov
Spb Software House

That's great to see. The Spb Benchmark Web Site only showed elapsed time on the screen shots.

Wow -- looks like a great benchmarking tool!

Len M.

reddeerguy

06-26-2003, 10:44 PM

Sorry to beat this to death, but I'm a little confused. Why is it that 3 years after the release of the first Ipaq (3600) we have finally "caught up" with its original speed? I would have thought we would be miles ahead. Is the main advantage of the Xscale mostly battery performance, not actual speed improvement?? Seems odd to me to see the 3650 right next to the 2210/2215 in the benchmarks. Sorry if this is obvious to everyone else.

Jason Dunn

06-26-2003, 11:07 PM

pro_worm

06-26-2003, 11:09 PM

(dare I ask?)
Is that an overclocked 3650?
Otherwise... 8O

Even today, the original 3600 is an impressive device... 3.8" screen, relatively light (for that screen size class) and uber-nice in terms of styling - with its sleek curves and no visible plastic, it would still sell if released today...
Indeed, it took the 2215 to finally make me switch
:mrgreen:

huangzhinong

06-26-2003, 11:20 PM

Here is Spb Benchmark Index of all devices we have tested:
http://www.spbsoftwarehouse.com/downloads/full.gif

Vassili Philippov
Spb Software House

If this is the all benchmark, I am really worried about H2215 speed. I have owned h2215 three days now, I didn't see much speed increase which your guys feel so excited about. From the table, H2215 is only a little bit faster than E775(Pocket PC 2002 version). While immier has reported and benchmarked that E775 running PPC 2003 is much faster than E775 Running PPC 2002, it is possible that E775 PPC 2003 version is faster than H2215.

denivan

06-26-2003, 11:25 PM

(dare I ask?)
Even today, the original 3600 is an impressive device... 3.8" screen, relatively light (for that screen size class) and uber-nice in terms of styling - with its sleek curves and no visible plastic, it would still sell if released today...
Indeed, it took the 2215 to finally make me switch
:mrgreen:

Exactly, my first pocketpc was a 3660 and I haven't upgraded since ;) I love the way it looks, I think it was the best design of all pocketpc's ever and had great features overclassing all others at that time. The only thing that will make me switch now is battery life, I really hate battery life on my iPaq and I hope that the current versions will do better (in real life, not in bechmarks ;)) In fact, the only thing keeping me off at upgrading is the fact that I think the 2210 doesn't look as classy :roll:

iPaq 36xx power forever ! :rock on dude!:

Jason Dunn

06-26-2003, 11:51 PM

(dare I ask?)
Is that an overclocked 3650?

No, if it was overclocked, we'd indicate so in the device description. That's a stock iPAQ.

Len M.

06-27-2003, 12:12 AM

One more piece of data would help a alot and make the benchmark results more useful: document the type (and size) of the storage media, its manufacturer and its date code. CF flash cards are much faster than SD cards which are much faster than MMC cards. CF and PC Card hard disks are usually faster than all the flash cards.

Within the range of CF flash cards, there's a 4 to 1 range of write speeds.

Len M.

Newsboy

06-27-2003, 12:26 AM

I'll be providing data on an over and underclocked HP 1910 as soon as I get my download.

Jason Dunn

06-27-2003, 12:30 AM

One more piece of data would help a alot and make the benchmark results more useful: document the type (and size) of the storage media, its manufacturer and its date code.

If you look back at the original post that I did with the memory cards (http://www.pocketpcthoughts.com/index.php?topic_id=13886) you'll see the data taxonomy is [SIZE] [BRAND] [TYPE].

ie: 1 GB Sandisk CF, 28 MB Sandisk MMC, 256 MB Sandisk SD

The date code is too esoteric for me - what difference would that make? Lot numbers? I don't think the average person would understand or care about that. 8O

I do appreciate your feedback and constructive criticism though - it's nice to see someone who is as hard-core about benchmarks as I am. :mrgreen:

pro_worm

06-27-2003, 01:14 AM

Exactly, my first pocketpc was a 3660 and I haven't upgraded since ;) I love the way it looks, I think it was the best design of all pocketpc's ever and had great features overclassing all others at that time. The only thing that will make me switch now is battery life, I really hate battery life on my iPaq and I hope that the current versions will do better (in real life, not in bechmarks ;)) In fact, the only thing keeping me off at upgrading is the fact that I think the 2210 doesn't look as classy :roll:

I sent my 3600 series into Pocket PC Techs during one of their periodic uber-sales. Had the whole thing upgraded, with a battery twice the size of the original and an extra 64 megs of memory - all for less than any Pocket PC available on the martket.
The result? Mind-blowing. The iPAQ's battery NEVER died.
If you have some spare cash, and want your iPAQ to keep chugging, I strongly recommend sending to it Pocket PC Techs - about 1 week turnaround time.

iPaq 36xx power forever ! :rock on dude!:

Here here 8)

pro_worm

06-27-2003, 01:18 AM

If this is the all benchmark, I am really worried about H2215 speed. I have owned h2215 three days now, I didn't see much speed increase which your guys feel so excited about. From the table, H2215 is only a little bit faster than E775(Pocket PC 2002 version). While immier has reported and benchmarked that E775 running PPC 2003 is much faster than E775 Running PPC 2002, it is possible that E775 PPC 2003 version is faster than H2215.

These benchmarks aren't completely accurate. After all, it was widely agreed that the 3900 was the fastest second generation PPC (currently supplanted by the 2215) but these benchmarks don't show that. In fact, they show the 3900 series to be slower than the original iPAQ - which is simply not the case, in my understanding. The 3900 is zippy (I speak from PPC-as-a-hobby experience; I have never owned the 3900.)

Finally, play Donkey Kong Country on your 2215 - with SOUND - and I'm sure you'll be convinced you made the right choice :D

smashcasi

06-27-2003, 02:21 AM

These benchmarks aren't completely accurate. After all, it was widely agreed that the 3900 was the fastest second generation PPC (currently supplanted by the 2215) but these benchmarks don't show that. In fact, they show the 3900 series to be slower than the original iPAQ - which is simply not the case, in my understanding. The 3900 is zippy (I speak from PPC-as-a-hobby experience; I have never owned the 3900.)

Just offering my opinion here, but I think the benchmarks posted so far look pretty accurate. Through the Pocket PC development I do at work I've had a chance to use and abuse a fair number of these devices and the difference in speed between the 3600 series and later is pretty easy to see. Personally I still use my trusty 3670 while the later devices sit around collecting dust. In terms of overall performance, running apps compiled for 2003 on 2003 devices is the first time that I've felt that we've taken a major step forward since the advent of XScale.

Jason Dunn

06-27-2003, 04:44 AM

These benchmarks aren't completely accurate. After all, it was widely agreed that the 3900 was the fastest second generation PPC (currently supplanted by the 2215) but these benchmarks don't show that. In fact, they show the 3900 series to be slower than the original iPAQ - which is simply not the case, in my understanding. The 3900 is zippy (I speak from PPC-as-a-hobby experience; I have never owned the 3900.)

So you're saying that because people "felt" the 3900 was faster, it therefore was, but you've never actually owned one? 8O No offense, but that's an awfully bold statement to make without having any hard facts to back it up with. Spb Benchmark will destroy many pre-concieved notions about Pocket PC speed, and the truth will set you free!

The sad fact is that the 1st generation Xscale CPU was such a piece of crap, it hobbled every Pocket PC it touched, the 3900 included. Nobody wants to admit that they're brand new Pocket PC was really slower than their old 3650, so I think you'll find that emotion and personal pride plays the primary role in the "my 3900 feels fast" benchmark. ;-)

Benchmarks, when done fairly and accurately, don't lie.

Kaber

06-27-2003, 05:31 AM

I overclock both my 3765 and my 3955.

My 3955 "feels" faster.

Kirkaiya

06-27-2003, 06:17 AM

So Jason -

Any idea when we might get some benchmarks from the new HP 1940 model, with the 266 MHz ARM-based (?) processor?

I would think that, if it's got the 200 MHz memory-bus, and the processor is a souped up version of the StrongArm used in the 36xx series, then I would think it would rock.

I haven't read a lot about that processor, other than it is supposedly fast.

I would LOVE to see some numerical comparisons to the 2215....

Newsboy

06-27-2003, 06:24 AM

Just submiitted results for the HP iPAQ 1910 @ 300 Mhz running PPC2002.

Fourth in the overal index, 7th in CPU, 7th in file index, 1st in graphics index (3077.9!!!), 10th in ActiveSync index, and 8th in platform index.

Not finished with battery tests yet.

Jason's right, benchmarks don't lie. Which makes me wonder why the 1910 displays graphics so much faster than anything else? Hmmmm!

Jason Dunn

06-27-2003, 06:32 AM

Any idea when we might get some benchmarks from the new HP 1940 model, with the 266 MHz ARM-based (?) processor?

Whenever they start shipping - right now they're not on the market anywhere AFAIK. :wink:

Jason Dunn

06-27-2003, 06:32 AM

Just submiitted results for the HP iPAQ 1910 @ 300 Mhz running PPC2002.

Do me a favour and email me those same results - jason at pocket pc thoughts dot com. Thanks!

Newsboy

06-27-2003, 06:35 AM

Also of note: the 1910 @ 300 Mhz ran Arkaball at 139 frames per second, beating the previous record holder of 108 frames per second, the iPAQ 3600.

vassiliphilippov

06-27-2003, 09:59 AM

Here are results of Compaq iPAQ 3970 (2002, 400Mhz) where Compaq iPAQ 3600 Series (2000, 206Mhz) is considered as a base.

Indices: (all indices are 1000 for Compaq iPAQ 3600)
http://www.spbsoftwarehouse.com/downloads/3900index.gif

Test results: (100% corresponds to Compaq iPAQ 3600 speed)
http://www.spbsoftwarehouse.com/downloads/3900tests.gif

So as you see for some tests 3970 is faster (>100% in the right column), for some tests 3600 is faster.

Vassili Philippov
Spb Software House

Kaber

06-27-2003, 04:56 PM

It'd be interesting to compare the results for the 39xx from spb's Benchmark against Anton Tomov's Pocket PC Mark (http://www.pockethackmaster.com/ppcmark.htm). That's what I'm using on my iPAQs.

Jason Dunn

06-27-2003, 05:24 PM

It'd be interesting to compare the results for the 39xx from spb's Benchmark against Anton Tomov's Pocket PC Mark (http://www.pockethackmaster.com/ppcmark.htm). That's what I'm using on my iPAQs.

Unless he's using exactly the same benchmarks as Spb Benchmark is, it's impossible to compare. What sorts of tests does his app do?

Len M.

06-27-2003, 06:02 PM

I'm having a small problem with Spb Benchmark. With an iPAQ 3650 and either an HP single CF expansion pack or a PItech dual CF MemPlug, Benchmark can't find the storage card.

File Explorer sees it just fine and lists all of the files on the card, but Benchmark doesn't seem to know it exists.

I selected the "Storage Cards" benchmark group and pressed "next." The next screen is titled "Spb Benchmark: Storage Card Tests" and the Storage Card list is blank.

The card is a RiDATA 256 MB CF card.

Any help you can offer would be appreciated.

I'll try it in a 3835 next, again with the 256 MB CF card and also with a 256 MB SD card and a 64 MB MMC card.

Len M.

Newsboy

06-28-2003, 08:10 AM

I'm bettin' five bucks that Jason created the Benchmark video using Muvee! :D I think he's addicted to that program. :rotfl:

Len M.

06-29-2003, 06:08 AM

Vassili,

For Storage Card tests, why should Write 1 MB File be so much slower than Copy 1 MB File? I'd have thought that writing a new file to the Storage Card would be faster than copying a file already in memory.

Could you offer some insight as to why this is so? Is it that the CopyFile API function is so much faster than the WriteFile function?

Thanks!

Len M.

vassiliphilippov

06-30-2003, 05:00 PM

For Storage Card tests, why should Write 1 MB File be so much slower than Copy 1 MB File? I'd have thought that writing a new file to the Storage Card would be faster than copying a file already in memory.

Could you offer some insight as to why this is so? Is it that the CopyFile API function is so much faster than the WriteFile function?

That is because of CopyFile and WriteFile implementation. CopyFile "knows" much more about the process (it knows final file size, etc) and can better optimize writing process. I think that is a reason.

Vassili Philippov
Spb Software House