More On Persistent Storage From The WM Team Blog

[Janak Parekh]

Jon linked to a post on the Windows Mobile Team Blog <a href="http://www.pocketpcthoughts.com/forums/viewtopic.php?p=354754">last week</a> discussing some of the technical aspects and rationale of persistent storage, and this of course drew questions from the community. Not only was Mike Calligaro listening, he was also reading our thread from the subject, and has written up a two part set of answers addressing some of the issues raised. Here's <a href="http://blogs.msdn.com/windowsmobile/archive/2005/07/18/440092.aspx">part 1</a>, and here's <a href="http://blogs.msdn.com/windowsmobile/archive/2005/07/18/440139.aspx">part 2</a>.<br /><br />This is informative reading, even for MVPs like me who knew that WM5 would have persistent storage for some time -- so if you're planning to get a WM5 device, check it out. I would also like to footnote this post by saying Mike's tone in these posts is refreshingly clear and strikingly straightforward. Kudos to Mike for spelling out a lot of the team's design decisions. 8)

[PneTool]

Great write up. I am still confused as to why people are against PS. To me this is the best thing to happen to handhelds since the Newton (which had PS). I can still plug in a newton and have all my data where I left off, its great! I welcome safe data and longer battery life doesn't everybody?

[gibson042]

Those posts, especially the second, were wonderful. It is great to read such well-reasoned, well-written statements that came from within Microsoft. If Mike is a good representative of the whole team, then I'm even more excited about WM5 now than I was before!

[Leon]

It has been said before: great piece. I feel much more comfortable now and I am looking forward to the upgrade.

[Jason Dunn]

Great response from Mike, and it helped clear up some of my own concerns as well - they're doing it exactly the way I was hoping they would, and I'm looking forward to it.

[Sven Johannsen]

If Mike is a good representative of the whole team, then I'm even more excited about WM5 now than I was before!

He is. If there is one thing I got out of being an MVP it's actually getting to meet some of the folks responsible for the OS and apps on my PPC. Haven't met one that wasn't truly commited to bringing out the best product they knew how, within the constraints levied on them by the business, and reality.

[surur]

If Mike is a good representative of the whole team, then I'm even more excited about WM5 now than I was before!

He is. If there is one thing I got out of being an MVP it's actually getting to meet some of the folks responsible for the OS and apps on my PPC. Haven't met one that wasn't truly commited to bringing out the best product they knew how, within the constraints levied on them by the business, and reality.

The constraints of reality is always so annoying. If it wasn't for that I would have had Office 2025 on my PPC by now ;)

Surur

[Kevin Daly]

As a developer, I've already assimilated one important lesson.
- I have to admit that with my Windows Mobile applications I've sometimes tended to be a bit lazy and apply logic like this at certain points: "Something might have changed, so save the config data". I'll now replace this with "Check whether anything has changed, and if it has, save the config data".

I'm oversimplifying both what I've been doing and what I'm going to do, but that's the gist of it.

[Darius Wey]

Mike definitely resolved a few issues which have been floating around the forums over the past month. It was very informative. :)

[disconnected]

It was nice to read something that sounded like English, instead of like a manual. I especially like the prospect of longer battery life.

The only thing that I'm a little uneasy about is the timing of writing stuff back to ROM and the possibility of losing data on a soft reset, but I guess if things are saved often enough it shouldn't be a problem. Since it's doing all this writing every time you turn off the PPC, I wonder what happens if you try to turn it right back on again. Would there be a delay, or would it get confused?

Now if only someone would explain the lack of an option for WiFi Synch....

[Gerard]

Well, as the "One poster was going to thousands of web pages every day" he mentioned I must now express some gratitude at having been heard. not accurately, but the point was taken and addressed, and for that I am happier.

I didn't suggest that I was downloading thousands of web pages every day, but rather a couple of hundred or so, and that many of these included lots of accessory files. I had made a reference to having visited 4 or 5 pages in forums and already having well over 100 files of various types in my temp folders... thousands of pages would leave no time for anything else, even sleep, and certainly not actually reading anything in those pages.

Thousands of files written per day, as would be typical in looking at anything like 100 or more pages with even a few dozen icons or whatever small graphics, could easily use up 250,000 write cycles long before the next device model was released, never mind the lifespan of the battery running out or the screen developing wear patterns. I do not expect to have to replace my devices every couple of months just because a lot of people cannot be bothered to run regular backups, and therefore Microsoft has moved to a ROM-smashing OS.

His tip about stopping Pocket IE caching will go a long way to stopping my worry. I should think that this hack will find its way into all the several tweak apps, offering peace of mind for any user who shares this concern about wearing out ROM. A simple explanation, a couple of sentences even, should be enough for any non-power user to understand the advantage. And since Pocket IE is so weakly designed for recording actual paths to temp files anyway, I have often found dupicates and triplicates of files from visited pages in those folders, all sequentially numbered of course. There are, in my experience at least, no savings in download speed to be had in keeping a cache in a PPC.

I too am heartened by reading some intelligent and thoughtful words which are in direct response to our questions. And already a developer says he's adapting to the new order, which one can only hope will become common thinking among software makers for the PPC. Things do appear to be looking up!

[ipaq_wannabe]

Well, as the "One poster was going to thousands of web pages every day" he mentioned I must now express some gratitude at having been heard. not accurately, but the point was taken and addressed, and for that I am happier. I didn't suggest that I was downloading thousands of web pages every day, but rather a couple of hundred or so, and that many of these included lots of accessory files. I had made a reference to having visited 4 or 5 pages in forums and already having well over 100 files of various types in my temp folders... thousands of pages would leave no time for anything else, even sleep, and certainly not actually reading anything in those pages. Thousands of files written per day, as would be typical in looking at anything like 100 or more pages with even a few dozen icons or whatever small graphics, could easily use up 250,000 write cycles long before the next device model was released, never mind the lifespan of the battery running out or the screen developing wear patterns. I do not expect to have to replace my devices every couple of months just because a lot of people cannot be bothered to run regular backups, and therefore Microsoft has moved to a ROM-smashing OS. His tip about stopping Pocket IE caching will go a long way to stopping my worry. I should think that this hack will find its way into all the several tweak apps, offering peace of mind for any user who shares this concern about wearing out ROM. A simple explanation, a couple of sentences even, should be enough for any non-power user to understand the advantage. And since Pocket IE is so weakly designed for recording actual paths to temp files anyway, I have often found dupicates and triplicates of files from visited pages in those folders, all sequentially numbered of course. There are, in my experience at least, no savings in download speed to be had in keeping a cache in a PPC. I too am heartened by reading some intelligent and thoughtful words which are in direct response to our questions. And already a developer says he's adapting to the new order, which one can only hope will become common thinking among software makers for the PPC. Things do appear to be looking up!

when he (in the blog) mentioned "that PIE would cache" - would this still be true when the PIE's cache is set to an external storage card (SD or CF)?

im assuming that this is what he is talking about, right?

if this were the case, then i have nothing to worry EVEN if i view THOUSDANDS (assuming it is possible) of web pages a day?

[Gerard]

Your CF or SD card won't last so long, but these cards are rated for at least a million write cycles anyway... But of course that levelling thing might not apply to cards. Even if it did, I've seen a lot of errors on flash storage cards from various companies. I'd rather not cache PIE to one, as cards aren't free.

[Darius Wey]

when he (in the blog) mentioned "that PIE would cache" - would this still be true when the PIE's cache is set to an external storage card (SD or CF)?

im assuming that this is what he is talking about, right?

if this were the case, then i have nothing to worry EVEN if i view THOUSDANDS (assuming it is possible) of web pages a day?

By default, the OS stores it in ROM, but if you manage to tweak the temporary internet files to be stored on a storage card, then I believe that'd be another solution to the problem.

[Dyvim]

Am I confused, or is the write cycle limit (e.g. 250,000 writes) per sector (or byte or whatever) not for the ROM as a whole. That is, each sector can be written approx 250,000 times not that the ROM can only be written to 250,000 times.

So assuming that they evenly distribute writes throughout the entire ROM (which the MS blog indicates that they do) and you were doing 100KB writes to a 64 MB ROM (I'm just making up these numbers) then wouldn't you be able to get approx 10 X 64 X 250,000 = 160,000,000 writes to your ROM?

[Robert Levy]

The only thing that I'm a little uneasy about is the timing of writing stuff back to ROM and the possibility of losing data on a soft reset

I'm (slowly) working on another blog post to answer your data loss question, but the general idea is that you shouldn't have to think about this.

Files get flushed pretty quickly once an app is done writing to them. The registry is where we saw lots of backwards compatibility &amp; data loss problems. So a bunch of work was done to flush the registry at appropriate times without interrupting the user (the wait cursor that randomly appears on Smartphone 2002 and hangs your device for a few seconds is a good example of what happens when you flush at the *wrong* time). Some of the things that cause us to flush on 5.0 (Pocket PC &amp; Smartphone):

0) As Mike said, after you hit the power button to put your device into standby
1) after the device has been idle for a certain number of seconds
2) after a control panel is closed [including 3rd party control panels]
3) after a CAB file has been installed
4) when a 3rd party app programmatically soft-resets the device using KernelIoCtl or ExitWindowsEx

[davea0511]

Your CF or SD card won't last so long, but these cards are rated for at least a million write cycles anyway... But of course that levelling thing might not apply to cards. Even if it did, I've seen a lot of errors on flash storage cards from various companies. I'd rather not cache PIE to one, as cards aren't free.

Good point. It would depend on how much free space you have on your storage card. I usually have mine filled to the hilt.

Cards are getting super cheap - each enough that I'll be concerned more with the hassle of a bad card than the cost. :D

[bdoherty]

Does any one know WM5 will allow the ability to use a CF card instead of ROM for your persistant storage? I have a hx4700 and have installed just about all my apps (except for today plug-ins) on my high speed 1Gb CF card which i never remove.

I'm considering upgrading to WM5 to increase battery length (and the ability to be a usb drive sounds useful to), and would like WM5 to use the CF card instead of the ROM where possible.

I've done a bit of benchmarking, and the FlashROM is at least 10 times slower than my CF card.

Thinking about it, i guess the registry will have to be on the ROM because of the delay in initilising the card slots. If any one knows about this, i'd be interested to hear.

[Nurhisham Hussein]

For programs, i don't think it's an issue, you can still use cards. The question is the registry (which really isn't a factor, given its small size and the buffered writes they're putting in) and cache (this one has me worried to).

[Gerard]

If Mike Calligaro is still following this thread, I'd just like to point out a possible error in his suggestions for reducing wear and tear on flash ROM. He had suggested that a possible solution to the Pocket IE temp cache write frequency problem was to:

".. use a registry editor to go to HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\Cache\Content and set the "CacheLimit" value to 0."

Well, perhaps this works under WM5.0. I couldn't know, as I don't have WM5.0, only WM2003. But I tried this a number of times since reading his posting, and the registry edit won't 'stick' if set to zero. Any value above that and it seems to stay at the user defined value, though I've not tested more than a few different values here. But plainly, any value above 0 is going to be allowing constant re-writing of the flash ROM unless the buffer is somehow preventing contant over-writes.

Could he, or someone else with access to aa wm5.0 device, please perform this registry edit, use the value 0, and then soft reset, open Pocket IE, and then check both their temp folders and their registry in this location? I end up with a default value over 21MB returning if setting to zero and following this routine. A value of 5MB sticks, as do several others.

[bdoherty]

I'm guessing we'll probably be able to hack the registry to move the cache off of the rom to the faster and longer lasting SD/CF cards?

Maybe somebody with WM5 already can confirm this?

[Janak Parekh]

I'm guessing we'll probably be able to hack the registry to move the cache off of the rom to the faster and longer lasting SD/CF cards?
Yes. (http://blogs.msdn.com/windowsmobile/archive/2005/07/20/441038.aspx)

--janak

[Darius Wey]

Could he, or someone else with access to aa wm5.0 device, please perform this registry edit, use the value 0, and then soft reset, open Pocket IE, and then check both their temp folders and their registry in this location? I end up with a default value over 21MB returning if setting to zero and following this routine. A value of 5MB sticks, as do several others.

It works fine in WM5.0 - both the cache limit hack and the storage card hack. As a suggestion, before you apply the hack, try clearing your cache from within Internet Explorer Mobile. Next open your registry editor and set the value to "0". Soft-reset your device, and see if it sticks.

[Gerard]

Um, yeah. I clear my PIE cache about 10 times per day, or my device would choke and freeze. I also cleared it from within PIE and then immediately closed PIE (I use GigaBar, so the X means close, not dumb minimize) before each and every test of this edit.

As this seemed not to be working I took a more radical step. Under prior OS versions the index.dat files had been deletable in PIE's cache, but under WM2003 this ceased to be the case. They just grow and grow... so I long ago saved a couple of REG files to use periodically with Resco Explorer, changing between two folders for the various types of cache files. I keep these in the root \Temp folder, calling them \Temp\1 and \Temp\2. After importing and then a soft reset, 100% of the files inside the prior folder are deletable, making for a clean start in PIE. Having done this, I then edited the registry per Mike's instructions, and again found that a value of 0 did not work, while a value above 0 worked fine.

Did I miss something?

[Darius Wey]

Well, FYI, I've tested this in WM2003SE and the value of "0" does not stick. So I guess you're not alone.

If I have the time within the next day or two, I'll dig around to see what in WM5.0 is allowing the value to stick that WM2003/WM2003SE is not.

[Nurhisham Hussein]

Tried as well (WM2003SE) - both directly through a registry editor and with Memmaid - and couldn't get it to stick either. The value I get is around 2241kb, and yes this was after clearing the cache in PIE. Memmaid reports all these as under the \Windows\profiles\guest directory if that helps any.

[Gerard]

Memmaid reports all these as under the \Windows\profiles\guest directory if that helps any.

Yes, but I didn't like having to dig for them (sometimes it's handy to go looking for files, when MultiIE can't find a page in the cache and such), so I made these two REG files:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders]
"History"="\\Temp\\1"
"Cookies"="\\Temp\\1"
"Cache"="\\Temp\\1"



Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders]
"History"="\\Temp\\2"
"Cookies"="\\Temp\\2"
"Cache"="\\Temp\\2"


Every week or two I just swap locations, then delete the bulked-up index.dat file.