Menneisyys
08-13-2005, 08:40 PM
Reducing Internet bandwidth usage on the Pocket PC - A complete roundup & comparison of Toonel, OnSpeed, Skweezer, WebWarper and the like
There're two problems with most web sites.
First: particularly if you enable images, navigating to desktop-optimized pages may result in getting HTML pages and images totalling even hundreds of kilobytes. Mostly, you won’t be able to make use of high resolution, high quality images on a PDA (unless you want to save them/zoom into them); that is, in most cases, downloading desktop-optimized pages to a PDA is an overkill. The same can be said about HTML pages thsemselves; they are, in general, textual, and, therefore, highly compressible: text can be compressed with plain compression tools very good (JPEG’s and other Web image types are a different story – more on their compression later).
Second: being HTML, standard (non-PDA-optimized) Web pages tend to are also full of unnecessary stuff: whitespaces, scripts and formatting that are completely useless for a lot of PDA users that prefer viewing pages in the so-called “One Column” view mode of the Pocket Internet Explorer (PIE) in WM2003SE+ (or, if you use an older PIE version, similar rendering modes of the ftxPBrowser and the PIEPlus PIE plug-ins; or, if you use NetFront 3.2, the Smart-Fit Rendering mode). (Please see the “Built-in one-column view on WM2003 devices” section in my article on Web browsers (http://winmobiletech.com/072005BrowserRoundup/)) on the One Column view mode and its alternatives.) Then, almost all <table>, <frame> and the related tags become useless. This means a service that strips all these additional whitespaces, tables and converts everything in one column may also be of advantage.
In this article, I scrutinize both subjects and the services/applications meant for Pocket PC users.
1. What is this all about? How does content cleanup and compression work?
1.1 Content cleanup
As has alredy been mentioned, the majority of web pages contain a lot of unnecessary whitespaces. Web browsers simply ignore them - for example, if you insert ten space characters one after another, the Web browser will just display one instead of ten. (This is why   has been introduced BTW.) Also, Web browsers just ignore CR/LF characters unless they are in <PRE> blocks - that is, they can also be safely removed to save bandwidth usage.
Also, there're a lot of additional constructs that aren't necessarily needed in a "simple", "dumb" browser like a Pocket PC-based one. For example, objects, plug-ins and, in a lot of cases, stylesheets are not needed and can safely be omitted. These also help a lot in reducing bandwidth usage.
Please note that not all HTML files can be content stripped. For example, HTML files that don't contain special tags or excess whitespace can't be "compressed" this way. For example, a page that is already PDA-compliant (because it has no tables/frames) can’t be content stripped. You'll even see an example of these HTML pages later, in the "How did I test? " section.
1.2 Compression
You can not only strip unnecessary textual information (table/frame tags, whitespace etc.) from a HTML file before sending it to the client, but also compress it with a compressor technology that is compatible with the client browser. The most common compressor technology is gzip. Note that the HTTP protocol, the protocol that is used to request and return Web resources, allows for other compression techniques too (for example, compress), but they are much more rarely used in lightweight clients like PDA-based browsers.
Gzip can achieve much better compression ratios than the above-described content stripping/whitespace eliminating algorithm. For example, as you'll see in the benchmark session, an ordinary test page that is originally 582 kbytes, with content/whitespace stripping can be "compressed" to around 200 kbytes; with gzip compression, however, to 20 kbytes, which is an order of magnitude better compression rate than with simple content/whitespace stripping.
As has already been pointed out, HTTP, the protocol in charge of sending page and resource requests and returning pages and images, allows for gzip compression. However, almost none of the current Web servers actually do return compressed pages to save CPU processing. (It takes much more CPU resources to compress a page before returning it to the client than the bandwidth overhead of the uncompressed page.) That is, in the easiest setup (a simple browser is communicating with a Web server, as depicted in the following figure), the server will almost always send back an uncompressed response:
http://winmobiletech.com/062005CompressionTester/index_tiedostot/image002.gif
Here, along the Internet connection (the arrow in this figure), only uncompressed data flows. (From now on, I use solid arrows to denote uncompressed and dotted arrows to denote compressed data flow to avoid making the figures hard-to-read by adding excessive comments.)
This is where so-called proxies come into picture. What are they?
1.2.1 HTTP Proxies in/outside the PDA requiring explicit PDA-side configuration
These two solutions are the same in that they require explicit HTTP proxy configuration on the PDA, unlike the other solutions.
Configuring a HTTP proxy on the PDA can/should be done in one of two ways. If your particular browser (right now, NetFront only; Minimo may support this in the future) supports local proxy configuration, then, you just enter its address right into the browser. If you, however, plan to use PIE, you end up using the global, system-level proxy configuration tools.
Both of these cases (with NetFront as the example of the only PPC-based Web browser capable of using a local proxy configuration) are explained here (http://www.pocketpcmag.com/forum/topic.asp?TOPIC_ID=16017); the reader is referred to the tutorial there.
Incidentally, if you end up configuring the system-level proxy because you want to stick to PIE, you won’t be able to use the built-in MSN Messenger unless you use a so-called “Socks” proxy. Freely usable Socks proxies are pretty common, you can find a list of them at, say, here (http://www.aliveproxy.com/socks5-list/). Note that I’ve tested some local, Java-language, free Socks proxies (for example, jSocks (http://jsocks.sourceforge.net)), but I couldn’t make the Pocket PC MSN Messenger communicate through them, unlike with ‘real’ Socks servers found on the Internet.
1.2.1.1 HTTP Proxies running outside the PDA
Proxies are servers that act as an intermediary server for the browser. They receive HTML page and related resource (for example, linked images) requests from the browser and send them to the real Web server as if they (the proxies) were the clients requesting these resources. They receive the response from the Web server and send it over to their client, that is, the originating browser.
They (the proxy servers) do not need to send the response to the client browser verbatim, without any transformation. If they notice that the client is, for example, a PDA-based Web browser, they may choose to remove not needed HTML markup from HTML pages (that is, they may optimize the HTML) and/or compress it with gzip. They can do something similar to images, especially JPEG images: they may opt for downscaling them to save bandwidth.
How can I tell a PDA Web browser what proxy to use? - you may ask. Proxy server addresses must be told to browsers in their respective proxy setting dialog. An exception is PIE in the Pocket PC 2002 operating system and higher: in recent Pocket PC/Windows Mobile operating systems, the HTTP proxy settings are separated from PIE. In alternative browsers like NetFront, however, they can be set straight in them (note that if you use the “Auto-retrieve” option in NetFront, you can also use the system-level proxy setting). I've explained setting a proxy server in both PIE (that is, the operating system) and Netfront 3.1/3.2 here (http://www.pocketpcmag.com/forum/topic.asp?TOPIC_ID=16017).
With proxy servers coming into picture, the previous two-tier model (a browser and a Web server directly communicates) effectively becomes three-tier, as in the following figure. Please note that in this figure I've used a dotted arrow to denote compressed responses from the proxy server to the client.
http://winmobiletech.com/062005CompressionTester/index_tiedostot/image004.gif
What does this image convey? It's pretty simple: if the proxy server is running somewhere on the Internet and not on the PDA, then, because the (probably very slow and expensive) Internet connection of the client is already in the "compressed" zone (that is, where only compressed data is transferred - the thick, vertical line in the figure denotes this), you'll save up quite much money/time on using compression services!
There're quite a few public, free proxies on the Internet that you can use. A list of pages containing free proxy addresses will follow in the Anonymity section. (Almost) none of them support compression by default, though, so, we need to keep searching for other alternatives. The above introduction was still needed to draw a generic picture of the simplest case so that more advanced architectures can be understood.
Because of the lack of free, but compressing public servers, you may opt for deploying compressing proxy servers on your own, say, home desktop PC or, if it isn't in a heavily-defended (firewalls, other restrictions) environment, even on your work PC. These proxy servers are different from both third-party proxy servers, Web-based compression/content stripping services and on-PDA clients in that they must be run on one of your desktop PC's. This means they're only useful for people that have a PC on the Net with sufficiently large Internet bandwidth. If you aren't one of them, you may want to forget this option entirely. If you, on the other hand, do have one, these proxy servers are for you because they have several advantages over other solutions:
1, as it's your own server without other users, it will be much faster than other, mostly heavily overloaded services (including PDA-based proxy clients), assuming the desktop machine is running on a fast Internet connection
2, you can freely configure them
3, they are not only able to do basic stuff (content compression, image downscaling), but also goodies like ad filtering.
One of the best personal, free proxy servers is RabbIT (http://www.khelekore.org/rabbit/). It is a great tool because it supports both gzipping, HTTP/1.1 (just like PIE), image downscaling, proxy chaining (in order to remain anonymous) and all the goodies you can think of. It's Java (so, you'll need a JVM to run it) but uses the external, C-based ImageMagick (http://imagemagick.org/script/index.php) for super-fast image conversion.
Please read this article (http://www.winmobiletech.com/062005CompressionTester/UsingRabbIT.html) on setting up and running RabbIT if interested.
If you don't have the resources (a preferably 24/7 desktop PC to run your proxy server) or are afraid of hackers and unwanted guests using it (the latter is not really an issue because, for example, RabbIT has quite sophisticated autentication and client filtering capabilities), another option is using a third-party compressing proxy servers.
1.2.1.2 HTTP Proxies running inside the PDA
There is another category with services like toonel (http://www.toonel.net/downloads.html) and the OnSpeed (http://www.onspeed.com/us/index.php) client. They run locally on the PDA (so far, we only used proxy servers and/or third-party compressing servers on the Web somewhere, but definitely not on the same PDA) and are even more useful because
1, as far as Web-browsing (that is, the HTTP protocol) is concerned, they not only compress incoming textual HTML pages but also everything - outgoing requests, JavaScript, CSS pages (they are generally not compressed, not even by Web-based servers), everything.
2, they may (toonel supports this, OnSpeed, unfortunately, doesn’t) even go further by allowing for compressing SMTP and POP3, the two most important mail sending and receiving protocols. This is a big thing because, as per the standards defining these protocols (along with the MIME standard), they don't allow for any kind of compression (as opposed to HTTP). This is why it's only with locally (on the PDA in this case) running clients that you can use any kind of (decent; that is, much better than the simple Run-Length Encoding of V42.bis, the protocol widely used in modem-based communication, including GPRS) compression - because the TCP/IP protocol being used (POP3 / SMTP) officially doesn't allow for any kind of compression.
Local proxy-like clients, as they are behind the internet connection of the PDA (all programs running on the PDA share the same, with mobile phone-based connections, slow and expensive connection), they need another (hidden; meaning, the PDA user doesn't need to know its address) server somewhere in the internet to connect to. This server will compress the incoming (POP3/)HTTP responses and the outgoing ((POP3/)SMTP/)HTTP requests as is depicted in the following figure:
http://winmobiletech.com/062005CompressionTester/index_tiedostot/image008.gif
Please note that the mailer client communicates with the compresser client uncompressed (as per the POP3/SMTP standards); it's only between the compresser client and server that communication is compressed. This causes no runtime/bandwidth problems: as with the previous case, over the PDA-Internet connection, only compressed content is sent. It should also be pointed out, again, that in this case, the bandwidth save ratio may even be better than in the previous one because, with an external HTTP proxy, it’s only HTTP responses that can be compressed, HTTP requests, POP3 and SMTP can’t be.
This solution results in the most bandwidth saving (not only HTTP inbound, but also SMTP outbound, POP3 inbound etc), but it definitely eats up some of the (on a PDA, quite rare and meager) system resources. Advanced compressing clients like toonel and OnSpeed, however, don't deliver a considerable runtime hit to the device.
Setting up a local client on your PDA is easier than you think. As far as the Java-based toonel is concerned, you may want to read this tutorial (http://www.pocketpcmag.com/forum/topic.asp?TOPIC_ID=16017) on this subject. Setting up the OnSpeed client is even easier – you just fire its installer and fill in your OnSpeed login/password after installing.
As toonel/OnSpeed don't integrate into the core operating system unlike some of their alternatives (namely, the Globility client, which, being still in beta tests, will only be discussed in a later version of this article), you will need to manually configure the operating system / the browsers (with both clients) / the mailer client(s) (with toonel only) to use them. This is why you need to explicitly configure a proxy server on the PDA and fool the mailer client to believe the SMTP/POP3 server being at localhost. It isn't particularly complicated, though, and is thoroughly explained in the above link.
Should you want to choose your app from this category, which of the two services/clients to go for?
Both have advantages and disadvantages.
Toonel is Java-based. This means you’ll need to get a Java Virtual Machine (JVM) in order to run it. Some JVM’s are very cheap (for example, the IBM J9, Personal Profile 1.0 (http://www.handango.com/PlatformProductDetail.jsp?productType=2&optionId=1_2_2&platformId=2&siteId=1&productId=140997&programId=458&affiliateId=1218)) and will run Toonel just great. Add the price of Toonel (free!) and the overall reliability of the Toonel service, the ability to compress e-mail communication (unlike with OnSpeed), being highly anonymous and HTTP/1.1 compliant (unlike, again, OnSpeed), and you have a real killer. Hopefully, Toonel continues to be free and having great bandwidth, even from Europe. Should you need help with setting up Toonel, please read the above link. Also, you may want to read this article (http://www.pocketpcthoughts.com/forums/viewtopic.php?t=41081) on the available JVM’s. Its only real disadvantage is the lack of image downscaling.
OnSpeed is a different animal. It costs comparatively quite much (45 US$ a year), puts more burden on the connected HTTP server as it doesn’t support HTTP/1.1, isn’t highly anonymous and only offers HTTP compression (while Toonel also has e-mail compression). It, however, has configurable image downscaling.
Please note that some OnSpeed users have been faced with tremendous slowdowns (http://www.pocketpcthoughts.com/forums/viewtopic.php?t=40986). I haven’t noticed anything like that (have also tested everything over Wi-Fi with DHCP). They may have used an earlier client version – I don’t know.
Neither of these proxies support HTTPS. This means pages like this (https://internet-banking.dbs.com.sg/) won’t work at all, unlike with some “real”, outside-the-PDA proxies (not with RabbIT though!).
All in all, as of now (14-Aug-2005), I recommend Toonel over OnSpeed.
1.2.2 HTTP Proxy-like Web-based compression/content stripping services requiring no PDA-side proxy configuration
There're also Web-based services that do essentially the same as in/outside-the-PDA proxies explained above. This means you won't need to set a proxy server on your Pocket PC, but will need to navigate into a proxy-like web page with your browser and enter the URL of the target Web server that you want to access there. Content stripping and compression will be done by the middle-layer, proxy-like Web server and you will only receive well-compressed pages (that is, if you choose the right service that does this - for example, Skweezer, WebWarper or MobileLeap v2 – their list will come later). They also save a lot of bandwidth because there're already-compressed HTML pages transferred over the direct connection of the PDA. There're quite a few of services like these; the majority of services I've tested in this article are in this category.
The infrastructure can be depicted as follows:
http://winmobiletech.com/062005CompressionTester/index_tiedostot/image006.gif
It has clear advantages over the pure-proxy solution. First, they are much easier to use for the technologically changelled because no proxy options need to be set on the PDA (the latter is not that simple if you use PIE and not NetFront; furthermore, with browsers that don’t support proxies at all - for example, the current (0.007) build of Minimo – this is the only way to achieve HTML/image compression). You just navigate to their page (in this example, to http://1.2.3.4 in the example) and enter the URL of the target Web server you want to access on the page. (With more advanced PIE plug-ins, for example, with MultiIE, this can be even easier by using their macroing capabilities – please read the MultiIE macro section in my article on Web browsers (http://winmobiletech.com/072005BrowserRoundup/) for more info.)
Second, they not only compress their contents, but may also convert them to a more PDA-friendly format (convert them to one-column to avoid the need for horizontal scrolling etc.)
Their disadvantages will be explained later when I point out the problems of several Web-based services: for example, the hidden URL, the lack of decent cookie handling etc.
1.2.3 Integrated, op. system-level compression clients running on the PDA
The second category of local services integrates into the Windows Mobile operating system even more than local proxies that require explicit configuration. They just sit in the background and silently compress everything, without user interaction/configuration. As with the category of local proxies, it uses a Web-based compression server, also hidden from user.
Unlike with the case of local proxies, you don't need to make your Web browsers'/the operating system's proxy and/or the mailer clients point to these local services. This is why they are much easier to set up (no need for proxy configuration / mailer reconfiguration) and use.
Their structure is depicted as follows:
http://winmobiletech.com/062005CompressionTester/index_tiedostot/image010.gif
The only solution in this category is being developed by Globility (http://www.globility.co.nz). It has not been released yet, only as a (closed) beta. It is very good and works just great, I can tell you ;) As soon as it becomes final, I will also publish its benchmark results and my opinion on it.
1.2.4 All-in-one solutions
Some (namely, Thunderhawk (http://www.bitstream.com/wireless/)) solutions not only heavily strip the Web content and make it much more PDA-friendly, but also use their own client browser instead of PIE/Netfront/Minimo. This approach has clear advantages:
• Thunderhawk has much better JavaScript and, in most cases, Java support than PIE (please read my browser roundup for th
• Consumes way less bandwidth – the Thunderhawk server fully transforms the contents of the HTML file and also greatly downscales images before sending it to the PDA. (Just a side remark: the latter can also be a pain in the buck expecially on VGA devices.)
• No need for setting up any kind of server/no need for using third-party Web-based services. That is, it's an excellent solution for the technically challenged.
The browser, effectively, works as follows:
http://winmobiletech.com/062005CompressionTester/index_tiedostot/image012.gif
Thunderhawk is a very good solution. It, however, has its share of problems. These are as follows:
- as there is no VGA-optimized version, on VGA devices, the visual experience Thunderhawk delivers is not as good as it could be. For example, as far as images are concerned, it's much inferior to both NetFront and Pocket Internet Explorer in rendering quality.
- it only allows for displaying one Web page at a time; there're no tabs unlike with all PIE plug-ins or NetFront. This makes it very hard to, say, copy information between different web pages (incidentally, there is not even copy functionality!).
- its persistent cookie handling is buggy
- it lacks almost all the features other Web browsers have (copy link addess, select/copy text to clipboard, save image, view source etc)
- it doesn't have a local cache, unlike other PPC-based Web browsers. This is certainly a very bad aspect of Thunderhawk: local caches really help keeping down the used bandwidth. Assuming they aren’t buggy, that is; for example, previous versions of NetFront had cache-related bugs explained at, say, here (http://www.pocketpcthoughts.com/forums/viewtopic.php?t=39674).
- it does strip Web content, but doesn’t use any kind of compression - see the compression benchmarks below.
1.3 Anonymity
You may have guessed that as stand-alone (that is, proxy servers running somewhere on the Internet and not on your PDA) proxy servers have a different Internet address than your PDA and, as they act as the client for the real HTTP servers you're accessing resources at, you can effectively "hide" behind them. Then, the target HTTP server you access will only see the proxy server as the client. This is a very important question for people that want to hide their Internet address (just Google for the word "anonymity" to see how popular a question this is).
Most public proxy servers (and compressing services because they effectively also act as proxy servers in this case), however, send the X-Forwarded-For HTTP header to the HTTP server, which undermines anonymity. This header tells the Web server the real IP address of the originator client. This is why I've paid special attention to checking the anonymity. I've written a simple HTTP server that just resends the headers it receives to the client. The Java source code can be found here (http://winmobiletech.com/062005CompressionTester/ReturnHTTPHeaders.java).
Unfortunately, both Skweezer and Onspeed tell the client Internet address to the HTTP server. Toonel, on the other hand, doesn't - much as it sends the above HTTP header, its value is "unknown".
You can hide your identity even if you keep using Skweezer if you use an anonymous proxy server to access Skweezer (this is, however, impossible with OnSpeed because the latter is itself a proxy). This is possible because Skweezer is just a HTTP server and not a proxy server - that is, even without so-called 'proxy chaining', you can use a real proxy server before accessing Skweezer. Then, the problem with Skweezer's not hiding your identity goes away. A figure depicting this situation follows:
http://winmobiletech.com/062005CompressionTester/index_tiedostot/image014.gif
In this case, the anonymous proxy server the PDA uses by default behaves like a client for the Skweezer-like service; this is why it's the address of the anonymous server (which is entirely different than that of your PDA and can, of course, be located anywhere in the internet) that is passed further to the real Web server (and, again, not that of your PDA).
As most public anonymous proxy servers don't touch the contents flowing through them in any way (they just add/rewrite/remove HTTP headers), the compressed content from the Sweezer-like Web-based crunching services continue their way to the PDA unmodified - that is, still compressed.
The anonymity situation with PDA-based proxy servers is pretty different: you can't chain any non-PDA proxy to toonel/OnSpeed or similar, on-PDA proxy servers. This doesn't cause any problem with PDA-based proxies, however, as they physically use a proxy server somewhere in the Internet. That is, you don't need to be afraid of PDA-based proxy servers that also offer a high degree of anonymity. You may still want to check out whether they send over the X-Forwarded-For HTTP header to the accessed server. For this, please see the table row "Anonymity?" in the tables summarizing my article.
Personal HTTP proxy servers like RabbIT that you need to run at your, say, home or workplace, are a little bit different because they act as clients to the accessed HTTP servers by default; that is, the HTTP server identifies the device running your personal proxy (for example, your work computer, which you don't necessarily want) server as the client. This may be undesirable; this is where proxy chaining comes into picture. I've given detailed instructions on setting up RabbIT to use an external chained proxy server in the section dedicated to RabbIT.
You may want to check out the following pages for public proxy servers and more on anonymity: 1 (http://www.atomintersoft.com/products/alive-proxy/proxy-list/), 2 (http://www.publicproxyservers.com/page1.html) and 3 (http://www.checker.freeproxy.ru/).
These pages also list commercial proxy services that offer full anonymity and sometimes even compression/image downscaling. They don't do content ripping/PDA-specific formatting, though.
2. Benchmarks of Thunderhawk, RabbIT and the two (three) local PDA-based solutions
For this test, I've used two HTML files. One of them, a 45k-long standalone HTML file, only used one row (<p>this is just a test) repeated. This means that there is no strippable content (except for the CR-LF characters at the end of each line) in it. This file is, however, very well compressable: both WinZip and WinRAR produce compressed file archives between 100 and 200 bytes of this file.
The other HTML test file was a snapshot of the Pocket PC Magazine 'Active Topics' (http://pocketpcmag.com/forum/active.asp). It's full of repeated whitespace, font and color declarations, tables and JavaScript, as with most other webpages, so, content and whitespace stripping will work very well with it. WinZip and WinRAR is able to compress it to 23k and 28k, respectively.
The two input files can be found at here (http://www.winmobiletech.com/062005CompressionTester/ppcmag596k.htm) and here (http://www.winmobiletech.com/062005CompressionTester/45kToistuva.html).
The table can be found here (http://winmobiletech.com/062005CompressionTester/1stTable.html). (Sorry, the forum engine doesn’t allow including HTML tables in-line.)
Explanation for the table:
Integration into the op. system?: as has already ben explained, there’re two approaches for implementing PDA-based clients: the separate proxy with all its advantages (definitely cleaner/less CPU-hungry solution) and disadvantages (need for configuring the clients to use them and problems with other built-in clients – on this, see the next test!) and the solution that takes integration to a next step. As the GHO client is still in beta, I don’t really want to comment on its CPU usage.
The two down/upload tests: I’ve measured (via GPRS, on my mobile phone) the bandwidth used during communication.
As can clearly be seen, Thunderhawk doesn't use any compression, just some light content stripping. It really failed the test of the unstrippable 45kbyte-long file - it used twice the bandwith as the original size!
The two Buggy! records are of definite interest with the OnSpeed client. Much as the current build, 3.3.65535, works great with most pages, it certainly fails with large(r) pages or with special pages like that of the page with <p>this is just a test repeated.
None of the browsers routed through OnSpeed (I’ve tested PIE and NetFront 3.2 on WM2003SE and PIE on WM2003) was able to fully download and read the large PPCMag page. PIE on WM2003 was able to read in fully the 45k <p>this is just a test page; the WM2003SE browsers weren’t. Please note that it wasn’t able to read the current Search page (about 1.2 Mbytes in size) and another test “Active Topics” (about 600k) of the PPCMag forum either.
Interestingly, the data was fully transferred to the PDA during loading the PPCMag test page, both according to my mobile phone data counter and to the built-in statistics page, OnSpeed just haven’t been able to deliver it to the browser:
http://winmobiletech.com/062005CompressionTester/OnSpeedClientGoodCompression.gif.png
There must have been some kind of internal timeout in the proxy, or an overflown buffer – I don’t know.
Bottom line: much as 99% of pages will work OK with OnSpeed, there will be cases when it just stalls. You’ll need to browse these pages with disabled OnSpeed. This can indeed be a big annoyance with this client.
Killed op. system process? : unfortunately, Windows CE (even WM2003SE!) has a tendency to kill background processes because it can only run 32 processes at the same time and, even if you don’t start any apps, there’re at least 8 standard processes running (not taken into account poutlook.exe):
http://winmobiletech.com/062005CompressionTester/TypicalProcessesRunning-PL720.bmp.png
(note that MemMaid.exe is not a standard process (it’s only in the process list because it was with MemMaid that I made the screenshot) and neither is CommLoader.exe, which is started by SOTI Pocket Controller (and isn’t needed for Pocket Controller to work via USB) (http://www.pocketpcthoughts.com/forums/viewtopic.php?p=355320); also, you won’t see BTTrayCE.exe on non-BT-enabled devices; furthermore, WLNPWRCTL.exe is specific to the Pocket Loox 720. All the other processes are standard and mostly invoked from HKLM/init, except for poutlook.exe, the background process in charge of downloading new POP3/IMAP mail if you configured yoru accounts to download (and is, therefore, not needed at all if you don’t have POP3/IMAP mailboxes and/or you don’t peridically download them in Messaging), which is started from \Windows\Startup. Note that Today plug-ins, fortunately, aren’t considered a separate process.)
If you start a compression client on your PDA and open more than, say, ten browser instances with one of the PIE plug-ins (MultiIE, Spb Pocket Plus or PIEPlus; note that ftxPBrowser, the fourth option, behaves entirely differently in this respect), your proxy process will most likely be killed, independent of the free RAM size or their CPU and memory usage. That is, even OnSpeed, which only takes some 900-1300 kbytes of RAM (unlike with Toonel, which takes 4-5 Mbytes because of the JVM), will be shut down.
In this test, I’ve tested two (Toonel, OnSpeed) of all the three PDA-based solutions (Toonel, OnSpeed and the GHO client) to see what happens when the operating system kills the given app. With Toonel, the proxy just ceases to exist, but it will still be configured in the Registry; this means the WM2003SE PIE will no longer be able to access Web pages, while the WM2003 PIE will continue accessing it after connecting in again (changes to proxy settings should be followed by a reconnection so that the changes are visible (http://www.firstloox.org//forums/showthread.php?p=28878)), now without a proxy. OnSpeed behaves much better in this respect: when it is shut down by the operating system, it also reconfigures the PDA (including the registry) not to use the proxy server any more. It means even the WM2003SE PIE will continue to access the net (after the automatic (!) reconnection). Nevertheless, you should be aware of the bandwidth / anonymity implications of this behaviour.
Memory consumption? : this speaks for itself. OnSpeed only takes some 900-1300 kbytes. Toonel, on the other hand, takes 4-5 Mbytes, depending on the JVM it runs under.
HTTP/1.1 compliant? :does the (hidden) server use HTTP/1.1 to download stuff from Web servers? This is not (that) important for PDA users; but, it certainly contributes to less overhead and a bit faster operation on the server side. All solutions except OnSpeed support HTTP/1.1. Note that I haven’t tested the HTTP/1.1 compliance of the PDA browser – local client communication. (Incidentally, all PIE's starting with Windows CE 2.11 have used persistent, HTTP/1.1 connections to avoid the need for building up separate TCP/IP connections for fetching resources. Would you have thought that? ;) ).
Image downscaling? : this also speaks for itself. Both OnSpeed and the GHO client support image downscaling to save bandwidth. Toonel doesn’t.
HTTPS?: will you be able to access secure Web servers through the given apps? Unfortunately, both of the two (in this respect) tested local clients fail in this respect. You’ll need to switch down HTTP proxies entirely to be able to access secure pages.
Socks or at least pass-through?: please see the MSN Messenger-related comments at the end of Section 1.2.1.2. You’ll need a Socks-compliant proxy server if you want to use MSN Messenger along with a HTTP proxy server.
3. Web-based compression/content ripper services
Now comes the second category; those of the Web-based, non-proxy-like services.
3.1 Skweezer (http://www.skweezer.net/)
The most important, very good service that also has a free version.
Pros:
• not only supports GZIP HTML body encoding, but also POP3 access with (slight) compression. The Pro version, fortunately, only differs from the free service in that it's ad-free.
• Unlike some of its direct competitors, it always uses the right URL added to the leading 'skweezer.net' URL. Therefore, URL's can even be put in the local favorites and you can also use MultiIE’s macroing capabilities to seamlessly use Skweezer’s services in loading/compressing a given page.
• three kinds of view that you can dynamically switch between: Desktop (it still GZIP compresses the HTML response; otherwise, it's almost the same as with the original), Mobile ("One Column") and text-only (like the Mobile view, but without images).
• much as the service doesn't directly pass server cookies to the client, it preserves them on the proxy during browser restarts. This, however, requires that you create a (free!) user account on the site and log in. The SKWEEZER.AUTH cookie is presistent so you won't need to log in once more.
Cons:
• it doesn't downsample images, unlike some of its competitors
This is indeed a top-notch service. However, as this service is currently the best client-less Internet GZIP compression and content stripping service, image downscaling could be implemented - at least with the Pro version.
I've also listed the URL format it (and MobileLeap & WebWarper) uses to show how easy it is to pass these services a URL without entering it into the field in their homepage. Fortunately, you can easily concatenate any URL to both Skweezer, WebWarper and MobileLeap v2. This is of extreme importance with PIE plug-ins like MultiIE. Please read the MultiIE macro section in my article on Web browsers (http://winmobiletech.com/072005BrowserRoundup/) for more info on the importance of this. With Skweezer, the full URL is as follows:
http://www.skweezer.net/skweeze.aspx?m=2&q=<URL>
3.2 WebWarper (http://webwarper.net/)
Cons:
• at startup, it downloads a 195 kbyte-long JavaScript file ( http://webwarper.net/ww/~WWServiceScriptNoGZ~av/wwserv.js ). This will always happen even in the same web session if you, for example, force a page refresh (or, just re-enter the page address into the WebWarper URL text field). I've tested this with both PIE and NetFront 3.2. This means using this service may actually result in much higher bandwidth usage than with any of its alternatives.
• it has no one-column emulation mode, unlike all its alternatives and not even in text-only mode. This means on pre-WM2003SE devices, you may have problems browsing wide tables (typically, forum pages).
Pros:
• GZIP HTML body compression
• Full server cookie support; stored even over browser restarts
URL format: http://webwarper.net/ww/~av/<URL>; in text-only mode, use http://webwarper.net/ww/~s/ instead of this.
Because of the JavaScript download problem, I do not recommend this service. Skweezer is much better than this service in terms of bandwidth usage.
3.3 clickfisch.net (http://ppcp.clickfisch.net/)
Cons:
• completely messes up URL's after a POST'ed HTML form: most of them won't work (give a try to, say, the PPCThoughts or PPCMagazine forums and try to log in!)
• no login will work
Pros:
• image compression
I can't really recommend this service because of the major problems described above.
Also see this thread (http://www.pocketpcthoughts.com/forums/viewtopic.php?p=345365) on these problems.
3.4 WebXCope from Jenner Research (http://216.103.91.135/rfxDM/ppcframe.php)
Doesn't compress but cuts contents pretty well.
Cookies are real server cookies (unlike with Skweezer) but they aren't persistent. Furthermore, if you want to navigate to different sections in a lot of pages (for example, Pocket PC Thoughts forum), you'll need to click [+] a lot of times, which really goes into nerves. Furthermore, if you have to use this link, all the subsequent pages will uses static HTML (like /rfx/ss1868008.htm) URLs and not real URL's, unlike, say, Skweezer. This means you won't be able to bookmark a lot of pages.
3.5 MobileLeap v2 (http://mobileleap.net/v2/)
(Note that this is the second version of MobileLeap, only recently updated. In the first version of this article (http://winmobiletech.com/062005CompressionTester/v1.htm) published back in June, 2005, I’ve only tested V1. It was far worse than the current version. You can find the review of the old version here (http://www.winmobiletech.com/062005CompressionTester/MobileLeapV1.html) if interested.)
A lot have been fixed/upgaded in this version, compared to the first version and it alreay offers a lot of goodies (anonymity, image downscaling etc – in these two respects, the commercial Skweezer is worse!) for free. It, unfortunately, still has problems:
- It doesn’t use line breaks where it’d be necessary. See for example the way it renders this BrightHand page (http://discussion.brighthand.com/showthread.php?s=&threadid=119807):
http://winmobiletech.com/062005CompressionTester/MobileLeapNoLineBreaks.gif.png
Both WebWarper and Skweezer are far better in this respect. This is how Skweezer renders the list:
http://winmobiletech.com/062005CompressionTester/SkweezerBHPage.gif.png
- It doesn’t really support inter-session cookies
- You can’t directly use the direct URL for direct invocation - for example, via a MultiIE macro, which, incidentally, would be the following: http://mobileleap.net/v2/#^;-Mobileleap. (See my article on Web browsers (http://winmobiletech.com/072005BrowserRoundup/) on the MultiIE macros for WebWarper and Skweezer.)
As, however, this service is constantly developed further, I really think these problems will be fixed very soon. Therefore, I really recommend visiting the MobileLeap homepage often / follow the discussions on it.
3.6 The WML Proxy conversion service of Google (http://www.google.com/xhtml)
Much as this is, officially, only a Google front-end, you can enter the full page URL you want to access into it without the leading http:// . Then, if you are lucky (it doesn't work with a lot of cases!) you'll be able to use this service to access the page without any URL hacking or manual editing (which would be rather compicated with WML Proxy). It doesn't have images and are, at times, because of its being a widely used service, very slow or doesn't even work, though.
Because it didn't work when I finished this article, I couldn't test its capabilities. This is why it's also missing from the comparison chart.
5. Web-based cutting/compression service comparison chart
In addition to the first table/series of tests, I had to introduce some new tests here regarding cookie handling. The reason for this is very easy: proxy-like services pass HTTP headers almost verbatim between the Web server and the browser on the PDA. That is, if a server issues a cookie, the client will receive it; and, if a client sends back a cookie to the Web server, the latter will definitely receive it.
Web-based services don’t pass cookies verbatim between the browser and the Web server. Instead, they keep them themselves and only send their own cookies back to the client. This also means a plethora of cookie-handling related bugs. If you check the “Cookie time test” row in the table below, you’ll see that neither MobileLeap nor Skweezer did process the ‘expire’ attribute in cookies returned by the web server properly, which is a very common error with a lot of Web clients. (Please see, for example, this thread (http://www.pocketpcthoughts.com/forums/viewtopic.php?p=351433) on this subject. Incidentally, I used the Java application used in there to test exact expire time processing).
Generally, however, the cookies in these two apps worked with generic Web servers, just as with WebWarper.
Other tests remained the same as with the strictly proxy-based-tests.
The table can be found here (http://winmobiletech.com/062005CompressionTester/2ndTable.html). (Sorry, the forum engine doesn’t allow including HTML tables in-line.)
6. Bottom line
It's not very hard to say which bandwidth conserving solutions are the best as of now (13/08/2005):
- Thunderhawk, if you can put up with the relatively high price of the service and the problems (QVGA, no tabs, lack of real compression) of the Thunderhawk client
- Toonel, if you don't mind running a personal proxy server on your PDA. I’ve chosen it over OnSpeed because 1, it’s free 2, has mail compression support.
- Skweezer. Please note that I've chosen it over
1, WebWarper: mostly due to the 195k JavaScript download-problem and the lack of a PDA-formatted view that still has images of the latter service. WebWarper, however, offers complete anonymity.
2, MobileLeap v2: because of the new line-rendering bug, the lack of inter-session cookie support and the inability to directly navigate to URL’s
There're two problems with most web sites.
First: particularly if you enable images, navigating to desktop-optimized pages may result in getting HTML pages and images totalling even hundreds of kilobytes. Mostly, you won’t be able to make use of high resolution, high quality images on a PDA (unless you want to save them/zoom into them); that is, in most cases, downloading desktop-optimized pages to a PDA is an overkill. The same can be said about HTML pages thsemselves; they are, in general, textual, and, therefore, highly compressible: text can be compressed with plain compression tools very good (JPEG’s and other Web image types are a different story – more on their compression later).
Second: being HTML, standard (non-PDA-optimized) Web pages tend to are also full of unnecessary stuff: whitespaces, scripts and formatting that are completely useless for a lot of PDA users that prefer viewing pages in the so-called “One Column” view mode of the Pocket Internet Explorer (PIE) in WM2003SE+ (or, if you use an older PIE version, similar rendering modes of the ftxPBrowser and the PIEPlus PIE plug-ins; or, if you use NetFront 3.2, the Smart-Fit Rendering mode). (Please see the “Built-in one-column view on WM2003 devices” section in my article on Web browsers (http://winmobiletech.com/072005BrowserRoundup/)) on the One Column view mode and its alternatives.) Then, almost all <table>, <frame> and the related tags become useless. This means a service that strips all these additional whitespaces, tables and converts everything in one column may also be of advantage.
In this article, I scrutinize both subjects and the services/applications meant for Pocket PC users.
1. What is this all about? How does content cleanup and compression work?
1.1 Content cleanup
As has alredy been mentioned, the majority of web pages contain a lot of unnecessary whitespaces. Web browsers simply ignore them - for example, if you insert ten space characters one after another, the Web browser will just display one instead of ten. (This is why   has been introduced BTW.) Also, Web browsers just ignore CR/LF characters unless they are in <PRE> blocks - that is, they can also be safely removed to save bandwidth usage.
Also, there're a lot of additional constructs that aren't necessarily needed in a "simple", "dumb" browser like a Pocket PC-based one. For example, objects, plug-ins and, in a lot of cases, stylesheets are not needed and can safely be omitted. These also help a lot in reducing bandwidth usage.
Please note that not all HTML files can be content stripped. For example, HTML files that don't contain special tags or excess whitespace can't be "compressed" this way. For example, a page that is already PDA-compliant (because it has no tables/frames) can’t be content stripped. You'll even see an example of these HTML pages later, in the "How did I test? " section.
1.2 Compression
You can not only strip unnecessary textual information (table/frame tags, whitespace etc.) from a HTML file before sending it to the client, but also compress it with a compressor technology that is compatible with the client browser. The most common compressor technology is gzip. Note that the HTTP protocol, the protocol that is used to request and return Web resources, allows for other compression techniques too (for example, compress), but they are much more rarely used in lightweight clients like PDA-based browsers.
Gzip can achieve much better compression ratios than the above-described content stripping/whitespace eliminating algorithm. For example, as you'll see in the benchmark session, an ordinary test page that is originally 582 kbytes, with content/whitespace stripping can be "compressed" to around 200 kbytes; with gzip compression, however, to 20 kbytes, which is an order of magnitude better compression rate than with simple content/whitespace stripping.
As has already been pointed out, HTTP, the protocol in charge of sending page and resource requests and returning pages and images, allows for gzip compression. However, almost none of the current Web servers actually do return compressed pages to save CPU processing. (It takes much more CPU resources to compress a page before returning it to the client than the bandwidth overhead of the uncompressed page.) That is, in the easiest setup (a simple browser is communicating with a Web server, as depicted in the following figure), the server will almost always send back an uncompressed response:
http://winmobiletech.com/062005CompressionTester/index_tiedostot/image002.gif
Here, along the Internet connection (the arrow in this figure), only uncompressed data flows. (From now on, I use solid arrows to denote uncompressed and dotted arrows to denote compressed data flow to avoid making the figures hard-to-read by adding excessive comments.)
This is where so-called proxies come into picture. What are they?
1.2.1 HTTP Proxies in/outside the PDA requiring explicit PDA-side configuration
These two solutions are the same in that they require explicit HTTP proxy configuration on the PDA, unlike the other solutions.
Configuring a HTTP proxy on the PDA can/should be done in one of two ways. If your particular browser (right now, NetFront only; Minimo may support this in the future) supports local proxy configuration, then, you just enter its address right into the browser. If you, however, plan to use PIE, you end up using the global, system-level proxy configuration tools.
Both of these cases (with NetFront as the example of the only PPC-based Web browser capable of using a local proxy configuration) are explained here (http://www.pocketpcmag.com/forum/topic.asp?TOPIC_ID=16017); the reader is referred to the tutorial there.
Incidentally, if you end up configuring the system-level proxy because you want to stick to PIE, you won’t be able to use the built-in MSN Messenger unless you use a so-called “Socks” proxy. Freely usable Socks proxies are pretty common, you can find a list of them at, say, here (http://www.aliveproxy.com/socks5-list/). Note that I’ve tested some local, Java-language, free Socks proxies (for example, jSocks (http://jsocks.sourceforge.net)), but I couldn’t make the Pocket PC MSN Messenger communicate through them, unlike with ‘real’ Socks servers found on the Internet.
1.2.1.1 HTTP Proxies running outside the PDA
Proxies are servers that act as an intermediary server for the browser. They receive HTML page and related resource (for example, linked images) requests from the browser and send them to the real Web server as if they (the proxies) were the clients requesting these resources. They receive the response from the Web server and send it over to their client, that is, the originating browser.
They (the proxy servers) do not need to send the response to the client browser verbatim, without any transformation. If they notice that the client is, for example, a PDA-based Web browser, they may choose to remove not needed HTML markup from HTML pages (that is, they may optimize the HTML) and/or compress it with gzip. They can do something similar to images, especially JPEG images: they may opt for downscaling them to save bandwidth.
How can I tell a PDA Web browser what proxy to use? - you may ask. Proxy server addresses must be told to browsers in their respective proxy setting dialog. An exception is PIE in the Pocket PC 2002 operating system and higher: in recent Pocket PC/Windows Mobile operating systems, the HTTP proxy settings are separated from PIE. In alternative browsers like NetFront, however, they can be set straight in them (note that if you use the “Auto-retrieve” option in NetFront, you can also use the system-level proxy setting). I've explained setting a proxy server in both PIE (that is, the operating system) and Netfront 3.1/3.2 here (http://www.pocketpcmag.com/forum/topic.asp?TOPIC_ID=16017).
With proxy servers coming into picture, the previous two-tier model (a browser and a Web server directly communicates) effectively becomes three-tier, as in the following figure. Please note that in this figure I've used a dotted arrow to denote compressed responses from the proxy server to the client.
http://winmobiletech.com/062005CompressionTester/index_tiedostot/image004.gif
What does this image convey? It's pretty simple: if the proxy server is running somewhere on the Internet and not on the PDA, then, because the (probably very slow and expensive) Internet connection of the client is already in the "compressed" zone (that is, where only compressed data is transferred - the thick, vertical line in the figure denotes this), you'll save up quite much money/time on using compression services!
There're quite a few public, free proxies on the Internet that you can use. A list of pages containing free proxy addresses will follow in the Anonymity section. (Almost) none of them support compression by default, though, so, we need to keep searching for other alternatives. The above introduction was still needed to draw a generic picture of the simplest case so that more advanced architectures can be understood.
Because of the lack of free, but compressing public servers, you may opt for deploying compressing proxy servers on your own, say, home desktop PC or, if it isn't in a heavily-defended (firewalls, other restrictions) environment, even on your work PC. These proxy servers are different from both third-party proxy servers, Web-based compression/content stripping services and on-PDA clients in that they must be run on one of your desktop PC's. This means they're only useful for people that have a PC on the Net with sufficiently large Internet bandwidth. If you aren't one of them, you may want to forget this option entirely. If you, on the other hand, do have one, these proxy servers are for you because they have several advantages over other solutions:
1, as it's your own server without other users, it will be much faster than other, mostly heavily overloaded services (including PDA-based proxy clients), assuming the desktop machine is running on a fast Internet connection
2, you can freely configure them
3, they are not only able to do basic stuff (content compression, image downscaling), but also goodies like ad filtering.
One of the best personal, free proxy servers is RabbIT (http://www.khelekore.org/rabbit/). It is a great tool because it supports both gzipping, HTTP/1.1 (just like PIE), image downscaling, proxy chaining (in order to remain anonymous) and all the goodies you can think of. It's Java (so, you'll need a JVM to run it) but uses the external, C-based ImageMagick (http://imagemagick.org/script/index.php) for super-fast image conversion.
Please read this article (http://www.winmobiletech.com/062005CompressionTester/UsingRabbIT.html) on setting up and running RabbIT if interested.
If you don't have the resources (a preferably 24/7 desktop PC to run your proxy server) or are afraid of hackers and unwanted guests using it (the latter is not really an issue because, for example, RabbIT has quite sophisticated autentication and client filtering capabilities), another option is using a third-party compressing proxy servers.
1.2.1.2 HTTP Proxies running inside the PDA
There is another category with services like toonel (http://www.toonel.net/downloads.html) and the OnSpeed (http://www.onspeed.com/us/index.php) client. They run locally on the PDA (so far, we only used proxy servers and/or third-party compressing servers on the Web somewhere, but definitely not on the same PDA) and are even more useful because
1, as far as Web-browsing (that is, the HTTP protocol) is concerned, they not only compress incoming textual HTML pages but also everything - outgoing requests, JavaScript, CSS pages (they are generally not compressed, not even by Web-based servers), everything.
2, they may (toonel supports this, OnSpeed, unfortunately, doesn’t) even go further by allowing for compressing SMTP and POP3, the two most important mail sending and receiving protocols. This is a big thing because, as per the standards defining these protocols (along with the MIME standard), they don't allow for any kind of compression (as opposed to HTTP). This is why it's only with locally (on the PDA in this case) running clients that you can use any kind of (decent; that is, much better than the simple Run-Length Encoding of V42.bis, the protocol widely used in modem-based communication, including GPRS) compression - because the TCP/IP protocol being used (POP3 / SMTP) officially doesn't allow for any kind of compression.
Local proxy-like clients, as they are behind the internet connection of the PDA (all programs running on the PDA share the same, with mobile phone-based connections, slow and expensive connection), they need another (hidden; meaning, the PDA user doesn't need to know its address) server somewhere in the internet to connect to. This server will compress the incoming (POP3/)HTTP responses and the outgoing ((POP3/)SMTP/)HTTP requests as is depicted in the following figure:
http://winmobiletech.com/062005CompressionTester/index_tiedostot/image008.gif
Please note that the mailer client communicates with the compresser client uncompressed (as per the POP3/SMTP standards); it's only between the compresser client and server that communication is compressed. This causes no runtime/bandwidth problems: as with the previous case, over the PDA-Internet connection, only compressed content is sent. It should also be pointed out, again, that in this case, the bandwidth save ratio may even be better than in the previous one because, with an external HTTP proxy, it’s only HTTP responses that can be compressed, HTTP requests, POP3 and SMTP can’t be.
This solution results in the most bandwidth saving (not only HTTP inbound, but also SMTP outbound, POP3 inbound etc), but it definitely eats up some of the (on a PDA, quite rare and meager) system resources. Advanced compressing clients like toonel and OnSpeed, however, don't deliver a considerable runtime hit to the device.
Setting up a local client on your PDA is easier than you think. As far as the Java-based toonel is concerned, you may want to read this tutorial (http://www.pocketpcmag.com/forum/topic.asp?TOPIC_ID=16017) on this subject. Setting up the OnSpeed client is even easier – you just fire its installer and fill in your OnSpeed login/password after installing.
As toonel/OnSpeed don't integrate into the core operating system unlike some of their alternatives (namely, the Globility client, which, being still in beta tests, will only be discussed in a later version of this article), you will need to manually configure the operating system / the browsers (with both clients) / the mailer client(s) (with toonel only) to use them. This is why you need to explicitly configure a proxy server on the PDA and fool the mailer client to believe the SMTP/POP3 server being at localhost. It isn't particularly complicated, though, and is thoroughly explained in the above link.
Should you want to choose your app from this category, which of the two services/clients to go for?
Both have advantages and disadvantages.
Toonel is Java-based. This means you’ll need to get a Java Virtual Machine (JVM) in order to run it. Some JVM’s are very cheap (for example, the IBM J9, Personal Profile 1.0 (http://www.handango.com/PlatformProductDetail.jsp?productType=2&optionId=1_2_2&platformId=2&siteId=1&productId=140997&programId=458&affiliateId=1218)) and will run Toonel just great. Add the price of Toonel (free!) and the overall reliability of the Toonel service, the ability to compress e-mail communication (unlike with OnSpeed), being highly anonymous and HTTP/1.1 compliant (unlike, again, OnSpeed), and you have a real killer. Hopefully, Toonel continues to be free and having great bandwidth, even from Europe. Should you need help with setting up Toonel, please read the above link. Also, you may want to read this article (http://www.pocketpcthoughts.com/forums/viewtopic.php?t=41081) on the available JVM’s. Its only real disadvantage is the lack of image downscaling.
OnSpeed is a different animal. It costs comparatively quite much (45 US$ a year), puts more burden on the connected HTTP server as it doesn’t support HTTP/1.1, isn’t highly anonymous and only offers HTTP compression (while Toonel also has e-mail compression). It, however, has configurable image downscaling.
Please note that some OnSpeed users have been faced with tremendous slowdowns (http://www.pocketpcthoughts.com/forums/viewtopic.php?t=40986). I haven’t noticed anything like that (have also tested everything over Wi-Fi with DHCP). They may have used an earlier client version – I don’t know.
Neither of these proxies support HTTPS. This means pages like this (https://internet-banking.dbs.com.sg/) won’t work at all, unlike with some “real”, outside-the-PDA proxies (not with RabbIT though!).
All in all, as of now (14-Aug-2005), I recommend Toonel over OnSpeed.
1.2.2 HTTP Proxy-like Web-based compression/content stripping services requiring no PDA-side proxy configuration
There're also Web-based services that do essentially the same as in/outside-the-PDA proxies explained above. This means you won't need to set a proxy server on your Pocket PC, but will need to navigate into a proxy-like web page with your browser and enter the URL of the target Web server that you want to access there. Content stripping and compression will be done by the middle-layer, proxy-like Web server and you will only receive well-compressed pages (that is, if you choose the right service that does this - for example, Skweezer, WebWarper or MobileLeap v2 – their list will come later). They also save a lot of bandwidth because there're already-compressed HTML pages transferred over the direct connection of the PDA. There're quite a few of services like these; the majority of services I've tested in this article are in this category.
The infrastructure can be depicted as follows:
http://winmobiletech.com/062005CompressionTester/index_tiedostot/image006.gif
It has clear advantages over the pure-proxy solution. First, they are much easier to use for the technologically changelled because no proxy options need to be set on the PDA (the latter is not that simple if you use PIE and not NetFront; furthermore, with browsers that don’t support proxies at all - for example, the current (0.007) build of Minimo – this is the only way to achieve HTML/image compression). You just navigate to their page (in this example, to http://1.2.3.4 in the example) and enter the URL of the target Web server you want to access on the page. (With more advanced PIE plug-ins, for example, with MultiIE, this can be even easier by using their macroing capabilities – please read the MultiIE macro section in my article on Web browsers (http://winmobiletech.com/072005BrowserRoundup/) for more info.)
Second, they not only compress their contents, but may also convert them to a more PDA-friendly format (convert them to one-column to avoid the need for horizontal scrolling etc.)
Their disadvantages will be explained later when I point out the problems of several Web-based services: for example, the hidden URL, the lack of decent cookie handling etc.
1.2.3 Integrated, op. system-level compression clients running on the PDA
The second category of local services integrates into the Windows Mobile operating system even more than local proxies that require explicit configuration. They just sit in the background and silently compress everything, without user interaction/configuration. As with the category of local proxies, it uses a Web-based compression server, also hidden from user.
Unlike with the case of local proxies, you don't need to make your Web browsers'/the operating system's proxy and/or the mailer clients point to these local services. This is why they are much easier to set up (no need for proxy configuration / mailer reconfiguration) and use.
Their structure is depicted as follows:
http://winmobiletech.com/062005CompressionTester/index_tiedostot/image010.gif
The only solution in this category is being developed by Globility (http://www.globility.co.nz). It has not been released yet, only as a (closed) beta. It is very good and works just great, I can tell you ;) As soon as it becomes final, I will also publish its benchmark results and my opinion on it.
1.2.4 All-in-one solutions
Some (namely, Thunderhawk (http://www.bitstream.com/wireless/)) solutions not only heavily strip the Web content and make it much more PDA-friendly, but also use their own client browser instead of PIE/Netfront/Minimo. This approach has clear advantages:
• Thunderhawk has much better JavaScript and, in most cases, Java support than PIE (please read my browser roundup for th
• Consumes way less bandwidth – the Thunderhawk server fully transforms the contents of the HTML file and also greatly downscales images before sending it to the PDA. (Just a side remark: the latter can also be a pain in the buck expecially on VGA devices.)
• No need for setting up any kind of server/no need for using third-party Web-based services. That is, it's an excellent solution for the technically challenged.
The browser, effectively, works as follows:
http://winmobiletech.com/062005CompressionTester/index_tiedostot/image012.gif
Thunderhawk is a very good solution. It, however, has its share of problems. These are as follows:
- as there is no VGA-optimized version, on VGA devices, the visual experience Thunderhawk delivers is not as good as it could be. For example, as far as images are concerned, it's much inferior to both NetFront and Pocket Internet Explorer in rendering quality.
- it only allows for displaying one Web page at a time; there're no tabs unlike with all PIE plug-ins or NetFront. This makes it very hard to, say, copy information between different web pages (incidentally, there is not even copy functionality!).
- its persistent cookie handling is buggy
- it lacks almost all the features other Web browsers have (copy link addess, select/copy text to clipboard, save image, view source etc)
- it doesn't have a local cache, unlike other PPC-based Web browsers. This is certainly a very bad aspect of Thunderhawk: local caches really help keeping down the used bandwidth. Assuming they aren’t buggy, that is; for example, previous versions of NetFront had cache-related bugs explained at, say, here (http://www.pocketpcthoughts.com/forums/viewtopic.php?t=39674).
- it does strip Web content, but doesn’t use any kind of compression - see the compression benchmarks below.
1.3 Anonymity
You may have guessed that as stand-alone (that is, proxy servers running somewhere on the Internet and not on your PDA) proxy servers have a different Internet address than your PDA and, as they act as the client for the real HTTP servers you're accessing resources at, you can effectively "hide" behind them. Then, the target HTTP server you access will only see the proxy server as the client. This is a very important question for people that want to hide their Internet address (just Google for the word "anonymity" to see how popular a question this is).
Most public proxy servers (and compressing services because they effectively also act as proxy servers in this case), however, send the X-Forwarded-For HTTP header to the HTTP server, which undermines anonymity. This header tells the Web server the real IP address of the originator client. This is why I've paid special attention to checking the anonymity. I've written a simple HTTP server that just resends the headers it receives to the client. The Java source code can be found here (http://winmobiletech.com/062005CompressionTester/ReturnHTTPHeaders.java).
Unfortunately, both Skweezer and Onspeed tell the client Internet address to the HTTP server. Toonel, on the other hand, doesn't - much as it sends the above HTTP header, its value is "unknown".
You can hide your identity even if you keep using Skweezer if you use an anonymous proxy server to access Skweezer (this is, however, impossible with OnSpeed because the latter is itself a proxy). This is possible because Skweezer is just a HTTP server and not a proxy server - that is, even without so-called 'proxy chaining', you can use a real proxy server before accessing Skweezer. Then, the problem with Skweezer's not hiding your identity goes away. A figure depicting this situation follows:
http://winmobiletech.com/062005CompressionTester/index_tiedostot/image014.gif
In this case, the anonymous proxy server the PDA uses by default behaves like a client for the Skweezer-like service; this is why it's the address of the anonymous server (which is entirely different than that of your PDA and can, of course, be located anywhere in the internet) that is passed further to the real Web server (and, again, not that of your PDA).
As most public anonymous proxy servers don't touch the contents flowing through them in any way (they just add/rewrite/remove HTTP headers), the compressed content from the Sweezer-like Web-based crunching services continue their way to the PDA unmodified - that is, still compressed.
The anonymity situation with PDA-based proxy servers is pretty different: you can't chain any non-PDA proxy to toonel/OnSpeed or similar, on-PDA proxy servers. This doesn't cause any problem with PDA-based proxies, however, as they physically use a proxy server somewhere in the Internet. That is, you don't need to be afraid of PDA-based proxy servers that also offer a high degree of anonymity. You may still want to check out whether they send over the X-Forwarded-For HTTP header to the accessed server. For this, please see the table row "Anonymity?" in the tables summarizing my article.
Personal HTTP proxy servers like RabbIT that you need to run at your, say, home or workplace, are a little bit different because they act as clients to the accessed HTTP servers by default; that is, the HTTP server identifies the device running your personal proxy (for example, your work computer, which you don't necessarily want) server as the client. This may be undesirable; this is where proxy chaining comes into picture. I've given detailed instructions on setting up RabbIT to use an external chained proxy server in the section dedicated to RabbIT.
You may want to check out the following pages for public proxy servers and more on anonymity: 1 (http://www.atomintersoft.com/products/alive-proxy/proxy-list/), 2 (http://www.publicproxyservers.com/page1.html) and 3 (http://www.checker.freeproxy.ru/).
These pages also list commercial proxy services that offer full anonymity and sometimes even compression/image downscaling. They don't do content ripping/PDA-specific formatting, though.
2. Benchmarks of Thunderhawk, RabbIT and the two (three) local PDA-based solutions
For this test, I've used two HTML files. One of them, a 45k-long standalone HTML file, only used one row (<p>this is just a test) repeated. This means that there is no strippable content (except for the CR-LF characters at the end of each line) in it. This file is, however, very well compressable: both WinZip and WinRAR produce compressed file archives between 100 and 200 bytes of this file.
The other HTML test file was a snapshot of the Pocket PC Magazine 'Active Topics' (http://pocketpcmag.com/forum/active.asp). It's full of repeated whitespace, font and color declarations, tables and JavaScript, as with most other webpages, so, content and whitespace stripping will work very well with it. WinZip and WinRAR is able to compress it to 23k and 28k, respectively.
The two input files can be found at here (http://www.winmobiletech.com/062005CompressionTester/ppcmag596k.htm) and here (http://www.winmobiletech.com/062005CompressionTester/45kToistuva.html).
The table can be found here (http://winmobiletech.com/062005CompressionTester/1stTable.html). (Sorry, the forum engine doesn’t allow including HTML tables in-line.)
Explanation for the table:
Integration into the op. system?: as has already ben explained, there’re two approaches for implementing PDA-based clients: the separate proxy with all its advantages (definitely cleaner/less CPU-hungry solution) and disadvantages (need for configuring the clients to use them and problems with other built-in clients – on this, see the next test!) and the solution that takes integration to a next step. As the GHO client is still in beta, I don’t really want to comment on its CPU usage.
The two down/upload tests: I’ve measured (via GPRS, on my mobile phone) the bandwidth used during communication.
As can clearly be seen, Thunderhawk doesn't use any compression, just some light content stripping. It really failed the test of the unstrippable 45kbyte-long file - it used twice the bandwith as the original size!
The two Buggy! records are of definite interest with the OnSpeed client. Much as the current build, 3.3.65535, works great with most pages, it certainly fails with large(r) pages or with special pages like that of the page with <p>this is just a test repeated.
None of the browsers routed through OnSpeed (I’ve tested PIE and NetFront 3.2 on WM2003SE and PIE on WM2003) was able to fully download and read the large PPCMag page. PIE on WM2003 was able to read in fully the 45k <p>this is just a test page; the WM2003SE browsers weren’t. Please note that it wasn’t able to read the current Search page (about 1.2 Mbytes in size) and another test “Active Topics” (about 600k) of the PPCMag forum either.
Interestingly, the data was fully transferred to the PDA during loading the PPCMag test page, both according to my mobile phone data counter and to the built-in statistics page, OnSpeed just haven’t been able to deliver it to the browser:
http://winmobiletech.com/062005CompressionTester/OnSpeedClientGoodCompression.gif.png
There must have been some kind of internal timeout in the proxy, or an overflown buffer – I don’t know.
Bottom line: much as 99% of pages will work OK with OnSpeed, there will be cases when it just stalls. You’ll need to browse these pages with disabled OnSpeed. This can indeed be a big annoyance with this client.
Killed op. system process? : unfortunately, Windows CE (even WM2003SE!) has a tendency to kill background processes because it can only run 32 processes at the same time and, even if you don’t start any apps, there’re at least 8 standard processes running (not taken into account poutlook.exe):
http://winmobiletech.com/062005CompressionTester/TypicalProcessesRunning-PL720.bmp.png
(note that MemMaid.exe is not a standard process (it’s only in the process list because it was with MemMaid that I made the screenshot) and neither is CommLoader.exe, which is started by SOTI Pocket Controller (and isn’t needed for Pocket Controller to work via USB) (http://www.pocketpcthoughts.com/forums/viewtopic.php?p=355320); also, you won’t see BTTrayCE.exe on non-BT-enabled devices; furthermore, WLNPWRCTL.exe is specific to the Pocket Loox 720. All the other processes are standard and mostly invoked from HKLM/init, except for poutlook.exe, the background process in charge of downloading new POP3/IMAP mail if you configured yoru accounts to download (and is, therefore, not needed at all if you don’t have POP3/IMAP mailboxes and/or you don’t peridically download them in Messaging), which is started from \Windows\Startup. Note that Today plug-ins, fortunately, aren’t considered a separate process.)
If you start a compression client on your PDA and open more than, say, ten browser instances with one of the PIE plug-ins (MultiIE, Spb Pocket Plus or PIEPlus; note that ftxPBrowser, the fourth option, behaves entirely differently in this respect), your proxy process will most likely be killed, independent of the free RAM size or their CPU and memory usage. That is, even OnSpeed, which only takes some 900-1300 kbytes of RAM (unlike with Toonel, which takes 4-5 Mbytes because of the JVM), will be shut down.
In this test, I’ve tested two (Toonel, OnSpeed) of all the three PDA-based solutions (Toonel, OnSpeed and the GHO client) to see what happens when the operating system kills the given app. With Toonel, the proxy just ceases to exist, but it will still be configured in the Registry; this means the WM2003SE PIE will no longer be able to access Web pages, while the WM2003 PIE will continue accessing it after connecting in again (changes to proxy settings should be followed by a reconnection so that the changes are visible (http://www.firstloox.org//forums/showthread.php?p=28878)), now without a proxy. OnSpeed behaves much better in this respect: when it is shut down by the operating system, it also reconfigures the PDA (including the registry) not to use the proxy server any more. It means even the WM2003SE PIE will continue to access the net (after the automatic (!) reconnection). Nevertheless, you should be aware of the bandwidth / anonymity implications of this behaviour.
Memory consumption? : this speaks for itself. OnSpeed only takes some 900-1300 kbytes. Toonel, on the other hand, takes 4-5 Mbytes, depending on the JVM it runs under.
HTTP/1.1 compliant? :does the (hidden) server use HTTP/1.1 to download stuff from Web servers? This is not (that) important for PDA users; but, it certainly contributes to less overhead and a bit faster operation on the server side. All solutions except OnSpeed support HTTP/1.1. Note that I haven’t tested the HTTP/1.1 compliance of the PDA browser – local client communication. (Incidentally, all PIE's starting with Windows CE 2.11 have used persistent, HTTP/1.1 connections to avoid the need for building up separate TCP/IP connections for fetching resources. Would you have thought that? ;) ).
Image downscaling? : this also speaks for itself. Both OnSpeed and the GHO client support image downscaling to save bandwidth. Toonel doesn’t.
HTTPS?: will you be able to access secure Web servers through the given apps? Unfortunately, both of the two (in this respect) tested local clients fail in this respect. You’ll need to switch down HTTP proxies entirely to be able to access secure pages.
Socks or at least pass-through?: please see the MSN Messenger-related comments at the end of Section 1.2.1.2. You’ll need a Socks-compliant proxy server if you want to use MSN Messenger along with a HTTP proxy server.
3. Web-based compression/content ripper services
Now comes the second category; those of the Web-based, non-proxy-like services.
3.1 Skweezer (http://www.skweezer.net/)
The most important, very good service that also has a free version.
Pros:
• not only supports GZIP HTML body encoding, but also POP3 access with (slight) compression. The Pro version, fortunately, only differs from the free service in that it's ad-free.
• Unlike some of its direct competitors, it always uses the right URL added to the leading 'skweezer.net' URL. Therefore, URL's can even be put in the local favorites and you can also use MultiIE’s macroing capabilities to seamlessly use Skweezer’s services in loading/compressing a given page.
• three kinds of view that you can dynamically switch between: Desktop (it still GZIP compresses the HTML response; otherwise, it's almost the same as with the original), Mobile ("One Column") and text-only (like the Mobile view, but without images).
• much as the service doesn't directly pass server cookies to the client, it preserves them on the proxy during browser restarts. This, however, requires that you create a (free!) user account on the site and log in. The SKWEEZER.AUTH cookie is presistent so you won't need to log in once more.
Cons:
• it doesn't downsample images, unlike some of its competitors
This is indeed a top-notch service. However, as this service is currently the best client-less Internet GZIP compression and content stripping service, image downscaling could be implemented - at least with the Pro version.
I've also listed the URL format it (and MobileLeap & WebWarper) uses to show how easy it is to pass these services a URL without entering it into the field in their homepage. Fortunately, you can easily concatenate any URL to both Skweezer, WebWarper and MobileLeap v2. This is of extreme importance with PIE plug-ins like MultiIE. Please read the MultiIE macro section in my article on Web browsers (http://winmobiletech.com/072005BrowserRoundup/) for more info on the importance of this. With Skweezer, the full URL is as follows:
http://www.skweezer.net/skweeze.aspx?m=2&q=<URL>
3.2 WebWarper (http://webwarper.net/)
Cons:
• at startup, it downloads a 195 kbyte-long JavaScript file ( http://webwarper.net/ww/~WWServiceScriptNoGZ~av/wwserv.js ). This will always happen even in the same web session if you, for example, force a page refresh (or, just re-enter the page address into the WebWarper URL text field). I've tested this with both PIE and NetFront 3.2. This means using this service may actually result in much higher bandwidth usage than with any of its alternatives.
• it has no one-column emulation mode, unlike all its alternatives and not even in text-only mode. This means on pre-WM2003SE devices, you may have problems browsing wide tables (typically, forum pages).
Pros:
• GZIP HTML body compression
• Full server cookie support; stored even over browser restarts
URL format: http://webwarper.net/ww/~av/<URL>; in text-only mode, use http://webwarper.net/ww/~s/ instead of this.
Because of the JavaScript download problem, I do not recommend this service. Skweezer is much better than this service in terms of bandwidth usage.
3.3 clickfisch.net (http://ppcp.clickfisch.net/)
Cons:
• completely messes up URL's after a POST'ed HTML form: most of them won't work (give a try to, say, the PPCThoughts or PPCMagazine forums and try to log in!)
• no login will work
Pros:
• image compression
I can't really recommend this service because of the major problems described above.
Also see this thread (http://www.pocketpcthoughts.com/forums/viewtopic.php?p=345365) on these problems.
3.4 WebXCope from Jenner Research (http://216.103.91.135/rfxDM/ppcframe.php)
Doesn't compress but cuts contents pretty well.
Cookies are real server cookies (unlike with Skweezer) but they aren't persistent. Furthermore, if you want to navigate to different sections in a lot of pages (for example, Pocket PC Thoughts forum), you'll need to click [+] a lot of times, which really goes into nerves. Furthermore, if you have to use this link, all the subsequent pages will uses static HTML (like /rfx/ss1868008.htm) URLs and not real URL's, unlike, say, Skweezer. This means you won't be able to bookmark a lot of pages.
3.5 MobileLeap v2 (http://mobileleap.net/v2/)
(Note that this is the second version of MobileLeap, only recently updated. In the first version of this article (http://winmobiletech.com/062005CompressionTester/v1.htm) published back in June, 2005, I’ve only tested V1. It was far worse than the current version. You can find the review of the old version here (http://www.winmobiletech.com/062005CompressionTester/MobileLeapV1.html) if interested.)
A lot have been fixed/upgaded in this version, compared to the first version and it alreay offers a lot of goodies (anonymity, image downscaling etc – in these two respects, the commercial Skweezer is worse!) for free. It, unfortunately, still has problems:
- It doesn’t use line breaks where it’d be necessary. See for example the way it renders this BrightHand page (http://discussion.brighthand.com/showthread.php?s=&threadid=119807):
http://winmobiletech.com/062005CompressionTester/MobileLeapNoLineBreaks.gif.png
Both WebWarper and Skweezer are far better in this respect. This is how Skweezer renders the list:
http://winmobiletech.com/062005CompressionTester/SkweezerBHPage.gif.png
- It doesn’t really support inter-session cookies
- You can’t directly use the direct URL for direct invocation - for example, via a MultiIE macro, which, incidentally, would be the following: http://mobileleap.net/v2/#^;-Mobileleap. (See my article on Web browsers (http://winmobiletech.com/072005BrowserRoundup/) on the MultiIE macros for WebWarper and Skweezer.)
As, however, this service is constantly developed further, I really think these problems will be fixed very soon. Therefore, I really recommend visiting the MobileLeap homepage often / follow the discussions on it.
3.6 The WML Proxy conversion service of Google (http://www.google.com/xhtml)
Much as this is, officially, only a Google front-end, you can enter the full page URL you want to access into it without the leading http:// . Then, if you are lucky (it doesn't work with a lot of cases!) you'll be able to use this service to access the page without any URL hacking or manual editing (which would be rather compicated with WML Proxy). It doesn't have images and are, at times, because of its being a widely used service, very slow or doesn't even work, though.
Because it didn't work when I finished this article, I couldn't test its capabilities. This is why it's also missing from the comparison chart.
5. Web-based cutting/compression service comparison chart
In addition to the first table/series of tests, I had to introduce some new tests here regarding cookie handling. The reason for this is very easy: proxy-like services pass HTTP headers almost verbatim between the Web server and the browser on the PDA. That is, if a server issues a cookie, the client will receive it; and, if a client sends back a cookie to the Web server, the latter will definitely receive it.
Web-based services don’t pass cookies verbatim between the browser and the Web server. Instead, they keep them themselves and only send their own cookies back to the client. This also means a plethora of cookie-handling related bugs. If you check the “Cookie time test” row in the table below, you’ll see that neither MobileLeap nor Skweezer did process the ‘expire’ attribute in cookies returned by the web server properly, which is a very common error with a lot of Web clients. (Please see, for example, this thread (http://www.pocketpcthoughts.com/forums/viewtopic.php?p=351433) on this subject. Incidentally, I used the Java application used in there to test exact expire time processing).
Generally, however, the cookies in these two apps worked with generic Web servers, just as with WebWarper.
Other tests remained the same as with the strictly proxy-based-tests.
The table can be found here (http://winmobiletech.com/062005CompressionTester/2ndTable.html). (Sorry, the forum engine doesn’t allow including HTML tables in-line.)
6. Bottom line
It's not very hard to say which bandwidth conserving solutions are the best as of now (13/08/2005):
- Thunderhawk, if you can put up with the relatively high price of the service and the problems (QVGA, no tabs, lack of real compression) of the Thunderhawk client
- Toonel, if you don't mind running a personal proxy server on your PDA. I’ve chosen it over OnSpeed because 1, it’s free 2, has mail compression support.
- Skweezer. Please note that I've chosen it over
1, WebWarper: mostly due to the 195k JavaScript download-problem and the lack of a PDA-formatted view that still has images of the latter service. WebWarper, however, offers complete anonymity.
2, MobileLeap v2: because of the new line-rendering bug, the lack of inter-session cookie support and the inability to directly navigate to URL’s