Kris Kumar
03-26-2007, 05:30 PM
AURA stands for Advanced User Resource Annotation. Behind the fancy name is an amazing technology that enables the Windows Mobile devices to read product barcodes and pull up product information. This software is one of the many useful concepts being demonstrated and tested by <a href="http://aura.research.microsoft.com/Aura">Microsoft Research.</a> It is the kind of technology that will makes our lives simpler and more productive. Every time you go shopping or you are in the library, and want to get more information on a product. Whether it is the reviews or the price in the competing stores, AURA shows how it can be done in a snap. The other big use for this prototype is that we can now record and track the products that we own. I am sure you have thought about maintaining a list of all the DVDs in your video collection, or all the books, CDs and games. If the product has a barcode then you can track it. This is something that I have always wanted to do but never had the means. Soon I will be able to do it with ease, with my handy Windows Mobile smartphone. :) In the future AURA will also be able to take advantage of other product tagging systems like RFID, but the current version is limited to barcodes.<br /><br /><a href="http://www.smartphonethoughts.com/images/Kris-Aura-Lens3Big.jpg"><img src="http://www.smartphonethoughts.com/images/Kris-Aura-Lens3Small.jpg" alt="User submitted image" title="User submitted image"/> </a><br /><i>Figure 1: Notice something weird about the Dash?</i><br /><br />Barcode is present on pretty much every product in the store. Our Windows Mobile camera phones have the means to capture them and have the processing power to recognize them, but one of of the biggest limitations of the camera phones in the market is that they have fixed lens or fixed focal length. What it means is that they are suitable for shooting pictures of objects within a certain range, but if they are too close or too far away then the images will not be sharp. This limitation prevents the current Windows Mobile smartphones from taking sharp pictures of barcodes at a close range. Blurry pictures are not good candidates for the barcode recognition software. Microsoft Research team for AURA is providing free lens kit for folks who want to participate in the BETA testing. I signed up and got a free lens for my Dash, that is what you are looking at on my Dash. This special lens adapter enables my Dash to take sharp pictures of the barcodes. If you are interested in learning more about this helpful technology then continue reading. <!> <br /><br /><span><b>Special Lens Adapter</b></span><br />Let's first check out the special lens kit that enables the Windows Mobile devices to take macro photographs. Without this lens adapter the camera, as mentioned earlier, will take blurry photos of the barcodes. We will see it in action shortly. In the future once the camera phones start supporting movable lens and auto focus, this special adapter will not be needed.<br /><br /><a href="http://www.smartphonethoughts.com/images/Kris-Aura-Lens6Big.jpg"><img src="http://www.smartphonethoughts.com/images/Kris-Aura-Lens6Small.jpg" alt="User submitted image" title="User submitted image"/></a> <br /><i>Figure 2: This is the lens kit adapter that I received from Microsoft Research AURA team. Click on the image for the hi-resolution version.</i><br /><br /><a href="http://www.smartphonethoughts.com/images/Kris-Aura-Lens1Big.jpg"><img src="http://www.smartphonethoughts.com/images/Kris-Aura-Lens1Small.jpg" alt="User submitted image" title="User submitted image"/></a> <br /><i>Figure 3: The lens adapter comes with a sticky pad. Click on the image for the hi-resolution version.</i><br /><br /><a href="http://www.smartphonethoughts.com/images/Kris-Aura-Lens2Big.jpg"><img src="http://www.smartphonethoughts.com/images/Kris-Aura-Lens2Small.jpg" alt="User submitted image" title="User submitted image"/></a> <br /><i>Figure 4: Another angle showing the special lens mounted on the Dash. The lens adapter can be moved out of way of the camera lens, so that we can take regular photos. Click on the image for the hi-resolution version.</i><br /><br /><span><b>Test Drive</b></span><br />To demonstrate the capability of Microsoft Research's AURA client for Windows Mobile, I am going to have it scan the barcode for this Sony product and see if it can identify it.<br /><br /><img src="http://www.smartphonethoughts.com/images/Kris-Aura-Lens5Small.jpg" alt="User submitted image" title="User submitted image"/><br /><i>Figure 5: I am going to try scanning the barcode at the bottom of this Sony product.</i><br /><br /><img src="http://www.smartphonethoughts.com/images/Kris-Aura-SP0.jpg" alt="User submitted image" title="User submitted image"/> <br /><i>Figure 6: The AURA client has a simple interface.</i><br /><br />The <i>Scan</i> soft-key button launches the native camera application. The following screen shows the Dash taking a picture of the barcode. You can see that I took the photo at an angle, to make it hard for the barcode recognition software. ;-)<br /><br /><img src="http://www.smartphonethoughts.com/images/Kris-Aura-SP9sony.jpg" alt="User submitted image" title="User submitted image"/><br /><i>Figure 7: Screen shot of the camera application on my Dash taking a picture of the barcode on the Sony product.</i><br /><br />The AURA client automatically reads the barcode from the photo and submits a request to the Microsoft site with the barcode information, to look up the product information. It then shows the product description in the pocket Internet Explorer.<br /><br /><img src="http://www.smartphonethoughts.com/images/Kris-Aura-SP9sonyweb.jpg" alt="User submitted image" title="User submitted image"/> <br /><i>Figure 8: Screen shot of the pocket IE showing my most recently scanned item at the top, the "Sony M-570V Microcassette Voice Recorder."</i><br /><br />Clicking on the item link shows me the product details and also links for finding the product on Amazon.com and MSN Shopping. I will show that shortly. Let us take another look at the client capabilities.<br /><br /><img src="http://www.smartphonethoughts.com/images/Kris-Aura-SP1.jpg" alt="User submitted image" title="User submitted image"/> <br /><i>Figure 9: The screen shot of the simple client interface. It shows all the items that were recently scanned.</i><br /><br /><img src="http://www.smartphonethoughts.com/images/Kris-Aura-SP2.jpg" alt="User submitted image" title="User submitted image"/> <br /><i>Figure 10: The menu options.</i> <br /><br /> <img src="http://www.smartphonethoughts.com/images/Kris-Aura-SP3.jpg" alt="User submitted image" title="User submitted image"/> <br /><i>Figure 11: The extended menu options.</i><br /><PAGEBREAK> <br /><span><b>On the Web</b></span><br />Let's take a closer look at the details that are shown in the Web browser, on the mobile device as well as on the desktop. First let's take a look at the experience in the pocket Internet Explorer.<br /><br /><img src="http://www.smartphonethoughts.com/images/Kris-Aura-SP4.jpg" alt="User submitted image" title="User submitted image"/> <br /><i>Figure 12: The main page shows the recently scanned item. In this case it is a DVD on my favourite cartoon character, Garfield.</i><br /><br /><img src="http://www.smartphonethoughts.com/images/Kris-Aura-SP5.jpg" alt="User submitted image" title="User submitted image"/> <br /><i>Figure 13: Clicking on the item link shows the barcode information, price, manufacturer and other key details.</i> <br /><br /><img src="http://www.smartphonethoughts.com/images/Kris-Aura-SP6.jpg" alt="User submitted image" title="User submitted image"/> <br /><i>Figure 14: Scrolling down on the item details page brings up the links for Amazon.com, MSN Shopping, MSN Search and Usenet Newsgroup. I find the Amazon.com link the most important one.</i> <br /><br /><img src="http://www.smartphonethoughts.com/images/Kris-Aura-SP7.jpg" alt="User submitted image" title="User submitted image"/> <br /><i>Figure 15: This screen shot shows the product details as shown on the Amazon.com's Web page. I had clicked on the link titled "Amazon" shown in the previous screen.</i><br /><br />Nowadays I constantly find myself researching a product on the Amazon.com's Web site before I buy it. Looking at the product ratings, user reviews and of course the price online. AURA makes this information available in a snap. I can be standing in brick and mortar store, needing help with a certain product that has caught my eye. I can instantly and accurately track down that product online. The same can be said for tagging all the products that I own. This brings us to the next set of screen. This time it is the desktop browser.<br /><br /><a href="http://www.smartphonethoughts.com/images/Kris-Aura-Web1Big.jpg"><img src="http://www.smartphonethoughts.com/images/Kris-Aura-Web1Small.jpg" alt="User submitted image" title="User submitted image"/></a><br /><i>Figure 16: The desktop browser experience is similar to the mobile experience. The "My AURA" Web page shows all the scanned items. Click on the image for hi-resolution version.</i> <br /><br /><a href="http://www.smartphonethoughts.com/images/Kris-Aura-Web3Big.jpg"><img src="http://www.smartphonethoughts.com/images/Kris-Aura-Web3Small.jpg" alt="User submitted image" title="User submitted image"/></a><br /><i>Figure 17: You can either click on the item links to perform Amazon.com or MSN Shopping look up, or can click on the "+" symbol to show the product details quickly. Click on the image for hi-resolution version.</i> <br /><br /><a href="http://www.smartphonethoughts.com/images/kris-aura-web4big.jpg"><img src="http://www.smartphonethoughts.com/images/kris-aura-web4small.jpg" alt="User submitted image" title="User submitted image"/></a> <br /><i>Figure 18: AURA also provides RSS feed listing all the scanned items. Click on the image for hi-resolution version.</i><br /><br /><span><b>Another Look at the Special Lens</b></span><br />Before I end this article I would like to show how the special lens adapter provided by the Microsoft Research team helps to capture a sharp photo of the barcode. You can also get one. All you have to do is sign up for the BETA testing and send a self-addressed envelope to the Microsoft Research at <a href="http://aura.research.microsoft.com/Aura/AuraClient.aspx">this address.</a><br /><br /><img src="http://www.smartphonethoughts.com/images/Kris-Aura-SP8bcno.jpg" alt="User submitted image" title="User submitted image"/> <br /><i>Figure 19: Without the special lens adapter, this is how the barcode photo looks like. It is blurry and the AURA client is unable to recognize it.</i><br /><br /><img src="http://www.smartphonethoughts.com/images/Kris-Aura-SP8bc.jpg" alt="User submitted image" title="User submitted image"/> <br /><i>Figure 20: With the special lens adapter in place the photo is sharp.</i><br /><br /><img src="http://www.smartphonethoughts.com/images/Kris-Aura-SP8bccomb.jpg" alt="User submitted image" title="User submitted image"/> <br /><i>Figure 21: This screen shot compares the photo through the special lens and through the regular lens. The top half shows the image through the special lens.</i><br /><br />AURA's client does a pretty good job of recognizing the photos of the barcodes taken through the special lens. It always fails if I take the photo without the special lens, no matter how steady and close I hold the camera. AURA successfully scanned and identified a lot of books and DVDs that I had at home. It failed to identify only two products in my testing. It read the barcode but was not able to look up product details. The first item that failed was Pillsbury White Bread. The second item that it could not identify was a children's book called <a href="http://www.amazon.com/Board-Book-Cuddles-Playmore-Publishing/dp/0866115943/ref=pd_bbs_sr_1/102-6444457-1524969?ie=UTF8&s=books&qid=1174889290&sr=8-1">"Cuddles the Cow."</a> <br /><br />One thing I am not sure of, and it would be awesome if this barcode recognition technology is made available as an API for the developers. If the developers can capture the barcodes in their applications, then we can see some cool applications. For example a coupon finder application; we scan the barcode in and the software finds the coupons for that product. Also enterprise developers can use it for tracking inventory and assets. ;-) <br /><br /><span><b>Closing Remarks</b></span><br />AURA is currently a Microsoft Research project and is in public BETA. This is definitely not the final product. I am sure it will undergo many revisions and introduce more features in its final version. The biggest challenge currently for the Microsoft Research folks and for the users is the need to attach the special adapter lens. The camera on our mobile device has to evolve before this product can become a reality. Till then we can definitely play with this BETA, and dream about the possibilities that are waiting to be unlocked in the future.<br /><br /><i>Kris Kumar is a software engineer with expertise in Microsoft technologies and developer tools. He is married to a wonderful woman, and lives in Rochester, New York, USA.</i>