Jason Dunn
05-21-2006, 04:46 AM
It's with a great sense of frustration that tonight I implemented a robots.txt ban of the Mobipocket eNews Creator bot. If that doesn't work I'll be implementing an IP address block. It's been a rough week for our server - we've had a lot of traffic and the server has been slow and unstable because of it. I dug into the logs to see where the traffic was coming from, and I was shocked to see huge amounts of traffic coming from a robot identifying itself as "eNews Creator". How huge? On Wednesday the 17th of this month, the eNews Creator bot hit Pocket PC Thoughts.com <b>37,499</b> times alone. Half a gig of bandwidth in one day. The entire month of April? Our server was hit 938,472 times, for a total of 18.55 GB of bandwidth. To put those numbers into perspective, the Google bot only hit us 431,010 times in the same month and only used 3.29 GB of bandwidth.<br /><br />I don't take this action lightly, because there are evidently people interested in using the Mobipocket client to read news from Digital Media Thoughts. The problem is that Mobipocket has evidently created their software without regard for the servers that they are scraping. They created it to hit servers without respecting the rights of the publisher. <b>No RSS/scraper client needs to hit a server 37 thousand times in one day, period</b>. <br /><br />Upon discovering this information, I attempted to contact Mobipocket to resolve the matter. I scoured their Web site looking for some way to contact them directly - nothing. No email addresses, no contact form - they direct everything into the forums. I managed to find one email address on their privacy page, but upon emailing it I received an autoresponder stating that they did not monitor the alias and only responded to posts in the forums. :evil: I tried webmater@ and postmaster@ - two email aliases that should <i>always</i> work. Neither did - both bounced back because the aliases do not exist. I sent two private messages to Mobipocket employees in the forums on Thursday, asking them to respond to this issue, and 48 hours later I haven't received a response. I tried sending a private message to the forum admin - they've configured the admin account to refuse all private messages. :roll:<br /><br />If you're a Digital Media Thoughts reader that uses the Mobipocket client, I'd encourage you to contact them and point them to this posting. I'd like to find a way to resolve this. In the meantime, use <a href="http://tinyurl.com/k4sd5">NewsBreak</a> and subscribe to our <a href="http://www.digitalmediathoughts.com/xml/">RSS feed</a>.