MobiPocket eNews Creator Banned from Thoughts Media Server

[Jason Dunn]

It's with a great sense of frustration that tonight I implemented a robots.txt ban of the Mobipocket eNews Creator bot. If that doesn't work I'll be implementing an IP address block. It's been a rough week for our server - we've had a lot of traffic and the server has been slow and unstable because of it. I dug into the logs to see where the traffic was coming from, and I was shocked to see huge amounts of traffic coming from a robot identifying itself as "eNews Creator". How huge? On Wednesday the 17th of this month, the eNews Creator bot hit Pocket PC Thoughts.com <b>37,499</b> times alone (the numbers for Smartphone Thoughts weren't as bit, but still large). Half a gig of bandwidth in one day. The entire month of April? Our server was hit 938,472 times, for a total of 18.55 GB of bandwidth. To put those numbers into perspective, the Google bot only hit us 431,010 times in the same month and only used 3.29 GB of bandwidth.I don't take this action lightly, because there are evidently people interested in using the Mobipocket client to read news from Smartphone Thoughts. The problem is that Mobipocket has evidently created their software without regard for the servers that they are scraping. They created it to hit servers without respecting the rights of the publisher. <b>No RSS/scraper client needs to hit a server 37 thousand times in one day, period</b>. Upon discovering this information, I attempted to contact Mobipocket to resolve the matter. I scoured their Web site looking for some way to contact them directly - nothing. No email addresses, no contact form - they direct everything into the forums. I managed to find one email address on their privacy page, but upon emailing it I received an autoresponder stating that they did not monitor the alias and only responded to posts in the forums. :evil: I tried webmater@ and postmaster@ - two email aliases that should <i>always</i> work. Neither did - both bounced back because the aliases do not exist. I sent two private messages to Mobipocket employees in the forums on Thursday, asking them to respond to this issue, and 48 hours later I haven't received a response. I tried sending a private message to the forum admin - they've configured the admin account to refuse all private messages. :roll:If you're a Smartphone Thoughts reader that uses the Mobipocket client, I'd encourage you to contact them and point them to this posting. I'd like to find a way to resolve this. In the meantime, use <a href="http://www.smartphone.net/smartphonethoughts/software_detail.asp?id=1497">NewsBreak</a> and subscribe to our <a href="http://www.smartphonethoughts.com/xml">RSS feed</a>.

[Rocco Augusto]

ouch! thats weak. what i would do next is send them a bill for the massive amount of bandwidth that they used :(

[Mike Temporale]

That works out to something like 26.045 times per minute. That is pretty impressive! Well, by impressive, I don't mean cool. It's extremely poorly designed.

Ban 'em. No worries. What kind of company doesn't have an easy way for their clients to contact them? :roll:

[Jerry Raia]

I haven't used them in a while. Send em to the corn field I say! :twisted:

[extravagant]

ya right thats a lot of bandwidth..... is it even possible to hold them responsible for using that much? or would it have to be stated in the Legal section of the website then it's only enforcable?

[Rocco Augusto]

I haven't used them in a while. Send em to the corn field I say! :twisted:

dont do that, i like corn! i mean if they hit the server 37,000 times in one day just imagine waht they would do to the corn! :lol:

if it was me, i would ban them. they obviously dont hold hold the sites they're scrapping in very high regards in the first place or they would have optimized their programs to not bash the servers as they have been doing

[edgar]

What's stupid about this is they obviously don't manage their own services.

If they are pulling that much bandwidth and data from you, think of the pipe on their end times 1000+ sites.

They are probably wasting tons of bits, tons of cpu cycles and tons of disk space. Gee wish I was their VC and knew they were wasting my money that way. A little tweaking of the poor coding and they are down to two Linux boxes in HA running on old Pentium II's and 200Gb in disk :rotfl:

Nothing a good sock full of soap upside the head wouldn't fix.

PS internic returns the admin contact for Mobiread.com as "Turcic, Turcic at206 at columbia dot edu" Registered via Godaddy. Send a note to godaddy support and complain.

They return mobipocket.com (registered at Networksolutions.com with tech contact at [email protected]) Admin contact "brethes, thierry tbrethes at mobipocket dot com"

[Mike Temporale]

It's my guess, that it isn't actually their server hitting the site. I would guess that each person who updates their client is actually hitting the site directly. Just a guess, but it's the only way I can see them hitting the site so many times. Otherwise, as Edgar points out, it would be a huge waste on their end too.

[edgar]

Mike is probably right, I see they have a "Custom e-news" service - that allows you to add your own channels.

Again, however, if you want to control and own your clients wouldn't it be a better service to scrape yourself (or even better rss feed to your own server) and serve from your own cache vs having your client leave your control to get the information on their own?

Its a simple non scrape hit on a site to see if its updated - its part of the initial response a site gives at ACK/connect isn't it?

Either way, mobipocket has a poor technical and business model. It won't take long before other major news sites like Washingtonpost.com or Newyorktimes.com etc kill their connection as well. Then the bottom falls out of their business and they become a non-entity.