MobiPocket eNews Creator Banned from Thoughts Media Server

[Jason Dunn]

It's with a great sense of frustration that tonight I implemented a robots.txt ban of the Mobipocket eNews Creator bot. If that doesn't work I'll be implementing an IP address block. It's been a rough week for our server - we've had a lot of traffic and the server has been slow and unstable because of it. I dug into the logs to see where the traffic was coming from, and I was shocked to see huge amounts of traffic coming from a robot identifying itself as "eNews Creator". How huge? On Wednesday the 17th of this month, the eNews Creator bot hit Pocket PC Thoughts.com 37,499 times alone. Half a gig of bandwidth in one day. The entire month of April? Our server was hit 938,472 times, for a total of 18.55 GB of bandwidth. To put those numbers into perspective, the Google bot only hit us 431,010 times in the same month and only used 3.29 GB of bandwidth.

I don't take this action lightly, because there are evidently people interested in using the Mobipocket client to read news from Pocket PC Thoughts. The problem is that Mobipocket has evidently created their software without regard for the servers that they are scraping. They created it to hit servers without respecting the rights of the publisher. No RSS/scraper client needs to hit a server 37 thousand times in one day, period.

Upon discovering this information, I attempted to contact Mobipocket to resolve the matter. I scoured their Web site looking for some way to contact them directly - nothing. No email addresses, no contact form - they direct everything into the forums. I managed to find one email address on their privacy page, but upon emailing it I received an autoresponder stating that they did not monitor the alias and only responded to posts in the forums. :evil: I tried webmater@ and postmaster@ - two email aliases that should always work. Neither did - both bounced back because the aliases do not exist. I sent two private messages to Mobipocket employees in the forums on Thursday, asking them to respond to this issue, and 48 hours later I haven't received a response. I tried sending a private message to the forum admin - they've configured the admin account to refuse all private messages. :roll:

If you're a Pocket PC Thoughts reader that uses the Mobipocket client, I'd encourage you to contact them and point them to this posting. I'd like to find a way to resolve this. In the meantime, use NewsBreak and subscribe to our RSS feed.

[Jason Dunn]

It's with a great sense of frustration that tonight I implemented a robots.txt ban of the Mobipocket eNews Creator bot. If that doesn't work I'll be implementing an IP address block. It's been a rough week for our server - we've had a lot of traffic and the server has been slow and unstable because of it. I dug into the logs to see where the traffic was coming from, and I was shocked to see huge amounts of traffic coming from a robot identifying itself as "eNews Creator". How huge? On Wednesday the 17th of this month, the eNews Creator bot hit Pocket PC Thoughts.com 37,499 times alone. Half a gig of bandwidth in one day. The entire month of April? Our server was hit 938,472 times, for a total of 18.55 GB of bandwidth. To put those numbers into perspective, the Google bot only hit us 431,010 times in the same month and only used 3.29 GB of bandwidth.

I don't take this action lightly, because there are evidently people interested in using the Mobipocket client to read news from Pocket PC Thoughts. The problem is that Mobipocket has evidently created their software without regard for the servers that they are scraping. They created it to hit servers without respecting the rights of the publisher. No RSS/scraper client needs to hit a server 37 thousand times in one day, period.

Upon discovering this information, I attempted to contact Mobipocket to resolve the matter. I scoured their Web site looking for some way to contact them directly - nothing. No email addresses, no contact form - they direct everything into the forums. I managed to find one email address on their privacy page, but upon emailing it I received an autoresponder stating that they did not monitor the alias and only responded to posts in the forums. :evil: I tried webmater@ and postmaster@ - two email aliases that should always work. Neither did - both bounced back because the aliases do not exist. I sent two private messages to Mobipocket employees in the forums on Thursday, asking them to respond to this issue, and 48 hours later I haven't received a response. I tried sending a private message to the forum admin - they've configured the admin account to refuse all private messages. :roll:

If you're a Pocket PC Thoughts reader that uses the Mobipocket client, I'd encourage you to contact them and point them to this posting. I'd like to find a way to resolve this. In the meantime, use NewsBreak and subscribe to our RSS feed.

[code-frog]

Maybe this will be helpful I doubt it based upon what you've expressed.

http://www.networksolutions.com


Registrant:
mobipocket
104 av. President Kennedy
PARIS, Ile de France 75016
FR

Domain Name: MOBIPOCKET.COM

Administrative Contact:
brethes, thierry tbrethes@mobipocket.com
Mobipocket.com
104 av. President Kennedy
PARIS, Ile de France 75016
FR
+33 1 44 14 15 52 fax: +33 1 44 14 15 56

Technical Contact:
Network Solutions, LLC. customerservice@networksolutions.com
13200 Woodland Park Drive
Herndon, VA 20171-3025
US
1-888-642-9675 fax: 571-434-4620

Record expires on 01-Feb-2010.
Record created on 01-Feb-2000.
Database last updated on 20-May-2006 23:51:14 EDT.

Domain servers in listed order:

NS81.WORLDNIC.COM 205.178.190.41
NS82.WORLDNIC.COM 205.178.189.41

Registry Status: REGISTRAR-LOCK
Registry Status: clientTransferProhibited
Registry Status: clientDeleteProhibited
Registry Status: clientUpdateProhibited

Domain Name: MOBIPOCKET.COM
Registrar: NETWORK SOLUTIONS, LLC.
Whois Server: whois.networksolutions.com
Referral URL: http://www.networksolutions.com
Name Server: NS82.WORLDNIC.COM
Name Server: NS81.WORLDNIC.COM
Status: REGISTRAR-LOCK
EPP Status: clientTransferProhibited
EPP Status: clientDeleteProhibited
EPP Status: clientUpdateProhibited
Updated Date: 07-Jan-2004
Creation Date: 01-Feb-2000
Expiration Date: 01-Feb-2010

>>> Last update of whois database: Sat, 20 May 2006 23:51:44 EDT <<<

[Jason Dunn]

Originally Posted by code-frog

Maybe this will be helpful I doubt it based upon what you've expressed.

Thanks, I'll give it a try - message sent. Though given the way they reject all other incoming email, I'd be surprised if they check email on that alias.

[code-frog]

Theoretically they have to. If you don't receive email on your domain registered email regularly your DNS provider can make changes that will break all of your stuff and you won't know because you never received the email they sent two weeks before the change.

I noticed a France origin on all the contact info so you might even get a response tonight if someone is at least polling on that address for hits.

The next step would be to register a complaint with Network Solutions as they are listed as the technical contact. While Network Solutions is dreadfully slow by email/web a phone call can work wonders.

If you want/need additional help on this let me know. I'm quite happy to do this stuff as it's ... well ... something I do a lot of.

- Rex

[Jason Dunn]

Originally Posted by code-frog

The next step would be to register a complaint with Network Solutions as they are listed as the technical contact. While Network Solutions is dreadfully slow by email/web a phone call can work wonders.

Nah, no need for that - we can just block them at the firewall level and that's that. I have this suspicion that anyone who would design a software tool like this would also design it to ignore robots.txt, so I may have to IP ban them.

[code-frog]

Banishing the IP would be the most direct route. I think they would be getting some complaints on that maneuver.

Well, I'm definitely curious to see if they are a manned operation checking those addresses if they aren't that puts them in the "suspicious" category as a web based company with web based products. Wonder if Mark Rossovich has had any contact with them.

[Jon Westfall]

This software strikes me as the work of a lazy programmer more than an irresponsible company (of course, lazy programmers probably work at irresponsible companies). It would never be in a company's best interest to destroy the servers of the content providers they hope to bring to their customers. Seems to me the more likely thing is that the software was just written poorly (Ok, VERY poorly) and now that the company realizes that, they've fortified themselves by directing everything to useless forums. My guess is that we aren't the first site to have this happen, and that they've probably honed their skills of avoidance.

Therefore, I think blocking them at our level is best, and moving on. I realize some people want to help us out, but I don't think Jason is ready to take it beyond a block, and I don't think it really makes much sense.

Of course, I may be wrong, and Jason may feel especially vengeful tonight :twisted:

[lapchinj]

Originally Posted by Jon Westfall

This software strikes me as the work of a lazy programmer...

Therefore, I think blocking them at our level is best, and moving on. I realize some people want to help us out, but I don't think Jason is ready to take it beyond a block, and I don't think it really makes much sense.

Of course, I may be wrong, and Jason may feel especially vengeful tonight :twisted:

Sounds like a quick and dirty web scraper that someone threw together (a non-programmer type) and never took the time to check out if it was working correctly (or really didn't care). Hitting a server 37k time in a day is kind of sleazy to put it nicely. Bandwidth cost $$$$ and it's with the kindness of the host to let some bot scrape it's pages not some idiots right to do it without regard for damages done by it.

My own 2cents says just block the IP and move on. By the fact that there is no way to contact him/her/them or their offices (it they even exist) just goes to show you that the admin of MobiPockets is just an irresponsible person that doesn't care much about anything that goes on outside their own perimeter.

Jeff-

[Janak Parekh]

The irony is... I really like Mobipocket as an ebook reader; in fact, it's my default ebook reader. However, their desktop tools do seem a little less polished/mature.

--janak

[dommasters]

I'd have your blood pressure checked after that experience Jason. You couldn't make it up 8O