05-21-2006, 04:35 AM
|
|
Executive Editor
Join Date: Aug 2006
Posts: 29,160
|
|
MobiPocket eNews Creator Banned from Thoughts Media Server
It's with a great sense of frustration that tonight I implemented a robots.txt ban of the Mobipocket eNews Creator bot. If that doesn't work I'll be implementing an IP address block. It's been a rough week for our server - we've had a lot of traffic and the server has been slow and unstable because of it. I dug into the logs to see where the traffic was coming from, and I was shocked to see huge amounts of traffic coming from a robot identifying itself as "eNews Creator". How huge? On Wednesday the 17th of this month, the eNews Creator bot hit Pocket PC Thoughts.com 37,499 times alone. Half a gig of bandwidth in one day. The entire month of April? Our server was hit 938,472 times, for a total of 18.55 GB of bandwidth. To put those numbers into perspective, the Google bot only hit us 431,010 times in the same month and only used 3.29 GB of bandwidth.
I don't take this action lightly, because there are evidently people interested in using the Mobipocket client to read news from Pocket PC Thoughts. The problem is that Mobipocket has evidently created their software without regard for the servers that they are scraping. They created it to hit servers without respecting the rights of the publisher. No RSS/scraper client needs to hit a server 37 thousand times in one day, period.
Upon discovering this information, I attempted to contact Mobipocket to resolve the matter. I scoured their Web site looking for some way to contact them directly - nothing. No email addresses, no contact form - they direct everything into the forums. I managed to find one email address on their privacy page, but upon emailing it I received an autoresponder stating that they did not monitor the alias and only responded to posts in the forums. :evil: I tried webmater@ and postmaster@ - two email aliases that should always work. Neither did - both bounced back because the aliases do not exist. I sent two private messages to Mobipocket employees in the forums on Thursday, asking them to respond to this issue, and 48 hours later I haven't received a response. I tried sending a private message to the forum admin - they've configured the admin account to refuse all private messages. :roll:
If you're a Pocket PC Thoughts reader that uses the Mobipocket client, I'd encourage you to contact them and point them to this posting. I'd like to find a way to resolve this. In the meantime, use NewsBreak and subscribe to our RSS feed.
|
| |
|
|
|
05-21-2006, 04:53 AM
|
|
Intellectual
Join Date: Aug 2006
Posts: 127
|
|
This might help don't know...
Maybe this will be helpful I doubt it based upon what you've expressed.
http://www.networksolutions.com
Registrant:
mobipocket
104 av. President Kennedy
PARIS, Ile de France 75016
FR
Domain Name: MOBIPOCKET.COM
Administrative Contact:
brethes, thierry [email protected]
Mobipocket.com
104 av. President Kennedy
PARIS, Ile de France 75016
FR
+33 1 44 14 15 52 fax: +33 1 44 14 15 56
Technical Contact:
Network Solutions, LLC. [email protected]
13200 Woodland Park Drive
Herndon, VA 20171-3025
US
1-888-642-9675 fax: 571-434-4620
Record expires on 01-Feb-2010.
Record created on 01-Feb-2000.
Database last updated on 20-May-2006 23:51:14 EDT.
Domain servers in listed order:
NS81.WORLDNIC.COM 205.178.190.41
NS82.WORLDNIC.COM 205.178.189.41
Registry Status: REGISTRAR-LOCK
Registry Status: clientTransferProhibited
Registry Status: clientDeleteProhibited
Registry Status: clientUpdateProhibited
Domain Name: MOBIPOCKET.COM
Registrar: NETWORK SOLUTIONS, LLC.
Whois Server: whois.networksolutions.com
Referral URL: http://www.networksolutions.com
Name Server: NS82.WORLDNIC.COM
Name Server: NS81.WORLDNIC.COM
Status: REGISTRAR-LOCK
EPP Status: clientTransferProhibited
EPP Status: clientDeleteProhibited
EPP Status: clientUpdateProhibited
Updated Date: 07-Jan-2004
Creation Date: 01-Feb-2000
Expiration Date: 01-Feb-2010
>>> Last update of whois database: Sat, 20 May 2006 23:51:44 EDT <<<
|
| |
|
|
|
05-21-2006, 04:57 AM
|
|
Executive Editor
Join Date: Aug 2006
Posts: 29,160
|
|
Re: This might help don't know...
Quote:
|
Originally Posted by code-frog
Maybe this will be helpful I doubt it based upon what you've expressed.
|
Thanks, I'll give it a try - message sent. Though given the way they reject all other incoming email, I'd be surprised if they check email on that alias.
|
| |
|
|
|
05-21-2006, 05:00 AM
|
|
Intellectual
Join Date: Aug 2006
Posts: 127
|
|
Theoretically they have to. If you don't receive email on your domain registered email regularly your DNS provider can make changes that will break all of your stuff and you won't know because you never received the email they sent two weeks before the change.
I noticed a France origin on all the contact info so you might even get a response tonight if someone is at least polling on that address for hits.
The next step would be to register a complaint with Network Solutions as they are listed as the technical contact. While Network Solutions is dreadfully slow by email/web a phone call can work wonders.
If you want/need additional help on this let me know. I'm quite happy to do this stuff as it's ... well ... something I do a lot of. 
- Rex
|
| |
|
|
|
05-21-2006, 05:12 AM
|
|
Executive Editor
Join Date: Aug 2006
Posts: 29,160
|
|
Quote:
|
Originally Posted by code-frog
The next step would be to register a complaint with Network Solutions as they are listed as the technical contact. While Network Solutions is dreadfully slow by email/web a phone call can work wonders.
|
Nah, no need for that - we can just block them at the firewall level and that's that. I have this suspicion that anyone who would design a software tool like this would also design it to ignore robots.txt, so I may have to IP ban them.
|
| |
|
|
|
05-21-2006, 05:16 AM
|
|
Intellectual
Join Date: Aug 2006
Posts: 127
|
|
Banishing the IP would be the most direct route. I think they would be getting some complaints on that maneuver.
Well, I'm definitely curious to see if they are a manned operation checking those addresses if they aren't that puts them in the "suspicious" category as a web based company with web based products. Wonder if Mark Rossovich has had any contact with them. 
|
| |
|
|
|
05-21-2006, 05:44 AM
|
|
Executive Editor, Android Thoughts
Join Date: Aug 2006
Posts: 3,233
|
|
This software strikes me as the work of a lazy programmer more than an irresponsible company (of course, lazy programmers probably work at irresponsible companies). It would never be in a company's best interest to destroy the servers of the content providers they hope to bring to their customers. Seems to me the more likely thing is that the software was just written poorly (Ok, VERY poorly) and now that the company realizes that, they've fortified themselves by directing everything to useless forums. My guess is that we aren't the first site to have this happen, and that they've probably honed their skills of avoidance.
Therefore, I think blocking them at our level is best, and moving on. I realize some people want to help us out, but I don't think Jason is ready to take it beyond a block, and I don't think it really makes much sense.
Of course, I may be wrong, and Jason may feel especially vengeful tonight :twisted:
__________________
Dr. Jon Westfall, MCSE, MS-MVP
Executive Editor - Android Thoughts
News Editor - Windows Phone Thoughts
|
| |
|
|
|
05-21-2006, 07:18 AM
|
|
Thinker
Join Date: May 2004
Posts: 481
|
|
Quote:
|
Originally Posted by Jon Westfall
This software strikes me as the work of a lazy programmer...
Therefore, I think blocking them at our level is best, and moving on. I realize some people want to help us out, but I don't think Jason is ready to take it beyond a block, and I don't think it really makes much sense.
Of course, I may be wrong, and Jason may feel especially vengeful tonight :twisted:
|
Sounds like a quick and dirty web scraper that someone threw together (a non-programmer type) and never took the time to check out if it was working correctly (or really didn't care). Hitting a server 37k time in a day is kind of sleazy to put it nicely. Bandwidth cost $$$$ and it's with the kindness of the host to let some bot scrape it's pages not some idiots right to do it without regard for damages done by it.
My own 2cents says just block the IP and move on. By the fact that there is no way to contact him/her/them or their offices (it they even exist) just goes to show you that the admin of MobiPockets is just an irresponsible person that doesn't care much about anything that goes on outside their own perimeter.
Jeff-
|
| |
|
|
|
05-21-2006, 07:19 AM
|
|
Editor Emeritus
Join Date: Aug 2006
Posts: 15,171
|
|
The irony is... I really like Mobipocket as an ebook reader; in fact, it's my default ebook reader. However, their desktop tools do seem a little less polished/mature. 
--janak
|
| |
|
|
|
05-21-2006, 09:08 AM
|
|
Intellectual
Join Date: Dec 2006
Posts: 256
|
|
I'd have your blood pressure checked after that experience Jason. You couldn't make it up 8O
|
| |
|
|
|
|
|
Linear Mode
