MobiPocket eNews Creator Banned from Thoughts Media Server

[Menneisyys]

Originally Posted by Jason Dunn

No solution is viable if it relies on people voluntarily doing something because I ask them to.

You can force them - MWC supports cookies / cookie-based authentication and, therefore, you can always know who is downloading content with MWC and who with IE. Then, you can easily filter out who hasn't switched to manual, bandwidth-friendly mode. Also, you can make MWC-based access account/password-based only - that is, say, unavailable for non-subscribers.

(Let me know if you need more info on the network model / the built-in capabilities of the latest MWC version and I check them to see if anything has changed in the last 3 years, networking-wise.)

[Menneisyys]

Originally Posted by Jason Dunn

No solution is viable if it relies on people voluntarily doing something because I ask them to.

You can force them - MWC supports cookies / cookie-based authentication and, therefore, you can always know who is downloading content with MWC and who with IE. Then, you can easily filter out who hasn't switched to manual, bandwidth-friendly mode. Also, you can make MWC-based access account/password-based only - that is, say, unavailable for non-subscribers.

(Let me know if you need more info on the network model / the built-in capabilities of the latest MWC version and I check them to see if anything has changed in the last 3 years, networking-wise.)

[FallN]

I've personally never liked Mobipocket products. They always seemed junky and unpolished to me. I would ban their IPs and be done with it. Save yourself and the servers the headache.

[manywhere]

Originally Posted by Menneisyys

They didn't make an IP ban - just haven't returned anything (except the Forbidden HTTP status code) when they sensed the eNews Creator HTTP User Agent. That is, everything (all IP's) passed to the server; of them, only ones that weren't using the eNews Creator-specific User Agent were (are) served with actual content.

Yup. I think they "issued the block" by configuring the Htaccess file to return a 403 error code (i.e. forbidden) when the mobipocket user agent is sent in the http request header. It's quite simple and used extensively, so a simple Google search on it will return lots of results. You could also put a custom message to be shown, if that feels like the right thing to do.

You're right in doing this Jason. I'm really surprised to see such bad coding, let alone ignoring the use of robots.txt and the "is this page updated" http request method, has been released into the public. Bad, bad developer(s). :twak:

[BevHoward]

Any company's decision to make it difficult to impossible to easily contact them on any matter is an important clue to how they view their customers and is one of the factors I use when making purchase decisions _before_ the purchase.

MobiPocket, despite offering a good lit generator, failed this criteria years ago, so, this event does not surprise.

Good luck persuing it... if they do respond, be sure to email them the url of this thread ;-)

[Jason Dunn]

I've traded a few private messages with Mobipocket, but the bulk of the conversation has been in this thread:

http://www.mobipocket.com/forum/view...hp?p=6018#6018

Basically their attitude is that it's not their problem my site is so popular. :?

I find it fascinating that not one person that uses their tool has popped into this thread to comment.

[Menneisyys]

Originally Posted by Jason Dunn

I've traded a few private messages with Mobipocket, but the bulk of the conversation has been in this thread:

http://www.mobipocket.com/forum/view...hp?p=6018#6018

Basically their attitude is that it's not their problem my site is so popular. :?

Well, it's indeed pretty hard do control independent scraper clients that don't rely on any central server persmission (or any TTL scheme) from a centralized location. That is, eNews creator users can freely add any site they want to replicate onto their configuration - noone can control this from anywhere else. The only way to stop the excessive downloads is selective prohibition.

The easiest and cleanest way is just implementing a User-Agent-based "ban" (that is, just returning a lightweight "Forbidden" message, putting almost no additional burden on the Web server) as the Register has done years ago when they banned out eNews Creator.

[Jason Dunn]

Originally Posted by Menneisyys

Well, it's indeed pretty hard do control independent scraper clients that don't rely on any central server persmission (or any TTL scheme) from a centralized location. That is, eNews creator users can freely add any site they want to replicate onto their configuration - noone can control this from anywhere else. The only way to stop the excessive downloads is selective prohibition.

The idea of scraper clients are pretty hostile to Web servers, but if Mobipocket had created it with some logical limits, I think it would have been ok. Things such as not hitting a server more than once every 60 minutes automatically, limiting the total number of pages scraped to 20 or something reasonable...my understanding is that they don't limit what the user does, and let's face it, most people don't think about server resources or what kind of damage they might be doing, they just click a few buttons and get what they want. A responsible software developer would build in some reasonable limits to balance the needs of the customer with that of the publisher.

[Menneisyys]

Originally Posted by Jason Dunn

The idea of scraper clients are pretty hostile to Web servers, but if Mobipocket had created it with some logical limits, I think it would have been ok. Things such as not hitting a server more than once every 60 minutes automatically, limiting the total number of pages scraped to 20 or something reasonable...my understanding is that they don't limit what the user does, and let's face it, most people don't think about server resources or what kind of damage they might be doing, they just click a few buttons and get what they want. A responsible software developer would build in some reasonable limits to balance the needs of the customer with that of the publisher.

Now I understand.

From the programmer's point of view, MWC is a very simple app. It lacks almost every advanced for example content filtering, selective link-following capabilities. (This is why I've written a complete, additional filter proxy to augment its capabilities.) This is why it also lacks any kind of constraints - except for a very simple 5-minute timeout.

As the Mobi folks seem to completely drop HTML-based eNews from the next version of MWC, I don't think they'll ever implement this. That is, just not serving MWC clients (based on their User-Agent HTTP header, which, if they don't use filtering proxies - which the casual user will never bother ton install and set up -, isn't spoofable/changable).

[Jason Dunn]

Oh, and by the way, we've now fully blocked their client. Anyone hitting the site with the eNews Creator client gets a message saying why it has been blocked, and the URL to my post here.

[GoldKey]

Originally Posted by Jason Dunn

Basically their attitude is that it's not their problem my site is so popular. :?

Technically, they've built a tool that depending how it was used could in essence be used as part of a DOS attack. Since it would be very easy to inadvertently set it up to be way to aggressive, you'd think they would be a little more concerned.