We're a Little Too Clever For Our Own Good

[Jason Dunn]

Most of you will have noticed that this site has been up and down today like a yo-yo - we've had three complete melt downs where processes have gotten completely out of control, resulting in a server too slow to process anything. Here's why...<br /><br />Web-site functionality is a double-edged sword. On one hand, our integration of the front page with phpBB is very functional and useful. It allows us to mine the database in different ways, exposing useful elements like the last ten posts, the top stories, number of responses to each post, a survey, and more. The problem is that for each one of those clever integrations, it requires a query to our database. Each query requires RAM and CPU resources to process - I believe our opening page has in excess of 25 SQL queries, and when 1000's of you are visiting every day, the net result is an overloaded server. Dynamic sites may be "cool", but they're brutal on a server if not designed to scale with large amounts of traffic. Every time you visit the opening page, if you're cookied, you get logged into the phpBB forum. Clever, yes, but resource consuming like you wouldn't believe!<br /><br />We're working on several ways to combat this problem, including the potential removal of some of functionality of the opening page. We're going to try and implement a caching solution that will reduce the load, but if that doesn't work something has to change - we can't continue to offer rich functionality if it kills the site on a regular basis. Thanks for your patience in this while we work through the issues, and I'll keep you informed of any changes that we need to make that will impact your use of the site.

[Xaximus]

I personally don't use the "last ten posts" and "top ten stories" thingies... never really noticed them to be honest with you. Perhaps they could be disabled by default and users who prefer can enable them in their preferences?

[Charles Pickrell]

Jason,

Is there any way to have the query results cached on the server? The home page doesn't change that often. It would be cool if you only fired a refresh event to the home page when the database content changed.

[KyleC]

I like everything that the front page of Pocket PC Thoughts has to offer, and I don't want to see ANY of it taken down. Well, maybe the ads! :wink: Anyway, perhaps you ought to switch to another server. Perhaps Microsoft will allow you use of some of their gigantic bandwidth because you are a site deticated to their product...?

[Janak Parekh]

Is there any way to have the query results cached on the server? The home page doesn't change that often. It would be cool if you only fired a refresh event to the home page when the database content changed.
Indeed, PDABuzz only updated last 10 posts once every 5-10 minutes or so, IIRC, and served it statically to everyone.

Wes, if you're reading - did you encounter any of these problems and did you have any novel ideas as to how to resolve it?

--bdj

[Venturello]

As someone said... make stuff static, and make it update every x minutes. 5 minutes should be short enought, and save you tons of bandwidth/cpu cycle. Yeah I know conceptually a fully dynamically website is very cool, and toying around with SQL is fun, but there's a price to pay. Another option is to make the stuff to the right (top ten posts, etc) optional, such that default users don't see it. Betcha a good percent of ppct users will not even miss it.

Anyone critizicing the ads... how you think we get this for 'free'? Jason and team are EXCELLENT and if I could help them by buying stuff throught their ads for the same price I would elsewhere, and from merchants that Jason and team trusts, then I will. I also open ads on the site often, I find interesting stuff AND contribute a bit to keep ppct running... just my 2cents!

[Venturello]

Hey BigDaddyJ, that's an IBM XT!! The original PC, at 4.77 Mhz, my father bought one when I was young. So cool all his geek friends came over and basicly spent a weekend at our place enjoying the machine! When my father upgraded to a hellishly fast 16Mhz 286, he left me that. I began programming in that computer!!

[Jason Dunn]

Is there any way to have the query results cached on the server?

"...We're going to try and implement a caching solution that will reduce the load..." :lol:

[JonnoB]

I like everything that the front page of Pocket PC Thoughts has to offer, and I don't want to see ANY of it taken down.

I too like what Thoughts has to offer. It is my one for-sure place to stop each day. Since cookies can auto-login you into the forums, perhaps that is a feature that should be turned on by advanced users otherwise it is disabled?

[icatar]

Now you're not actually suggesting that we visit the site less often to reduce the load....are you???? :lol: 8O

[Jason Dunn]

I like everything that the front page of Pocket PC Thoughts has to offer, and I don't want to see ANY of it taken down. Well, maybe the ads! :wink:

If the ads come down, so does the site - but, yeah, in an ideal world I'd be a billionaire who could fund this out of his pocket and do it just for fun. Maybe next year I'll win the lottery. :lol:

Anyway, perhaps you ought to switch to another server. Perhaps Microsoft will allow you use of some of their gigantic bandwidth because you are a site deticated to their product...?

Well, it's not really the server's fault - right now we're on a dedicated 1.7 Ghz Celeron box with 512 MB of RAM. Not super-top end, but pretty typical for a dedicated server. I just put in the upgrade for an extra 512 of RAM, which should help.

Microsoft has helped out quite a bit already - they gave us a grant to help cover the server moving costs + hosting, so the rest of up to me. :D

I'm not saying the site will shut down or anything - we just need to get smarter about some of the functionality...

[Janak Parekh]

Hey BigDaddyJ, that's an IBM XT!! The original PC, at 4.77 Mhz, my father bought one when I was young. So cool all his geek friends came over and basicly spent a weekend at our place enjoying the machine! When my father upgraded to a hellishly fast 16Mhz 286, he left me that. I began programming in that computer!!
To be more precise, it's the original IBM PC model 5150... but of course the XT had the nearly-identical housing. Yes, that's why I picked it ;)

There was a "why'd you pick your avatar" thread floating here in the Off-Topic forum, my "long explanation" as to why I use that avatar is there.

--bdj

[Jason Dunn]

As someone said... make stuff static, and make it update every x minutes. 5 minutes should be short enought, and save you tons of bandwidth/cpu cycle. Yeah I know conceptually a fully dynamically website is very cool, and toying around with SQL is fun, but there's a price to pay.

Yup, that's the lesson I'm learning here. :-) We're doing to make as much stuff static as possible moving forward...

[Jason Dunn]

Now you're not actually suggesting that we visit the site less often to reduce the load....are you???? :lol: 8O

Nope - come early, come often. :-) We'll figure out how to make this work, don't worry...

[gandi]

hi there,

just some thoughts of what i believe is the best sollution for both functionality and performance after my experience:
make a static frontpage instead of a dynamic one. these page may be refreshed every 5 minutes and after somebody posted some news - or something like that. because most of the users only take a look at the frontpage this will result in a huge performance boost whilst still having all your features onto the page - even if they are not "up to the second".

just my two cents...

[dazz]

Jason, I wish commercial sites treated their "clients" as well as you treat us. You are busy reassuring us on functionality and uptime when commercial site would usually just pretend the problem didn't happen.

'Thoughts and Jason...THANKS!! Anything that I can buy through an affiliate link I will.

dazz

[Jason Dunn]

make a static frontpage instead of a dynamic one. these page may be refreshed every 5 minutes and after somebody posted some news

Yes, that's exactly the tactic we're going to use to approach this... :-)

[Jason Dunn]

Jason, I wish commercial sites treated their "clients" as well as you treat us. You are busy reassuring us on functionality and uptime when commercial site would usually just pretend the problem didn't happen.

Thanks, I take that as a very strong compliment - ultimately I feel like all the readers are my "clients", and I feel like I have an obligation to give you a great service. Your support is appreciated!

[Venturello]

Jason, I wish commercial sites treated their "clients" as well as you treat us. You are busy reassuring us on functionality and uptime when commercial site would usually just pretend the problem didn't happen.

Thanks, I take that as a very strong compliment - ultimately I feel like all the readers are my "clients", and I feel like I have an obligation to give you a great service. Your support is appreciated!

Amen to both!! :D Jason if you ever come by Barcelona, let me know I'll invite you a Sangria over a Paella and show you the sights (and maybe, maybe talk a bit about pocketpc's?) :idea:

[heov]

Just a sugestion:

i've noticed now that you have limited the number of frontpage posts, and not all the posts from one day are on the front page. I think what would be best is to just include the headlines for the earliest posts that day at the bottom of the page. For example, you show the last 10 posts or so, and then another 5 are just displayed as headlines (like your archive page). That should reduce some load, shouldn't it? btw, an example site is like www.pdahandyman.com

BTW, what things are you thinking about removing?

[bbarker]

If the caching doesn't do enough to resolve things, some of the stuff in the right column of the home page could go. I do use some of them, such as last 10 posts and top stories. But I would rather see more news items on the front page if possible. The scroll off way too soon these days.

On another (related?) topic, any progress toward a mobile version of the forums? Or at least a way to click on the comments link and see a mobile-formatted version of what's already posted, if not the ability post additional comments.

Just wondering. We love you and all you do!

[Ed Hansberry]

Guys (and gals) - just because you don't use stuff doesn't mean others don't. Lots of this stuff came in as suggestions from users, well, except the ads. ;)

Let's see what Jason and his guru's can do with caching and static areas before removing things. BTW, if I had to guess, Jason is already thinking with this cache/static sections how he can put even more on the front page. :lol:

[Brad Adrian]

...these page may be refreshed every 5 minutes...
Just as a frame of reference, it wasn't that long ago that I would set AvantGo to synch my PPC Thoughts channel once or twice a day, and I was always sure to have all the freshest content. It is a tremendous tribute to Jason that he gets so much new stuff posted (and generates enough discussion) that we're talking about refreshing every few minutes.

You've always been my hero, Jason! And, I have absolutely no complaints with the way you run the site. Keep up the great work!

[Jonathon Watkins]

Well, it's kind of nice to have a problem of over-popularity. :D I only just noticed these thoughts about the server problems. I have been having a bunch of problems getting through to PPC thoughts recently - with lots of error messages etc. Hopefully we will be back to firing on all cylinders with not much functionally loss. I REALLY like the new look and the things you can do. I hope the caching will sort things out one and for all (until PPCthoughts grows again :wink: ).