We're Back...And Here's Our Tale of Woe

[Jason Dunn]

After what is, I believe, the longest down-time in the history of these sites, we're back and live. Such a long window of down-time deserves an explanation more detailed than a haiku (did anyone notice the first version of the haiku wasn't actually a proper haiku? And this is from the guy who just got back from Japan!), but this explanation has to reach back a few years to make complete sense.<br /><br />We've been running on the same main machine for web and database serving for several years - a dual Xeon box from ev1 (we also had a second box for email and image serving). Long story short(er), it was largely unmanaged - I got help for specific problems when I needed it, but by and large it was completely left alone for almost four years. Yeah, that's very bad news from a security standpoint. I'm NOT a server guy, so I think I buried my head in the sand and hoped for the best. Not a good plan, right? Part of the problem was that the OS was old and what we really needed was to move to a new box with a new OS - but I didn't want to do that until all the sites were on vBulletin and I knew how much hardware I needed (phpBB is much more resource intensive than vBulletin is). Getting from phpBB to vBulletin took, literally, almost two years of development work - we had to develop a new CMS from scratch, something flexible and powerful enough to be used for years to come.<br /><br />Somehow, miraculously, over the years we never had major problems on our main server (likely because it had no control panel software to hack) - until December of last year. A few days after my <a href="http://www.jasondunn.com/the-world-is-a-darker-place-today-644" target="_blank">dear friend Crystal passed away</a>, and during the weekend when I was preparing for her funeral (I was doing some video/photos stuff for it), our secondary server (email/images) got hacked via Cpanel. So while grieving her loss I was also dealing with having huge email problems and worrying about losing data. Over the next couple of days, Jorj was moving that server to a virtual machine on our main server. Incredibly, as if the universe itself was punishing me, the morning of the funeral day <strong>our other server got hacked</strong> and our install of phpBB was breached several times. No data was lost, but we had spammers and virus/spyware distributors trying to hijack our forums. We fought them off, and closed the holes as best we could. I can't explain how grateful I am to <a href="http://www.jorj.org" target="_blank">Jorj</a> and <a href="http://www.janak.net" target="_blank">Janak</a> for their help - Jorj in particular spent a great deal of time working on the server in between celebrating Christmas with his family (and this is all volunteered time). His selflessness was/is awe-inspiring. <MORE /><br /><br />The problem with being hacked is that once it happens, you're never entirely sure what they left behind to hack you later with - the best recourse of action is to &quot;burn the box&quot;, meaning to abandon the OS and start fresh. This is why I was never able to explain our holiday 2007 down-time to all of you - you never want to say &quot;Oh, we were hacked&quot; unless you're sure you're ready to withstand another attack - and we weren't. We needed a new server anyway, so we did an emergency move from our old server (which now contained two servers) to our new server purchased from The Planet. Because it happened so fast, and because we were still planning on migrating Pocket PC Thoughts to vBulletin, we didn't want to set up phpBB on the new server - so we ran our two previously hacked servers each inside a virtual machine (VM) on the new server.<br /><br />Things seemed to be OK for a while, and we successfully migrated Pocket PC Thoughts to vBulletin - all still on the same unstable and wobbly VM. We migrated our email/image server to a fresh VM, and were making preparations to do the same with our main web/database server now that we were 100% on vBulletin. I asked the team to work on it while I was in Japan, and we'd move soon when I got back. I returned home on Friday the 4th, and the morning of Saturday the 5th I upgraded vBulletin to the latest version that had just come out that day or perhaps a day earlier. Things were going fine - or so I thought.<br /><br />The next day, Sunday night the 6th of April, I was driving home with my wife from a family dinner, and Jon Westfall called my mobile phone: &quot;We've been hacked - bad.&quot; So I rushed home and saw that our vBulletin forums had been hacked. It turns out that our install of vBulletin had been hacked prior to moving Pocket PC Thoughts over - and we didn't know it because the hacker chose to wait. A scramble ensued, the almost-haiku went up, and the team of Janak, Jorj, Darius, Fabrizio and myself struggled to figure out what happened and how to fix it. The hack was deep and wide - and specifically targeted at our vBulletin database. We initiated an emergency move over to a fresh VM on the new server Monday morning, but the concerns over what happened to our vBulletin database remained. For a litany of reasons we didn't have a very recent database backup, so we had to take the slow and painful steps of seeking out every possible place the hacker could have gotten into. With some great help from <a href="http://www.SEOvB.com" target="_blank">David from SEOvB.com</a>, last night we managed to get things fixed up - though we lost our Pocket PC Thoughts forum template (style) and the developer I paid to create it didn't have a copy of his work...<sigh>.<br /><br />So here we are. We're on two fresh VM installs, managed by Jorj, and one some fast hardware: a Quad-Core 2.13 GHz Kentsfield Xeon 3210 CPU, 8 GB of RAM, and some decently fast 500 GB hard drives (though we might need to upgrade these later). Now that we've jettisoned phpBB, and our old VM, things around here should be faster - though Jorj did want me to point out that because we're essentially starting over with this new VM, performance tuning will need to be done before we're really optimized. But it sure seems faster to me already!<br /><br />There's a lot of work left to be done, but that's the story as to why we've been down since Sunday night. I'm much more confident now in the server we have, and the people managing it (instead of yours truly), so I believe we're going to have great uptime from now on. The Pocket PC Thoughts migration happened in a rush, so there are things that still need fixing there, including launching some subscriber features. Stay tuned, and thank you for your support - it's been a rough three months, but I believe the worst is behind us now and it's going to be a great remainder of the year for the Thoughts Media communities...<br /><br />Sincerely,<br />Jason Dunn<br />Thoughts Media Inc.</sigh>

[RobrechtV]

Glad you're back online! :-)

[Jorj Bauer]

After what is, I believe, the longest down-time in the history of these sites, we're back and live ... Jorj in particular spent a great deal of time working on the server ...

Aww, shucks. (blush)

Seriously, I'm glad to keep a great resource like this running. Jason, it's in my best interest to keep you going -- where else am I going to get all this great news?? It's like having my own private news team!

On the other hand, my wife would probably suggest a "donations to the wife of the guy that's always busy" site. So go take a look at her very awesome christmas baking website (http://www.christmas-baking.com), make yourself the recipies that keep my waistline expanding, and think of her. ;)

[nosaturn]

I'm very happy to have Thoughts back. after all these years, I got scared when I couldn't see ya'll!

-Brian Haley

[Happyman]

I'm glad for your come back. This site one of the world on my data information update.

[jaxim]

I'm so glad to see both the Digital Home & Zune Thoughts websites back up!!
:-)

[stevew]

I've been coming to this site daily since the beginning and I got kind of an empty feeling inside when it was gone. :)

[asif2020]

Regardless of the issue, I'm glad it was resolved. Good to have you back!

[paschott]

I feel for you with the mess that's been going on, but am glad that you're back up and running. Shame you can't easily track down the hacker(s) and initiate some form of justice there, but I realize that's much easier said than done.

Hope that this is the last of it and looking forward to the future of the sites.

-Pete

[JonnoB]

My heart skipped a couple of beats with the anxiety of PPCThoughts withdrawals. Thanks to all who helped Jason and this site get back on its virtual legs.

[cmchavez]

Very sorry to hear about the hack, but also glad to hear that all the midnight oil burned will make for a more secure platform. Helluva welcome back from vacation, but here's to better times in the future!

[Alsicole]

Sorry to hear you've been targetted, but glad to see you're back up and running!

Alison

[makicr]

Good to see you guys are back up. I have started going through withdrawals.

[jgrnt1]

Glad to see you back up and running. I knew you were in trouble when I went to the forums a few days ago and saw that they were hacked.

I didn't see any political messages or statements, so I assume the hacker(s) just wanted to prove they could do it. I have a real problem grasping the hacker mentality. I could at least understand the guerilla mentality of it if there were political statements involved, but I've never understood the destructive "look what I can do" mentality. It's probably some bullied, pimple-faced kid lashing out at a cruel world, or it could just be the hacker is compensating for something, like guys who buy Corvettes. ;)

Either way, I hope he/they are reading this and know that I think so little of them that they're not even worth getting angry at. I can only get mad at people I respect. The rest aren't worth the effort.

[hawkeye]

Welcome back.

This site has been a daily read since I found the site through my Ipaq 3670 in '00 or '01. PPCT is my favorite site but Jason has developed a great stable of sites in Thoughts Media.

Thank the Lord you are back. You were missed.

[gstenger]

I'm glad to see that you were able to get it back up and running. I've missed your sites the last few days. Being a developer myself and a part-time (involuntary ;)) server admin, I feel your pain. Thanks for all of your hard work!

[Jason Dunn]

Shame you can't easily track down the hacker(s) and initiate some form of justice there, but I realize that's much easier said than done.

I have a somewhat vengeful personality, so that certainly appeals to part of me, but I also believe in eternal justice - that everyone is called someday to answer for their actions in life. This hacker will one day have to answer for what he's done.

[amesskyway]

Welcome back!

[Reid Kistler]

Even though my X30 - running 2003SE - is so OLD that an ever increasing number of new apps won't even run on it anymore, PCT is still a daily stop.

Am glad you are back, and hope the Hacker(s) are done....

rkistler / se mi

[onlydarksets]

I had a "Pocket PC Passion" flashback there...

[Jason Dunn]

I had a "Pocket PC Passion" flashback there...

I've been having that flashback a LOT the past few days...trust me! :eek:

[cjhpe]

Welcome back. As I visit all of the sites at least once a day (usually more like 2 or 3 times a day) I was going through withdrawals. Not that you haven't had plenty on your plate to think about but what is the status of the re-launch contests??? I am hoping to win something cool to make my last few days better. I too had a server hacked last week and have been spending all of this week making sure none of my other servers suffer the same fate. I am a little weary of it all. Thanks J...

[onlydarksets]

I've been having that flashback a LOT the past few days...trust me! :eek:
Well, at least you made a triumphant return!

[Rocco Augusto]

Woohoo! We're back online! I was starting to get withdraw :D

[phreaker18]

Well im just glad that PPCT is back .... i was a nervous wreak .... i emailed Darius but never got a reply and that made things even worse ... now i was watching a movie n suddenly out of nowhere i had this urge to see if PPCT was back up and i was so relieved to see the site up and running ....

i hope we never experience such a situation ever again .... MAY THE FORCE BE WITH US !!:rolleyes:

[Jason Dunn]

i hope we never experience such a situation ever again .... MAY THE FORCE BE WITH US !!:rolleyes:

I can tell you that I've learnt many valuable lessons from this whole situation and I'm going to do everything in my power to avoid anything like it again...:cool:

[wocket]

Glad Thoughts is back. I knew you had a fantastic team behind you!

Being hacked is a painful experience. I've been there. I've spent many a day checking server logs and directories almost to the point of paranoia. But it's still totally worth it.


All the best

[DaleReeck]

"and the developer I paid to create it didn't have a copy of his work...."

You're kidding. Wow. No offense*, but that developer doesn't seem very conscientious. I hope it was just someone who does this on the side and not a professional.


* usually when someone says "no offense, but...", they are about to offend :D

[Jason Dunn]

You're kidding. Wow. No offense*, but that developer doesn't seem very conscientious. I hope it was just someone who does this on the side and not a professional.

Yeah, I wasn't very impressed either. :confused:

[mmidgley]

so was any user data stolen by the hacker?

i suppose my email address and thoughts password are all that could be had, but that's still something i care about.

i'm glad you guys could work through this successfully. :)

m.

[bmurphy]

This site has been my go-to site for all my Ppc problems- and it's good to know that it's still there as a wonderful resource!

[virain]

It's nice to have you back, guys! Now let's back to the news!

[martin_ayton]

This morning was a good one: No 404 notice on the mobile site but the real McCoy back up and running :D Thanks for all your efforts guys.

[iant54]

Welcome back!

Yesterday, on the thread on Win4Mobile, I posted:

The Pocket PC Thoughts Store (Windows Mobile Software on Pocket PC Thoughts Store (http://software.pocketpcthoughts.com/homeSoftware.asp)) is still available, and at the foot are hyperlinks to various websites, including some of the editors' blog pages (Jason's is not working, and Darius' seems to no longer be registered to him).

<EDIT>Though, the blogs haven't been updated in the last day or so.

Does Darius know about his blog?

[Stinger]

Glad to have you back. :)

[Jason Dunn]

Not that you haven't had plenty on your plate to think about but what is the status of the re-launch contests??? I am hoping to win something cool to make my last few days better.

Hopefully you saw my post yesterday about this? The contests will remain open until Friday then I'll start the drawings...

[Jason Dunn]

so was any user data stolen by the hacker?

It's certainly possible - I should make a post about that actually...

[Jason Dunn]

Welcome back! Yesterday, on the thread on Win4Mobile, I posted...

Ah, good catch. That's all controlled by MobiHand - I'll just ask them to remove the links actually (or update them at the very least).

[William]

Yeah, I wasn't very impressed either. :confused:

Nor am I. :rolleyes:

For a litany of reasons we didn't have a very recent database backup

Anyway, welcome back.

[Heatwave316]

I'm soooooo happy to have all the 'Thoughts' back. I was going through withdraw!!!

[Darius Wey]

Does Darius know about his blog?

Nice catch. The domain (dariuswey.com) is still registered to me (actually, I should say "always will be registered to me"; I don't plan to give it up to another Darius Wey anytime soon). :D

It is sans a blog at the moment, but I'm working on it - or at least just redesigning the whole page so it doesn't look so bland. :)

[Jason Dunn]

Nice catch. The domain (dariuswey.com) is still registered to me (actually, I should say "always will be registered to me"; I don't plan to give it up to another Darius Wey anytime soon).

Actually, he was referring to the footer on our software store which linked to your old blogspot account. The entire footer has now been removed from the software store - that seemed better than me trying to keep it updated via emails to MobiHand. :rolleyes:

[Thinkingmandavid]

Howdy and good work Jason and to the entire team as well!!!

I am glad your back and after reading your message, can understand you all have been working really hard while enduring stress. Good speed.

[haveblue128]

;)My kudos for getting though what sounds like a rough patch with professionalism. You and your collleages responded in a speedy manner to the tech problems that beseighed Though Media. As is often the case, when it rains, it pours in the Dept of Bad News and the death of a friend must have made things all the more difficult. My thanks to the entire crew,
BethR