RUSSELL BEATTIE -- Russ Beattie survived an attack on six of his websites. But it was messy and there are still unanswered questions about the source of the initial hack. The short version of Russ' ruined weekend is good reading for any server administrator.
Details from Russ:
- The hacker got in to my machine sometime early Friday morning Pacific time. He left some screed in Portugese that I didn't bother translating.
- Mike called me and woke me up after Elle noticed the hacked page. The Mobitopia guys were also contacting me, though via IM and SMS, which were a little less effective at that time of the morning in getting me up.
- I jumped on the server immediately and the guy was still there doing a bunch of stuff - or at least he had processes connected. He was in the middle of deleting *all* the logs on the server, so I have no idea where he came from (probably another hacked machine somewhere, so no biggie).
- I started shutting down services but there were all these scripts that would re-start stuff.
- I finally just rebooted the server.
- When it came back up, I didn't have access to the machine, ssh wouldn't log me in.
- I waited an impatient hour for ServerMatrix to go check out what was going on.
- After some back and forth, he sent a message back to me (via a message board no less) telling me that someone had logged in (was it me? No.).
- I told them to get out to the farm and rip the fucking machine out of the fucking wall if they needed to. (I was a bit tense at this point).
- When the box was down I could recover whatever was left. Which was actually a lot.
- All the hacker did was change my home pages on all my sites. This is quite the achievement considering I have six sites in my own home-grown setup running on Tomcat.
- He was obviously on the site long enough to figure out what was running, and figure out how to change all the home pages.
- All my email files were intact, MySQL, images, etc. but what did he *do* to them? Did he copy off all the private emails I read/sent in the past year from my IMAP store? Did he leave a back-door in one of my websites? Once the co-lo turned the machine on again, and restricted access to my home machine.
- I copied everything I could off the server and they did a complete format/OS reload.
- By Friday night I was starting to get things up and running again.
- Thanks to Matt who hand-held me while Debian's email was giving me fits.
- Thanks to Diego who went out to Bloglines and recovered the past month's posts and comments, and formatted them in an XML file I could just import into my DB.
- Thanks to Debian for easy-maintainability.
- Mac OSX is a killer Unix terminal. It *is* Unix, I know... a really nice one. With several terminal windows running, Expose, Transmit, and TextWrangler, I didn't miss my Windoze box at all.
- Expose is a *killer* feature, especially if you have a bunch of terminal windows up. It just rules.
How did the guy get in?
- No idea. The logs were gone. My best guess is a PHP CLI script I had running which allowed a Flash IRC app to re-route through my server to the freenode IRC servers. It was probably running as root and hackable as hell.
- I've also been playing with Apache and PHP 5 lately, so that was running on port 8080, and I really hadn't made any effort to secure it.
- It could have been any number of exploits out there that I never bothered to patch, or it could've been a bad password. We'll never know.
- It was my fault for not maintaining my site better. Hopefully this new setup is more secure, enough to deter another attack for a while at least.
Lessons Learned.
- Back up your data, NOW. I backed up my server last month, but the files were incomplete and a freakin' mess.
- Don't just back up, do it cleanly and in an organized, easy-to-find manner.
- Re-check your security. I've got a few more things to clean up and harden myself and I've been banging at the server all weekend.
Russ Beattie: "Fuck, is tomorrow Monday?"
tag: opsec x_ref125pr
