First things first – my server had to be brought back up. But bringing it back up was not a matter as simple as reinstalling Slackware and plopping the home directories back in. So the first thing that needed to be done was to figure out how the server’s been compromised.
The quickest way to do this, I figured, was to do both at the same time. I’d get the data off the server, reformat the hard drive, start reinstalling the operating system, and during that time would analyse the backed up files for evidence. Just to be throrough I wanted to keep a byte-by-byte copy of the hard drive – that may contain clues that simple copies of the files would lose, and using the image I could bring a clone of the hacked server back up in a controlled environment – exactly the same minus any processes that were running.
The command was easy enough, ran from the shell of a Slackware 12.2 installation CD:
dd if=/dev/hda1 | ssh 10.0.0.1 dd of=/media/backup/hacked/hda1.dd dd if=/dev/hda2 | ssh 10.0.0.1 dd of=/media/backup/hacked/hda2.dd
I’ve saved hda2 (the swap partition) just in case I ever found a tool to analyse contents of memory (unlikely to happen). Turns out the 80G image would take hours to transfer over, so I set the alarm clock for 2AM and went to sleep for a couple of hours, knowing I wouldn’t get any the next day.
The image hda1.dd (as any other image of a Linux-supported filesystem) can be mounted onto an existing system. I decided to go ahead and do that since nothing on the partition could damage a fresh system unless they found a bug in the filesystem driver and infected the FAT:
mount /media/backup/hacked/hda1.dd /media/hd -o loop
Looking around /media/hacked I quickly found something that got my attention: a file and a directory in /root/ that didn’t look familiar:
root@legrand-sw:/media/hd/root# ls -l heroina/ <stripped out some things> drwxr-xr-x 2 1003 users 304 2008-09-23 16:24 heroina/ -rw-r--r-- 1 root root 81920 2008-12-16 16:42 ketamin.tar root@legrand-sw:/media/hd/root# ls -l heroina/ total 92 -rwxr-xr-x 1 1003 users 82 2006-03-10 15:02 0-100* -rwxr-xr-x 1 1003 users 84 2006-03-10 21:02 100-200* -rwxr-xr-x 1 1003 users 84 2006-03-10 21:02 200-255* -rwxr-xr-x 1 1003 users 428 2009-01-07 23:44 dava* -rwxr-xr-x 1 1003 users 17145 2008-09-23 17:07 ftp_scanner* -rwxr-xr-x 1 1003 users 17246 2008-09-23 16:49 heroina.c* -rwxr-xr-x 1 1003 users 2094 2006-03-10 21:02 o* -rwxr-xr-x 1 1003 users 20313 2008-06-15 18:46 pass* -rwxr-xr-x 1 1003 users 119 2008-09-23 16:22 run* -rwxr-xr-x 1 1003 users 7490 2008-06-17 15:51 users*
There’s my ftp_scanner, and who would have thought, it even comes with source code. Well of course – who would use a binary to mess with an open source server? Yep, here’s heroina.c if you care to look at it, but basically it looks like general-purpose software:
Multi-thread FTP scanner v0.2.5 by Inode <inode@wayreth.eu.org>
users and pass are tiny dictionaries of common usernames and passwords, the 0-100 files just helpers for making ip addresses, o looked like the output from nmap. Not terribly sofisticated stuff, but interesting.
Of course this doesn’t come close to telling me how that software got on in the first place, but it does give some interesting clues.
The timestamps are really interesting, you could piece any number of stories looking just at those. See for example how ftp_scanner is 18 minutes older than heroina.c, that suggests it was built manually (what sort of automation takes 18 minutes to build one C file?). And ketamin.tar has been sitting there for more than 3 weeks, which is kind of depressing.
And look at the owner, 1003. That according to /media/hd/etc/passwd was mailman on the old littlesvr.ca. Well that’s not going back on the system in a hurry, and how was it able to write to /root/ anyway?
I already knew the web applications are the most vulnerable part of my system. I trust Slackware, I trust me configuring the running services, but I have no idea how many security problems the web apps on the system have. I haven’t read the source for Mailman, WordPress, SquirrelMail, or Roudcube, and they don’t come with Slackware – how am I supposed to trust them? Who vouched for them?
Anyway – by this time Slackware has been reinstalled and littlesvr.ca was sitting there in its vanilla state. I’ve decided to turn on sendmail and SVN – which would give me back two thirds of half my life, so I cautiously copied my home directories back. Apache was still off.
If you think this is the end of the story, you’re wrong. Come back later as I share more clues with you, I have at least one interesting thing left – plus there still are no certain conclusions about who and how and why did this.