On thursday evening I had a couple of hours to spare, so I’ve SSHed to my server (yours truly littlesvr.ca) to get the Apache logs from the last couple of months. I do this now and then because I like seing the ammount of traffic going up every month. But this time it was not going to be a gratifying experience. The first command I ran was (as usual) ‘ls’:
andrew@littlesvr:/$ ls /bin/ls: unrecognized prefix: do /bin/ls: unparsable value for LS_COLORS environment variable bin/ dev/ home/ media/ opt/ root/ srv/ sys/ usr/ boot/ etc/ lib/ mnt/ proc/ sbin/ svn/ tmp/ var/
A second of confusion, but I knew then and there my box has been hacked. You see I’m running Slackware because weird crap like this (‘ls’ broken) doesn’t happen on Slackware, and though I denied it for a few minutes (oh maybe it’s a terminal problem, oh maybe I deleted something by mistake) soon enough I’ve had ample evidence. I’ve looked at /bin/ls in ‘vi’, which opens binaries as a hex editor. Didn’t see anything obviously wrong there. I’ve examined all the profile files, and those all looked normal. I grepped everything I could think of for ‘do’, and didn’t find a problem. Then I had a look at /var/log/messages, and found lots and lots of lines of the form:
Jan 7 00:13:30 littlesvr in.identd[7245]: reply to 24.165.1.229: 59167 , 21 : USERID : OTHER :0
As I was complaining about this in the #seneca IRC channel, I thought it would be worth while seing just how many of these strange lines are in the log. A quick ‘grep | wc -l’ gave me a staggering number, over 325k. Then I ran ‘ps aux’:
root@littlesvr:/etc/rc.d# ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu,comm STAT EUID RUID TT TPGID SESS PGRP PPID PID %CPU COMMAND S 0 0 ? -1 1 1 0 1 0.0 init SW 0 0 ? -1 1 1 1 2 0.0 migration/0 RWN 0 0 ? -1 1 1 1 3 0.0 ksoftirqd/0 SW< 0 0 ? -1 1 1 1 4 0.0 events/0 SW< 0 0 ? -1 1 1 1 5 0.0 khelper SW< 0 0 ? -1 1 1 1 6 0.0 kthread SW< 0 0 ? -1 1 1 6 40 0.0 kblockd/0 SW< 0 0 ? -1 1 1 6 41 0.0 kacpid SW< 0 0 ? -1 1 1 6 95 0.0 ata/0 SW< 0 0 ? -1 1 1 6 96 0.0 ata_aux SW< 0 0 ? -1 1 1 6 97 0.0 ksuspend_usbd SW< 0 0 ? -1 1 1 6 100 0.0 khubd SW< 0 0 ? -1 1 1 6 102 0.0 kseriod SW< 0 0 ? -1 1 1 6 124 0.0 kswapd0 SW< 0 0 ? -1 1 1 6 125 0.0 aio/0 SW< 0 0 ? -1 1 1 6 783 0.0 scsi_tgtd/0 SW< 0 0 ? -1 1 1 6 814 0.0 kcryptd/0 SW< 0 0 ? -1 1 1 6 836 0.0 reiserfs/0 S< 0 0 ? -1 900 900 1 900 0.0 udevd SW< 0 0 ? -1 1 1 6 1853 0.0 kpsmoused S 0 0 ? -1 1927 1927 1 1927 0.0 klogd S 1 1 ? -1 2170 2170 1 2170 0.0 rpc.portmap S 99 99 ? -1 2174 2174 1 2174 0.0 rpc.statd S 0 0 ? -1 2195 2195 1 2195 0.0 ntpd S 0 0 ? -1 2200 2200 1 2200 0.0 acpid S 81 81 ? -1 2208 2208 1 2208 0.0 dbus-daemon S 82 82 ? -1 2213 2213 1 2213 0.0 hald S 0 0 ? -1 2213 2213 2213 2214 0.0 hald-runner S 82 82 ? -1 2213 2213 2214 2223 0.0 hald-addon-acpi S 0 0 ? -1 2229 2229 1 2230 0.0 crond S 2 0 ? -1 2232 2232 1 2232 0.0 atd S 0 0 ? -1 2235 2235 1 2235 0.0 saslauthd S 0 0 ? -1 2235 2235 2235 2236 0.0 saslauthd S 0 0 ? -1 2235 2235 2235 2237 0.0 saslauthd S 0 0 ? -1 2235 2235 2235 2238 0.0 saslauthd S 0 0 ? -1 2235 2235 2235 2239 0.0 saslauthd S 0 0 ? -1 2376 2376 1 2377 0.0 python S 0 0 ? -1 2379 2379 1 2379 0.0 svnserve S 1003 1003 ? -1 2381 2381 1 2381 0.0 mailmanctl S 0 0 vc/1 2382 2382 2382 1 2382 0.0 agetty S 0 0 vc/3 2384 2384 2384 1 2384 0.0 agetty S 0 0 vc/4 2385 2385 2385 1 2385 0.0 agetty S 0 0 vc/5 2386 2386 2386 1 2386 0.0 agetty S 0 0 vc/6 2387 2387 2387 1 2387 0.0 agetty S 1003 1003 ? -1 2381 2381 2381 2398 0.0 python S 1003 1003 ? -1 2381 2381 2381 2399 0.0 python S 1003 1003 ? -1 2381 2381 2381 2400 0.0 python S 1003 1003 ? -1 2381 2381 2381 2401 0.0 python S 1003 1003 ? -1 2381 2381 2381 2402 0.0 python S 1003 1003 ? -1 2381 2381 2381 2403 0.0 python S 1003 1003 ? -1 2381 2381 2381 2404 0.0 python S 1003 1003 ? -1 2381 2381 2381 2405 0.0 python S 0 0 vc/2 2422 2422 2422 1 2422 0.0 agetty SW 0 0 ? -1 1 1 6 3220 0.0 pdflush S 0 0 ? -1 3339 3339 1 3339 0.0 inetd S 0 0 ? -1 3508 3508 1 3508 0.0 httpd S 99 99 ? -1 7245 7245 1 7245 0.0 in.identd S 80 80 ? -1 3508 3508 3508 7754 0.1 httpd S 80 80 ? -1 3508 3508 3508 7755 0.0 httpd S 1000 1000 ? -1 8112 8112 3339 8112 0.0 imapd S 1002 1002 ? -1 8113 8113 3339 8113 0.0 imapd S 1004 1004 ? -1 8114 8114 3339 8114 0.0 imapd S 1007 1007 ? -1 8115 8115 3339 8115 0.0 imapd S 0 0 ? -1 8281 8281 25049 8281 0.0 sshd S 1000 1000 ? -1 8281 8281 8281 8292 0.0 sshd S 1000 1000 pts/2 11949 8293 8293 8292 8293 0.0 bash S 0 0 ? -1 8760 9062 1 9063 0.0 ddclient S 0 0 ? -1 8760 9079 1 9084 0.0 pppoe-connect S 0 0 ? -1 9345 9345 1 9345 0.0 sendmail S 25 25 ? -1 9348 9348 1 9348 0.0 sendmail S 0 0 pts/2 11949 8293 10005 8293 10005 0.0 bash S 0 0 ? -1 10377 10377 25049 10377 0.0 sshd S 1000 1000 ? -1 10377 10377 10377 10383 0.0 sshd S 1000 1000 pts/4 10420 10384 10384 10383 10384 0.0 bash S 0 0 pts/4 10420 10384 10420 10384 10420 0.0 bash S 80 80 ? -1 3508 3508 3508 10598 0.3 httpd S 80 80 ? -1 3508 3508 3508 10599 0.2 httpd S 80 80 ? -1 3508 3508 3508 11510 0.0 httpd S 80 80 ? -1 3508 3508 3508 11511 0.0 httpd S 80 80 ? -1 3508 3508 3508 11512 0.0 httpd R 0 0 pts/2 11949 8293 11949 10005 11949 0.0 ps S 0 0 ? -1 2379 2379 2379 15669 0.0 svnserve S 80 80 ? -1 3508 3508 3508 19216 0.1 httpd S 80 80 ? -1 3508 3508 3508 22340 0.1 httpd S 0 0 ? -1 25049 25049 1 25049 0.0 sshd S 0 0 ? -1 25295 25295 1 25295 0.0 syslogd S 0 0 ? -1 25316 25334 1 25345 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 25349 0.1 ftp_scanner S 0 0 ? -1 25316 25334 1 25380 0.5 ftp_scanner S 0 0 ? -1 25316 25334 1 25383 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 25385 0.4 ftp_scanner S 0 0 ? -1 25316 25334 1 25441 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 25444 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 25446 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 25627 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 25632 0.4 ftp_scanner S 0 0 ? -1 25316 25334 1 25732 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 25740 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 25741 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 25836 0.1 ftp_scanner S 0 0 ? -1 25316 25334 1 25846 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 25947 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 25949 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 25950 0.4 ftp_scanner S 0 0 ? -1 25316 25334 1 26052 0.1 ftp_scanner S 0 0 ? -1 25316 25334 1 26054 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 26157 0.2 ftp_scanner S 0 0 ? -1 25316 25334 1 26158 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 26262 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 26480 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 26586 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 26682 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 26689 0.1 ftp_scanner S 0 0 ? -1 25316 25334 1 26793 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 26795 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 26796 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 26894 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 26900 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 26901 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 26902 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 26998 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 27003 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 27005 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 27007 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 27107 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 27118 0.0 ftp_scanner S 0 0 ? -1 25316 25334 1 27221 0.3 ftp_scanner S 0 0 ? -1 25316 25334 1 27222 0.0 ftp_scanner S 0 0 ? -1 28433 28433 9084 28433 0.0 pppd S 99 99 ? -1 28433 28433 28433 28434 6.5 pppoe S 80 80 ? -1 3508 3508 3508 29836 0.1 httpd S 0 0 ? -1 25073 30993 1 31000 0.0 mysqld_safe S 27 27 ? -1 25073 30993 31000 31033 0.0 mysqld SW 0 0 ? -1 1 1 6 31163 0.0 pdflush
Ayayay. I do not (for those of you who think too much) habitually run anything called ftp_scanner. I found in the ps man page an example to help me show the parent PIDs, and sure enough – all the ftp scanners had 1 (a.k.a. init) for the parent. Which means that the hacker had root access, and likely the box has been rooted.
I’ve hesitated for 10 minutes. The ftp scanner had to stop, rootkit had to be removed, the hacker locked out. But littlesvr.ca is not a toy, half my life is on it – all my email, all my computer work (svn), revenue-generating pages such as ISO Master, and less popular pages that would lose the precious little search engine ranking they had if they went offline. But it had to be done. I braced myself for a 36 hour shift, and:
root@littlesvr:~# halt
By this time my brain was overwhelmed. Too much excitement at once. The server had to be cleaned up and brought back to life in a hurry, but first I had to know how it got compromised to begin with – no point in resuscitating it only to have it hacked again two days later.
But this is a long story, and I have yet to see the ending. This post will be the first of a short series, so come back later if you want to know more.