Grumble Grumble

By Andrew Smith

Suspend and hibernate have been a sad story on Linux for a very long time. Typically the hardware makers are blamed, saying “oh they all do it differently none of it’s documented”. That may be true but it’s a bunch of horseshit anyway – hardly any of the hardware that works at all or perfectly in Linux is the same or is documented.

A couple of days ago I upgraded to Slackware 13.1. I left suspend as the last thing to set up, as it wasn’t working at all in 13.0 and barely working in 12.2. And you know what? It works now, it just works. I could hardly believe it. No messing with ACPI options, no hooks to remove my USB modules, no fancy keyboard configuration. I just press my FnF5 and it goes to sleep.

That was pretty impressive to begin with. And then I tried to hibernate. One try and I realised I needed an extra resume=/dev/sda5 in my lilo.conf. That one change.. and.. hibernate also works.

I am shocked. In a good way. Suspend is a must on a laptop and hibernate is really nice to have. Thanks, whomever fixed this!

For reference: I have an HP G50 and Slackware 13.1 comes with the kernel 2.6.33.4

By Andrew Smith

At some point in the distant past I’ve set up my own email server, running Sendmail for SMTP and the UWO imapd server for IMAP. Part of the setup process is creating certificates – so that you can have a secure tunnel for free, using self-signed certs.

Like a good boy I followed the instructions to the letter. Now I’m presented with a message box in Thuderbird saying “your certificate expired”. Eh? Why? Oh, I know – it’s because you’re supposed to have an expiry date on your own generated certs, because bla proper bla security bla I know better bla. I never got the bla bla part, never had any use for it. And now I ran into the problem I knew I would run into eventually.

Security, expiry? Did these guys think? I’ll tell you something – I can still get to my email using Thunderbird. It just puts up an annoying warning and makes me click continue every time. What sort of security is this expiration providing? If anything, security is worse now.

I must have created this setup a long time ago (likely 5 years) – because today I would never do something I believe is stupid just because someone “who knows better” said so. Oh I’m glad I made that decision.

By Andrew Smith

I’ve been running my more-or-less regular backup of littlesvr.ca, and found that yet again the online APNG assembler data directory is the one taking most of the time. Turns out that even though I cleared it a few months ago there is over 4GB of PNG files in there (half of that – 5800 APNG files).

That’s really interesting – it means APNG usage has actually grown in the last year. Partly that might be explained by ongoing support in Firefox, but surely that’s not the only explanation. I don’t know what’s really driving the growth. I just know it isn’t me.

On the one hand it’s great that people took over (e.g. Max Stepin has been diligently maintaining the APNG patch for well over a year now) but on the other hand it’s sad that the project survived without me – I guess I wasn’t that important :)

I have been feeling sad about such things lately. I always do my best to bootstrap a project and then pass it on to someone else to maintain. This way I can keep doing new and exciting things while expanding the breadth of my knowledge and skills, and increasing the impact I have overall. But damn it’s depressing when I’m in between projects. I feel useless – not wanted and not needed.

Should I just accept that there is only so much one can do in a lifetime, and specialise in some irrelevant field? Become normal? What a dreadful thought.

By Andrew Smith

There is a magic thing called gettext. It’s magic because it allows some random Joe on the internet with no experience in programming to translate your application into his language. He does this by editing a .po file, then sends it to you, and you just include that file in your build system.

Ok there is some plumbing you need to enable that, but really it’s worth it.

In one of my applications I followed the advice in the gettext manual and used plural forms for messages. That is rather than saying “%d file(s)” the string had two forms: “%d file” (if %d is 1) and “%d files” (if %d is not one). This is specified in the po file like so:

“Plural-Forms: nplurals=2; plural= n!=1;\n”

msgid “%d file”
msgid_plural “%d files”
msgstr[0] “”
msgstr[1] “”

But of course different languages have different singular/plural rules. For example in Russian 0, 1, 2, and 5 have different plural forms. That means the russian translator has to figure out how this whole system works in order to translate strings with plural forms.

These translators are volunteers. That they do it at all is amazing. Having them read the gettext manual (which doesn’t have a lot of examples like it should) is disrespectful and unproductive. You will get fewer translations of not-as-good quality if you use plural forms.

I know, I learned this the hard way. One of my programs uses plural forms and another doesn’t. I have spent way too much time explaining to people how this works, and I am sure some of them ignored what I said and just gave me whatever compiles.

And believe me figuring this out isn’t easy. I mentioned the russian example – honestly I have no idea what the rules are exactly (who ever thinks of these things?) and I speak russian, and I maintain the application. How can I ask Joe random to understand this?

There you go. Use plural forms at your own peril. The gettext manual lies, plural forms are not a good thing and the problems they cause are bigger than the one problem they solve. I have just got a new translation, and guess what – the two strings with plural forms are untranslated. So I have to get a headache explaining this yet again, while kicking myself in the head for buying into the gettext dogma.

By Andrew Smith

Here’s a stupid post, it’s been a while. I’ve upgraded Slackware from 12.2 to 13.0 and now it is messing with me. Believe it or not the stupid thing knows when I’m annoyed and pisses me off even more. Here’s how it works:

Sometimes my pointer starts jumping all around the screen, for no good reason. That’s not entirely unexpected, I have a crappy wireless mouse – but sometimes it gets on my nerves. When it does the first thing I do is stop moving and breathing, to see if it moves on its own or it goes nutz. No, it doesn’t do anything when I freeze.

So I take it easy, and use the mouse for a few seconds normally – but then it starts messing with me again. I slam my hand on the table, put a firm hand on top of the mouse, and deliberatly move it down – and it’s like it dares me to slide it all the way off the table, the pointer keeps jerking back up. Repeat, same thing.

Hardware problem? I thought so too, and replaced the wireless USB mouse with a wired PS2 mouse. Same problem.

You’ll never guess what it was. Turns out Slackware 13 treats my Microsoft Sidewinder steering wheel as a mouse. So when I press the pedals, it moves the mouse. When I deliberatly move the mouse down I at the same time (without thinking) press the gas pedal, which moves the pointer back up. Great stuff :)

By Andrew Smith

I got an email from someone who appears to be helping KDE out with switching their licences. I guess at some point the guys decided that nothing in KDE shall be licenced GPLv2 only, and that’s the request I got – change the licence to “GPL2 or 3″ or “GPL2 or later”.

Here are a couple of links describing the process:

http://techbase.kde.org/Policies/Licensing_Policy
http://techbase.kde.org/Projects/KDE_Relicensing

I’m not surprised to see only people names listed on the second page, no project names. I wonder how much software KDE plans to scrap just because of some overly idealistic dudes with big beards decided the new GPL is much better and must be incompatible with the old GPL.

My piece of code in there is bkisofs – a library for reading and writing ISO files which is being used by Ark (the KDE archive manager). The integration was not my doing, I only helped – but the library itself is mine. KDE won’t explode the funcionality is removed from Ark.

I haven’t replied to the request yet. On the one hand I don’t care if it’s licenced GPL2 or 3, the changes I recall don’t affect this particular software. On the other hand – I am opposed to the GPL3 as a matter of principle, and I don’t like bending over or getting strong-armed into doing things with my volunteer time.

And if I allow this, that will count as a contribution to GPL3 zealotry.

I’ll go review the GPL3 some time in the next few days. If anyone still reads my blog and has ideas one way or the other – please share.

By Andrew Smith

I’m marking the first assignment for the data structures and algorithms course. Two problems – a painful but well-defined majority element and the other a linked list. I’m reading this code and I’m getting a headache. I need to take a break every 30 minutes, it’s so hard.

The problem is I’m looking at the code and I know that it can’t possibly work. But it does. I try creating a scenario where it will fail as I expect it should, I fiddle with the memory arrangement on the stack, but the bloody things work anyway.

I know the right way to do it, but who says what I think is the right way really is? I’m not going to take marks off for thinking out of the box, even if that’s just evidence of not paying attention in class.

While it’s extremely unlikely that these weird ways to solve problems are any good in the real world, where other people have to read your code – I’ve read enough code in my life to know that whether it works is the ultimate quality metric, and readability is but an illusion.

Back to work.

By Andrew Smith

I left off with my server almost completely back up, but not yet Apache. I’ve had to make sure the web apps off the internet weren’t full of security holes before allowing access to them again.

Though this is my third post 4 days later – in real time it’s been less than 24 hours since I’ve discovered the ftp scanner. So I had some breathing time to do a half-decent job.

Most of all I suspected Wordpress. They come up with a new version every few months and every time claim it’s a security update, which makes me wonder if it’s ever secure. Luckily both ISO Master and Grumble Grumble are just stock wordpress installs with a theme over them. Everything else is stored in the database. So I was able to delete all the code except for the themes, install Wordpress 2.7 on both, and miraculously both websites came back up with no trouble whatsoever.

I have not reviewed the theme code or the contents of the databases – I assume the Wordpress guys were smart enough not to store executable code in either of those places. Either way it was a risk I was willing to take.

A third Wordpress website (the old Canvas3D blog) I’ve decided not to bother with and left it with chmod 0.

Three web apps down, two to go – SquirrelMail and Roudcube. I have both installed because I have more than one email account, so I have both open in browser tabs at once. I’ve decided to look around before enabling these. RoudCube first. Some way or another I stumbled upon a php file in the logs directory:

root@legrand-sw:/media/hd/home/www/htdocs/mail/logs# ls -al
total 129
drwxrwxr-x  2 andrew apache    120 2008-12-30 18:00 ./
drwxr-xr-x 10 andrew users     480 2008-08-15 10:25 ../
-rw-r--r--  1 apache apache  14985 2009-01-07 13:29 errors
-rw-r--r--  1 apache apache   7033 2009-01-07 13:26 sendmail
-rw-r--r--  1 apache apache 105643 2008-12-30 18:00 zq.php

Which is a really strange place to keep a php file, especially since the directory is writeable by Apache. See how zh.php is owned by apache? That’s really bad. And lookie here what was at the top of zq.php:

/******************************************************************************************************/
/*
/*                                     #    #        #    #
/*                                     #   #          #   #
/*                                    #    #          #    #
/*                                    #   ##   ####   ##   #
/*                                   ##   ##  ######  ##   ##
/*                                   ##   ##  ######  ##   ##
/*                                   ##   ##   ####   ##   ##
/*                                   ###   ############   ###
/*                                   ########################
/*                                        ##############
/*                                 ######## ########## #######
/*                                ###   ##  ##########  ##   ###
/*                                ###   ##  ##########  ##   ###
/*                                 ###   #  ##########  #   ###
/*                                 ###   ##  ########  ##   ###
/*                                  ##    #   ######   #    ##
/*                                   ##   #    ####   #    ##
/*                                     ##                 ##
/*
/*
/*
/*  r57shell.php -
/*                                               : http://rst.void.ru
/*        : 1.3 (05.03.2006)
/*~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*/
/*                                          : blf, phoenix, virus, NorD                  RST/GHC.
/*                       -
/*     rst@void.ru.                                  .
/*~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*/
/*  (c)oded by 1dt.w0lf
/*  RST/GHC http://rst.void.ru , http://ghc.ru
/*  ANY MODIFIED REPUBLISHING IS RESTRICTED
/******************************************************************************************************/

Isn’t that interesting.. I didn’t quite know what to think of the bug – on the one hand looks like the hacker put it there to taunt me, on the other hand RoundCube is a young open source project so who knows what the developer was smoking?

Didn’t take long to figure out though: both http://rst.void.ru and http://ghc.ru are hacker websites, though ghc.ru looks half-commercial. And the contents of the file (which I’ve had no time to analyse) clearly do something that isn’t nice. I’ve renamed the file to .txt and put it up for you to study if you care.

As with mailman (see previous post) I decided this service is not going to be reenabled. Maybe I messed up something during setup, but just as likely RoudCube was at fault. After all it was the first ’stable’ version, 0.1. And I’ll keep SquirrelMail off too for good measure, I’ll just install Thunderbird on all my machines.

So there you go. One clue in Mailman, one in RoudCube. Are they related or are they two separate hacking attempts? Were they run by script kiddies or serious hackers? Having some evidence that the ftp scanner was set up manually I tend to favour the latter possibility. And what of this zq.php? Was the logo there because it’s a general-purpose cracking tool, or because those groups actually had something to do with my server?

There’s lots of investigation left still. At this point my server is completely operational so I have lots of time. I do hope I find more clues, mystery solving turns out to be a really exciting hobby.

By Andrew Smith

First things first – my server had to be brought back up. But bringing it back up was not a matter as simple as reinstalling Slackware and plopping the home directories back in. So the first thing that needed to be done was to figure out how the server’s been compromised.

The quickest way to do this, I figured, was to do both at the same time. I’d get the data off the server, reformat the hard drive, start reinstalling the operating system, and during that time would analyse the backed up files for evidence. Just to be throrough I wanted to keep a byte-by-byte copy of the hard drive – that may contain clues that simple copies of the files would lose, and using the image I could bring a clone of the hacked server back up in a controlled environment – exactly the same minus any processes that were running.

The command was easy enough, ran from the shell of a Slackware 12.2 installation CD:

dd if=/dev/hda1 | ssh 10.0.0.1 dd of=/media/backup/hacked/hda1.dd
dd if=/dev/hda2 | ssh 10.0.0.1 dd of=/media/backup/hacked/hda2.dd

I’ve saved hda2 (the swap partition) just in case I ever found a tool to analyse contents of memory (unlikely to happen). Turns out the 80G image would take hours to transfer over, so I set the alarm clock for 2AM and went to sleep for a couple of hours, knowing I wouldn’t get any the next day.

The image hda1.dd (as any other image of a Linux-supported filesystem) can be mounted onto an existing system. I decided to go ahead and do that since nothing on the partition could damage a fresh system unless they found a bug in the filesystem driver and infected the FAT:

mount /media/backup/hacked/hda1.dd /media/hd -o loop

Looking around /media/hacked I quickly found something that got my attention: a file and a directory in /root/ that didn’t look familiar:

root@legrand-sw:/media/hd/root# ls -l heroina/
<stripped out some things>
drwxr-xr-x 2   1003 users   304 2008-09-23 16:24 heroina/
-rw-r--r-- 1 root   root  81920 2008-12-16 16:42 ketamin.tar

root@legrand-sw:/media/hd/root# ls -l heroina/
total 92
-rwxr-xr-x 1 1003 users    82 2006-03-10 15:02 0-100*
-rwxr-xr-x 1 1003 users    84 2006-03-10 21:02 100-200*
-rwxr-xr-x 1 1003 users    84 2006-03-10 21:02 200-255*
-rwxr-xr-x 1 1003 users   428 2009-01-07 23:44 dava*
-rwxr-xr-x 1 1003 users 17145 2008-09-23 17:07 ftp_scanner*
-rwxr-xr-x 1 1003 users 17246 2008-09-23 16:49 heroina.c*
-rwxr-xr-x 1 1003 users  2094 2006-03-10 21:02 o*
-rwxr-xr-x 1 1003 users 20313 2008-06-15 18:46 pass*
-rwxr-xr-x 1 1003 users   119 2008-09-23 16:22 run*
-rwxr-xr-x 1 1003 users  7490 2008-06-17 15:51 users*

There’s my ftp_scanner, and who would have thought, it even comes with source code. Well of course – who would use a binary to mess with an open source server? Yep, here’s heroina.c if you care to look at it, but basically it looks like general-purpose software:

Multi-thread FTP scanner v0.2.5 by Inode <inode@wayreth.eu.org>

users and pass are tiny dictionaries of common usernames and passwords, the 0-100 files just helpers for making ip addresses, o looked like the output from nmap. Not terribly sofisticated stuff, but interesting.

Of course this doesn’t come close to telling me how that software got on in the first place, but it does give some interesting clues.

The timestamps are really interesting, you could piece any number of stories looking just at those. See for example how ftp_scanner is 18 minutes older than heroina.c, that suggests it was built manually (what sort of automation takes 18 minutes to build one C file?). And ketamin.tar has been sitting there for more than 3 weeks, which is kind of depressing.

And look at the owner, 1003. That according to /media/hd/etc/passwd was mailman on the old littlesvr.ca. Well that’s not going back on the system in a hurry, and how was it able to write to /root/ anyway?

I already knew the web applications are the most vulnerable part of my system. I trust Slackware, I trust me configuring the running services, but I have no idea how many security problems the web apps on the system have. I haven’t read the source for Mailman, Wordpress, SquirrelMail, or Roudcube, and they don’t come with Slackware – how am I supposed to trust them? Who vouched for them?

Anyway – by this time Slackware has been reinstalled and littlesvr.ca was sitting there in its vanilla state. I’ve decided to turn on sendmail and SVN – which would give me back two thirds of half my life, so I cautiously copied my home directories back. Apache was still off.

If you think this is the end of the story, you’re wrong. Come back later as I share more clues with you, I have at least one interesting thing left – plus there still are no certain conclusions about who and how and why did this.

By Andrew Smith

On thursday evening I had a couple of hours to spare, so I’ve SSHed to my server (yours truly littlesvr.ca) to get the Apache logs from the last couple of months. I do this now and then because I like seing the ammount of traffic going up every month. But this time it was not going to be a gratifying experience. The first command I ran was (as usual) ‘ls’:

andrew@littlesvr:/$ ls
/bin/ls: unrecognized prefix: do
/bin/ls: unparsable value for LS_COLORS environment variable
bin/    dev/    home/   media/  opt/    root/   srv/    sys/    usr/
boot/   etc/    lib/    mnt/    proc/   sbin/   svn/    tmp/    var/

A second of confusion, but I knew then and there my box has been hacked. You see I’m running Slackware because weird crap like this (‘ls’ broken) doesn’t happen on Slackware, and though I denied it for a few minutes (oh maybe it’s a terminal problem, oh maybe I deleted something by mistake) soon enough I’ve had ample evidence. I’ve looked at /bin/ls in ‘vi’, which opens binaries as a hex editor. Didn’t see anything obviously wrong there. I’ve examined all the profile files, and those all looked normal. I grepped everything I could think of for ‘do’, and didn’t find a problem. Then I had a look at /var/log/messages, and found lots and lots of lines of the form:

Jan  7 00:13:30 littlesvr in.identd[7245]: reply to 24.165.1.229: 59167 , 21 : USERID : OTHER :0

As I was complaining about this in the #seneca IRC channel, I thought it would be worth while seing just how many of these strange lines are in the log. A quick ‘grep | wc -l’ gave me a staggering number, over 325k. Then I ran ‘ps aux’:

root@littlesvr:/etc/rc.d# ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu,comm
STAT  EUID  RUID TT       TPGID  SESS  PGRP  PPID   PID %CPU COMMAND
S        0     0 ?           -1     1     1     0     1  0.0 init
SW       0     0 ?           -1     1     1     1     2  0.0 migration/0
RWN      0     0 ?           -1     1     1     1     3  0.0 ksoftirqd/0
SW<      0     0 ?           -1     1     1     1     4  0.0 events/0
SW<      0     0 ?           -1     1     1     1     5  0.0 khelper
SW<      0     0 ?           -1     1     1     1     6  0.0 kthread
SW<      0     0 ?           -1     1     1     6    40  0.0 kblockd/0
SW<      0     0 ?           -1     1     1     6    41  0.0 kacpid
SW<      0     0 ?           -1     1     1     6    95  0.0 ata/0
SW<      0     0 ?           -1     1     1     6    96  0.0 ata_aux
SW<      0     0 ?           -1     1     1     6    97  0.0 ksuspend_usbd
SW<      0     0 ?           -1     1     1     6   100  0.0 khubd
SW<      0     0 ?           -1     1     1     6   102  0.0 kseriod
SW<      0     0 ?           -1     1     1     6   124  0.0 kswapd0
SW<      0     0 ?           -1     1     1     6   125  0.0 aio/0
SW<      0     0 ?           -1     1     1     6   783  0.0 scsi_tgtd/0
SW<      0     0 ?           -1     1     1     6   814  0.0 kcryptd/0
SW<      0     0 ?           -1     1     1     6   836  0.0 reiserfs/0
S<       0     0 ?           -1   900   900     1   900  0.0 udevd
SW<      0     0 ?           -1     1     1     6  1853  0.0 kpsmoused
S        0     0 ?           -1  1927  1927     1  1927  0.0 klogd
S        1     1 ?           -1  2170  2170     1  2170  0.0 rpc.portmap
S       99    99 ?           -1  2174  2174     1  2174  0.0 rpc.statd
S        0     0 ?           -1  2195  2195     1  2195  0.0 ntpd
S        0     0 ?           -1  2200  2200     1  2200  0.0 acpid
S       81    81 ?           -1  2208  2208     1  2208  0.0 dbus-daemon
S       82    82 ?           -1  2213  2213     1  2213  0.0 hald
S        0     0 ?           -1  2213  2213  2213  2214  0.0 hald-runner
S       82    82 ?           -1  2213  2213  2214  2223  0.0 hald-addon-acpi
S        0     0 ?           -1  2229  2229     1  2230  0.0 crond
S        2     0 ?           -1  2232  2232     1  2232  0.0 atd
S        0     0 ?           -1  2235  2235     1  2235  0.0 saslauthd
S        0     0 ?           -1  2235  2235  2235  2236  0.0 saslauthd
S        0     0 ?           -1  2235  2235  2235  2237  0.0 saslauthd
S        0     0 ?           -1  2235  2235  2235  2238  0.0 saslauthd
S        0     0 ?           -1  2235  2235  2235  2239  0.0 saslauthd
S        0     0 ?           -1  2376  2376     1  2377  0.0 python
S        0     0 ?           -1  2379  2379     1  2379  0.0 svnserve
S     1003  1003 ?           -1  2381  2381     1  2381  0.0 mailmanctl
S        0     0 vc/1      2382  2382  2382     1  2382  0.0 agetty
S        0     0 vc/3      2384  2384  2384     1  2384  0.0 agetty
S        0     0 vc/4      2385  2385  2385     1  2385  0.0 agetty
S        0     0 vc/5      2386  2386  2386     1  2386  0.0 agetty
S        0     0 vc/6      2387  2387  2387     1  2387  0.0 agetty
S     1003  1003 ?           -1  2381  2381  2381  2398  0.0 python
S     1003  1003 ?           -1  2381  2381  2381  2399  0.0 python
S     1003  1003 ?           -1  2381  2381  2381  2400  0.0 python
S     1003  1003 ?           -1  2381  2381  2381  2401  0.0 python
S     1003  1003 ?           -1  2381  2381  2381  2402  0.0 python
S     1003  1003 ?           -1  2381  2381  2381  2403  0.0 python
S     1003  1003 ?           -1  2381  2381  2381  2404  0.0 python
S     1003  1003 ?           -1  2381  2381  2381  2405  0.0 python
S        0     0 vc/2      2422  2422  2422     1  2422  0.0 agetty
SW       0     0 ?           -1     1     1     6  3220  0.0 pdflush
S        0     0 ?           -1  3339  3339     1  3339  0.0 inetd
S        0     0 ?           -1  3508  3508     1  3508  0.0 httpd
S       99    99 ?           -1  7245  7245     1  7245  0.0 in.identd
S       80    80 ?           -1  3508  3508  3508  7754  0.1 httpd
S       80    80 ?           -1  3508  3508  3508  7755  0.0 httpd
S     1000  1000 ?           -1  8112  8112  3339  8112  0.0 imapd
S     1002  1002 ?           -1  8113  8113  3339  8113  0.0 imapd
S     1004  1004 ?           -1  8114  8114  3339  8114  0.0 imapd
S     1007  1007 ?           -1  8115  8115  3339  8115  0.0 imapd
S        0     0 ?           -1  8281  8281 25049  8281  0.0 sshd
S     1000  1000 ?           -1  8281  8281  8281  8292  0.0 sshd
S     1000  1000 pts/2    11949  8293  8293  8292  8293  0.0 bash
S        0     0 ?           -1  8760  9062     1  9063  0.0 ddclient
S        0     0 ?           -1  8760  9079     1  9084  0.0 pppoe-connect
S        0     0 ?           -1  9345  9345     1  9345  0.0 sendmail
S       25    25 ?           -1  9348  9348     1  9348  0.0 sendmail
S        0     0 pts/2    11949  8293 10005  8293 10005  0.0 bash
S        0     0 ?           -1 10377 10377 25049 10377  0.0 sshd
S     1000  1000 ?           -1 10377 10377 10377 10383  0.0 sshd
S     1000  1000 pts/4    10420 10384 10384 10383 10384  0.0 bash
S        0     0 pts/4    10420 10384 10420 10384 10420  0.0 bash
S       80    80 ?           -1  3508  3508  3508 10598  0.3 httpd
S       80    80 ?           -1  3508  3508  3508 10599  0.2 httpd
S       80    80 ?           -1  3508  3508  3508 11510  0.0 httpd
S       80    80 ?           -1  3508  3508  3508 11511  0.0 httpd
S       80    80 ?           -1  3508  3508  3508 11512  0.0 httpd
R        0     0 pts/2    11949  8293 11949 10005 11949  0.0 ps
S        0     0 ?           -1  2379  2379  2379 15669  0.0 svnserve
S       80    80 ?           -1  3508  3508  3508 19216  0.1 httpd
S       80    80 ?           -1  3508  3508  3508 22340  0.1 httpd
S        0     0 ?           -1 25049 25049     1 25049  0.0 sshd
S        0     0 ?           -1 25295 25295     1 25295  0.0 syslogd
S        0     0 ?           -1 25316 25334     1 25345  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25349  0.1 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25380  0.5 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25383  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25385  0.4 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25441  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25444  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25446  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25627  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25632  0.4 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25732  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25740  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25741  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25836  0.1 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25846  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25947  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25949  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25950  0.4 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26052  0.1 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26054  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26157  0.2 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26158  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26262  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26480  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26586  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26682  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26689  0.1 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26793  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26795  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26796  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26894  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26900  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26901  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26902  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26998  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 27003  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 27005  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 27007  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 27107  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 27118  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 27221  0.3 ftp_scanner
S        0     0 ?           -1 25316 25334     1 27222  0.0 ftp_scanner
S        0     0 ?           -1 28433 28433  9084 28433  0.0 pppd
S       99    99 ?           -1 28433 28433 28433 28434  6.5 pppoe
S       80    80 ?           -1  3508  3508  3508 29836  0.1 httpd
S        0     0 ?           -1 25073 30993     1 31000  0.0 mysqld_safe
S       27    27 ?           -1 25073 30993 31000 31033  0.0 mysqld
SW       0     0 ?           -1     1     1     6 31163  0.0 pdflush

Ayayay. I do not (for those of you who think too much) habitually run anything called ftp_scanner. I found in the ps man page an example to help me show the parent PIDs, and sure enough – all the ftp scanners had 1 (a.k.a. init) for the parent. Which means that the hacker had root access, and likely the box has been rooted.

I’ve hesitated for 10 minutes. The ftp scanner had to stop, rootkit had to be removed, the hacker locked out. But littlesvr.ca is not a toy, half my life is on it – all my email, all my computer work (svn), revenue-generating pages such as ISO Master, and less popular pages that would lose the precious little search engine ranking they had if they went offline. But it had to be done. I braced myself for a 36 hour shift, and:

root@littlesvr:~# halt

By this time my brain was overwhelmed. Too much excitement at once. The server had to be cleaned up and brought back to life in a hurry, but first I had to know how it got compromised to begin with – no point in resuscitating it only to have it hacked again two days later.

But this is a long story, and I have yet to see the ending. This post will be the first of a short series, so come back later if you want to know more.