Grumble Grumble

By Andrew Smith

I’ve been reading about literate programming, and was reminded that most programmers don’t write comments in their code. Whether to write them or not is a question that’s asked all the time, in all circles – starting in school and all the way to teams of masters of the known universe like Linus (who I’m assuming is responsible for this).

In school I thought comments are good. I’ve seen a few people who felt the same change their mind once they started writing real software, and I was tempted to drop it myself, but I didn’t.

I’ve had the same arguments with myself that you’re thinking of now. Should I document design, exceptions, explain what complicated code does, go over the assumptions, etc.? This post is not about that, it’s about something I’ve realised a few days ago – a thought that may be slightly original.

Writing comments in my code makes me feel good. I’m not a design freek and my code goes through a couple of iterations before it’s even checked into version control, often followed by more restructuring when flaws are found. And then I write comments.

I look over a piece of code and think – what the hell is this doing, here? Oh – that’s to handle the weird case of an idiot unplugging the USB stick before it’s done. Comment. Man this function is long and hard to read. Several comments throughout telling a story (this being possible explains why it’s all in one function). Wow, at this moment I can actually grasp how the entire subsystem works – comments on top of the class and every function. Why is this a member variable and not a local one? Oh yeah, I remember – that’s bizarre but it works. Comment.

Did I manage to explain it? As I read my code I remember all the interesting, strange, unfortunate, and neat things I’ve done. And I write it down in plain english. I feel empowered when I can explain something that appears to be a terrible mess. I think this is why I still do write comments. And I know they’re useful because when I go back to my code a month later I can actually figure out what’s where using them.

Maybe it will work for you too?

By Andrew Smith

I’ve been running my more-or-less regular backup of littlesvr.ca, and found that yet again the online APNG assembler data directory is the one taking most of the time. Turns out that even though I cleared it a few months ago there is over 4GB of PNG files in there (half of that – 5800 APNG files).

That’s really interesting – it means APNG usage has actually grown in the last year. Partly that might be explained by ongoing support in Firefox, but surely that’s not the only explanation. I don’t know what’s really driving the growth. I just know it isn’t me.

On the one hand it’s great that people took over (e.g. Max Stepin has been diligently maintaining the APNG patch for well over a year now) but on the other hand it’s sad that the project survived without me – I guess I wasn’t that important :)

I have been feeling sad about such things lately. I always do my best to bootstrap a project and then pass it on to someone else to maintain. This way I can keep doing new and exciting things while expanding the breadth of my knowledge and skills, and increasing the impact I have overall. But damn it’s depressing when I’m in between projects. I feel useless – not wanted and not needed.

Should I just accept that there is only so much one can do in a lifetime, and specialise in some irrelevant field? Become normal? What a dreadful thought.

By Andrew Smith

There is a magic thing called gettext. It’s magic because it allows some random Joe on the internet with no experience in programming to translate your application into his language. He does this by editing a .po file, then sends it to you, and you just include that file in your build system.

Ok there is some plumbing you need to enable that, but really it’s worth it.

In one of my applications I followed the advice in the gettext manual and used plural forms for messages. That is rather than saying “%d file(s)” the string had two forms: “%d file” (if %d is 1) and “%d files” (if %d is not one). This is specified in the po file like so:

“Plural-Forms: nplurals=2; plural= n!=1;\n”

msgid “%d file”
msgid_plural “%d files”
msgstr[0] “”
msgstr[1] “”

But of course different languages have different singular/plural rules. For example in Russian 0, 1, 2, and 5 have different plural forms. That means the russian translator has to figure out how this whole system works in order to translate strings with plural forms.

These translators are volunteers. That they do it at all is amazing. Having them read the gettext manual (which doesn’t have a lot of examples like it should) is disrespectful and unproductive. You will get fewer translations of not-as-good quality if you use plural forms.

I know, I learned this the hard way. One of my programs uses plural forms and another doesn’t. I have spent way too much time explaining to people how this works, and I am sure some of them ignored what I said and just gave me whatever compiles.

And believe me figuring this out isn’t easy. I mentioned the russian example – honestly I have no idea what the rules are exactly (who ever thinks of these things?) and I speak russian, and I maintain the application. How can I ask Joe random to understand this?

There you go. Use plural forms at your own peril. The gettext manual lies, plural forms are not a good thing and the problems they cause are bigger than the one problem they solve. I have just got a new translation, and guess what – the two strings with plural forms are untranslated. So I have to get a headache explaining this yet again, while kicking myself in the head for buying into the gettext dogma.

By Andrew Smith

In newsgroups and many mailing lists top-posting (putting your reply above the text you’re replying to) is a big no-no. You’ll get flamed even if what you say is very useful, constructive, and nice. I’ve never quite understood the zealotry, but I figured: whatever, it sort of makes sense – someone says something, and then you reply to it.

But in regular email I have noticed many, many people write their reply at the top of the message. Until recently I had no idea why.

Last week I got a Blackberry. It’s a nice device, but it has a really small screen compared to a computer. When you open an email on a smartphone, you want to read what matters (the reply in most cases) right away. Scrolling to where the reply begins is painful and very time consuming.

Hm, I nearly posted something educational rather than inflamatory on my blog.. Now if I can just find a way to write blog posts from my Blackberry I may actually post more than once per month. And don’t give me that retarded Twitter crap.

By Andrew Smith

I’ve been pondering how one may be able to get rid of spam. Not just what comes into my mailbox, but most spam everywhere. And I found such a simple solution it’s amazing I haven’t heard of it before. But of course it will never happen, so that probably explains it.

Spammers rely on that sending even extremely large volumes of spam is nearly free. It takes some effort to find a nice relay and/or create a small botnet, but following that every message a spammer sends is free. I don’t have the statistics handy, but the ‘positive’ response to spam is something like one for every hundred thousand messages sent.

Now imagine it cost five cents to deliver an email. That means to send 100k of messages the spammer would have to pay 5000$, which would make the business not feasible. How easy a solution is that!

And yeah, I know – gmail filters all the spam for you, and the wrong infrastructure is there, and some people would actually mind paying 5 cents to send an email, but I think it’s a great idea anyway, even though it is unlikely to happen in this form in my lifetime.

By Andrew Smith

Here’s a stupid post, it’s been a while. I’ve upgraded Slackware from 12.2 to 13.0 and now it is messing with me. Believe it or not the stupid thing knows when I’m annoyed and pisses me off even more. Here’s how it works:

Sometimes my pointer starts jumping all around the screen, for no good reason. That’s not entirely unexpected, I have a crappy wireless mouse – but sometimes it gets on my nerves. When it does the first thing I do is stop moving and breathing, to see if it moves on its own or it goes nutz. No, it doesn’t do anything when I freeze.

So I take it easy, and use the mouse for a few seconds normally – but then it starts messing with me again. I slam my hand on the table, put a firm hand on top of the mouse, and deliberatly move it down – and it’s like it dares me to slide it all the way off the table, the pointer keeps jerking back up. Repeat, same thing.

Hardware problem? I thought so too, and replaced the wireless USB mouse with a wired PS2 mouse. Same problem.

You’ll never guess what it was. Turns out Slackware 13 treats my Microsoft Sidewinder steering wheel as a mouse. So when I press the pedals, it moves the mouse. When I deliberatly move the mouse down I at the same time (without thinking) press the gas pedal, which moves the pointer back up. Great stuff :)

By Andrew Smith

I got an email from someone who appears to be helping KDE out with switching their licences. I guess at some point the guys decided that nothing in KDE shall be licenced GPLv2 only, and that’s the request I got – change the licence to “GPL2 or 3″ or “GPL2 or later”.

Here are a couple of links describing the process:

http://techbase.kde.org/Policies/Licensing_Policy
http://techbase.kde.org/Projects/KDE_Relicensing

I’m not surprised to see only people names listed on the second page, no project names. I wonder how much software KDE plans to scrap just because of some overly idealistic dudes with big beards decided the new GPL is much better and must be incompatible with the old GPL.

My piece of code in there is bkisofs – a library for reading and writing ISO files which is being used by Ark (the KDE archive manager). The integration was not my doing, I only helped – but the library itself is mine. KDE won’t explode the funcionality is removed from Ark.

I haven’t replied to the request yet. On the one hand I don’t care if it’s licenced GPL2 or 3, the changes I recall don’t affect this particular software. On the other hand – I am opposed to the GPL3 as a matter of principle, and I don’t like bending over or getting strong-armed into doing things with my volunteer time.

And if I allow this, that will count as a contribution to GPL3 zealotry.

I’ll go review the GPL3 some time in the next few days. If anyone still reads my blog and has ideas one way or the other – please share.

By Andrew Smith

I left off with my server almost completely back up, but not yet Apache. I’ve had to make sure the web apps off the internet weren’t full of security holes before allowing access to them again.

Though this is my third post 4 days later – in real time it’s been less than 24 hours since I’ve discovered the ftp scanner. So I had some breathing time to do a half-decent job.

Most of all I suspected Wordpress. They come up with a new version every few months and every time claim it’s a security update, which makes me wonder if it’s ever secure. Luckily both ISO Master and Grumble Grumble are just stock wordpress installs with a theme over them. Everything else is stored in the database. So I was able to delete all the code except for the themes, install Wordpress 2.7 on both, and miraculously both websites came back up with no trouble whatsoever.

I have not reviewed the theme code or the contents of the databases – I assume the Wordpress guys were smart enough not to store executable code in either of those places. Either way it was a risk I was willing to take.

A third Wordpress website (the old Canvas3D blog) I’ve decided not to bother with and left it with chmod 0.

Three web apps down, two to go – SquirrelMail and Roudcube. I have both installed because I have more than one email account, so I have both open in browser tabs at once. I’ve decided to look around before enabling these. RoudCube first. Some way or another I stumbled upon a php file in the logs directory:

root@legrand-sw:/media/hd/home/www/htdocs/mail/logs# ls -al
total 129
drwxrwxr-x  2 andrew apache    120 2008-12-30 18:00 ./
drwxr-xr-x 10 andrew users     480 2008-08-15 10:25 ../
-rw-r--r--  1 apache apache  14985 2009-01-07 13:29 errors
-rw-r--r--  1 apache apache   7033 2009-01-07 13:26 sendmail
-rw-r--r--  1 apache apache 105643 2008-12-30 18:00 zq.php

Which is a really strange place to keep a php file, especially since the directory is writeable by Apache. See how zh.php is owned by apache? That’s really bad. And lookie here what was at the top of zq.php:

/******************************************************************************************************/
/*
/*                                     #    #        #    #
/*                                     #   #          #   #
/*                                    #    #          #    #
/*                                    #   ##   ####   ##   #
/*                                   ##   ##  ######  ##   ##
/*                                   ##   ##  ######  ##   ##
/*                                   ##   ##   ####   ##   ##
/*                                   ###   ############   ###
/*                                   ########################
/*                                        ##############
/*                                 ######## ########## #######
/*                                ###   ##  ##########  ##   ###
/*                                ###   ##  ##########  ##   ###
/*                                 ###   #  ##########  #   ###
/*                                 ###   ##  ########  ##   ###
/*                                  ##    #   ######   #    ##
/*                                   ##   #    ####   #    ##
/*                                     ##                 ##
/*
/*
/*
/*  r57shell.php -
/*                                               : http://rst.void.ru
/*        : 1.3 (05.03.2006)
/*~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*/
/*                                          : blf, phoenix, virus, NorD                  RST/GHC.
/*                       -
/*     rst@void.ru.                                  .
/*~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*/
/*  (c)oded by 1dt.w0lf
/*  RST/GHC http://rst.void.ru , http://ghc.ru
/*  ANY MODIFIED REPUBLISHING IS RESTRICTED
/******************************************************************************************************/

Isn’t that interesting.. I didn’t quite know what to think of the bug – on the one hand looks like the hacker put it there to taunt me, on the other hand RoundCube is a young open source project so who knows what the developer was smoking?

Didn’t take long to figure out though: both http://rst.void.ru and http://ghc.ru are hacker websites, though ghc.ru looks half-commercial. And the contents of the file (which I’ve had no time to analyse) clearly do something that isn’t nice. I’ve renamed the file to .txt and put it up for you to study if you care.

As with mailman (see previous post) I decided this service is not going to be reenabled. Maybe I messed up something during setup, but just as likely RoudCube was at fault. After all it was the first ’stable’ version, 0.1. And I’ll keep SquirrelMail off too for good measure, I’ll just install Thunderbird on all my machines.

So there you go. One clue in Mailman, one in RoudCube. Are they related or are they two separate hacking attempts? Were they run by script kiddies or serious hackers? Having some evidence that the ftp scanner was set up manually I tend to favour the latter possibility. And what of this zq.php? Was the logo there because it’s a general-purpose cracking tool, or because those groups actually had something to do with my server?

There’s lots of investigation left still. At this point my server is completely operational so I have lots of time. I do hope I find more clues, mystery solving turns out to be a really exciting hobby.

By Andrew Smith

First things first – my server had to be brought back up. But bringing it back up was not a matter as simple as reinstalling Slackware and plopping the home directories back in. So the first thing that needed to be done was to figure out how the server’s been compromised.

The quickest way to do this, I figured, was to do both at the same time. I’d get the data off the server, reformat the hard drive, start reinstalling the operating system, and during that time would analyse the backed up files for evidence. Just to be throrough I wanted to keep a byte-by-byte copy of the hard drive – that may contain clues that simple copies of the files would lose, and using the image I could bring a clone of the hacked server back up in a controlled environment – exactly the same minus any processes that were running.

The command was easy enough, ran from the shell of a Slackware 12.2 installation CD:

dd if=/dev/hda1 | ssh 10.0.0.1 dd of=/media/backup/hacked/hda1.dd
dd if=/dev/hda2 | ssh 10.0.0.1 dd of=/media/backup/hacked/hda2.dd

I’ve saved hda2 (the swap partition) just in case I ever found a tool to analyse contents of memory (unlikely to happen). Turns out the 80G image would take hours to transfer over, so I set the alarm clock for 2AM and went to sleep for a couple of hours, knowing I wouldn’t get any the next day.

The image hda1.dd (as any other image of a Linux-supported filesystem) can be mounted onto an existing system. I decided to go ahead and do that since nothing on the partition could damage a fresh system unless they found a bug in the filesystem driver and infected the FAT:

mount /media/backup/hacked/hda1.dd /media/hd -o loop

Looking around /media/hacked I quickly found something that got my attention: a file and a directory in /root/ that didn’t look familiar:

root@legrand-sw:/media/hd/root# ls -l heroina/
<stripped out some things>
drwxr-xr-x 2   1003 users   304 2008-09-23 16:24 heroina/
-rw-r--r-- 1 root   root  81920 2008-12-16 16:42 ketamin.tar

root@legrand-sw:/media/hd/root# ls -l heroina/
total 92
-rwxr-xr-x 1 1003 users    82 2006-03-10 15:02 0-100*
-rwxr-xr-x 1 1003 users    84 2006-03-10 21:02 100-200*
-rwxr-xr-x 1 1003 users    84 2006-03-10 21:02 200-255*
-rwxr-xr-x 1 1003 users   428 2009-01-07 23:44 dava*
-rwxr-xr-x 1 1003 users 17145 2008-09-23 17:07 ftp_scanner*
-rwxr-xr-x 1 1003 users 17246 2008-09-23 16:49 heroina.c*
-rwxr-xr-x 1 1003 users  2094 2006-03-10 21:02 o*
-rwxr-xr-x 1 1003 users 20313 2008-06-15 18:46 pass*
-rwxr-xr-x 1 1003 users   119 2008-09-23 16:22 run*
-rwxr-xr-x 1 1003 users  7490 2008-06-17 15:51 users*

There’s my ftp_scanner, and who would have thought, it even comes with source code. Well of course – who would use a binary to mess with an open source server? Yep, here’s heroina.c if you care to look at it, but basically it looks like general-purpose software:

Multi-thread FTP scanner v0.2.5 by Inode <inode@wayreth.eu.org>

users and pass are tiny dictionaries of common usernames and passwords, the 0-100 files just helpers for making ip addresses, o looked like the output from nmap. Not terribly sofisticated stuff, but interesting.

Of course this doesn’t come close to telling me how that software got on in the first place, but it does give some interesting clues.

The timestamps are really interesting, you could piece any number of stories looking just at those. See for example how ftp_scanner is 18 minutes older than heroina.c, that suggests it was built manually (what sort of automation takes 18 minutes to build one C file?). And ketamin.tar has been sitting there for more than 3 weeks, which is kind of depressing.

And look at the owner, 1003. That according to /media/hd/etc/passwd was mailman on the old littlesvr.ca. Well that’s not going back on the system in a hurry, and how was it able to write to /root/ anyway?

I already knew the web applications are the most vulnerable part of my system. I trust Slackware, I trust me configuring the running services, but I have no idea how many security problems the web apps on the system have. I haven’t read the source for Mailman, Wordpress, SquirrelMail, or Roudcube, and they don’t come with Slackware – how am I supposed to trust them? Who vouched for them?

Anyway – by this time Slackware has been reinstalled and littlesvr.ca was sitting there in its vanilla state. I’ve decided to turn on sendmail and SVN – which would give me back two thirds of half my life, so I cautiously copied my home directories back. Apache was still off.

If you think this is the end of the story, you’re wrong. Come back later as I share more clues with you, I have at least one interesting thing left – plus there still are no certain conclusions about who and how and why did this.

By Andrew Smith

On thursday evening I had a couple of hours to spare, so I’ve SSHed to my server (yours truly littlesvr.ca) to get the Apache logs from the last couple of months. I do this now and then because I like seing the ammount of traffic going up every month. But this time it was not going to be a gratifying experience. The first command I ran was (as usual) ‘ls’:

andrew@littlesvr:/$ ls
/bin/ls: unrecognized prefix: do
/bin/ls: unparsable value for LS_COLORS environment variable
bin/    dev/    home/   media/  opt/    root/   srv/    sys/    usr/
boot/   etc/    lib/    mnt/    proc/   sbin/   svn/    tmp/    var/

A second of confusion, but I knew then and there my box has been hacked. You see I’m running Slackware because weird crap like this (‘ls’ broken) doesn’t happen on Slackware, and though I denied it for a few minutes (oh maybe it’s a terminal problem, oh maybe I deleted something by mistake) soon enough I’ve had ample evidence. I’ve looked at /bin/ls in ‘vi’, which opens binaries as a hex editor. Didn’t see anything obviously wrong there. I’ve examined all the profile files, and those all looked normal. I grepped everything I could think of for ‘do’, and didn’t find a problem. Then I had a look at /var/log/messages, and found lots and lots of lines of the form:

Jan  7 00:13:30 littlesvr in.identd[7245]: reply to 24.165.1.229: 59167 , 21 : USERID : OTHER :0

As I was complaining about this in the #seneca IRC channel, I thought it would be worth while seing just how many of these strange lines are in the log. A quick ‘grep | wc -l’ gave me a staggering number, over 325k. Then I ran ‘ps aux’:

root@littlesvr:/etc/rc.d# ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu,comm
STAT  EUID  RUID TT       TPGID  SESS  PGRP  PPID   PID %CPU COMMAND
S        0     0 ?           -1     1     1     0     1  0.0 init
SW       0     0 ?           -1     1     1     1     2  0.0 migration/0
RWN      0     0 ?           -1     1     1     1     3  0.0 ksoftirqd/0
SW<      0     0 ?           -1     1     1     1     4  0.0 events/0
SW<      0     0 ?           -1     1     1     1     5  0.0 khelper
SW<      0     0 ?           -1     1     1     1     6  0.0 kthread
SW<      0     0 ?           -1     1     1     6    40  0.0 kblockd/0
SW<      0     0 ?           -1     1     1     6    41  0.0 kacpid
SW<      0     0 ?           -1     1     1     6    95  0.0 ata/0
SW<      0     0 ?           -1     1     1     6    96  0.0 ata_aux
SW<      0     0 ?           -1     1     1     6    97  0.0 ksuspend_usbd
SW<      0     0 ?           -1     1     1     6   100  0.0 khubd
SW<      0     0 ?           -1     1     1     6   102  0.0 kseriod
SW<      0     0 ?           -1     1     1     6   124  0.0 kswapd0
SW<      0     0 ?           -1     1     1     6   125  0.0 aio/0
SW<      0     0 ?           -1     1     1     6   783  0.0 scsi_tgtd/0
SW<      0     0 ?           -1     1     1     6   814  0.0 kcryptd/0
SW<      0     0 ?           -1     1     1     6   836  0.0 reiserfs/0
S<       0     0 ?           -1   900   900     1   900  0.0 udevd
SW<      0     0 ?           -1     1     1     6  1853  0.0 kpsmoused
S        0     0 ?           -1  1927  1927     1  1927  0.0 klogd
S        1     1 ?           -1  2170  2170     1  2170  0.0 rpc.portmap
S       99    99 ?           -1  2174  2174     1  2174  0.0 rpc.statd
S        0     0 ?           -1  2195  2195     1  2195  0.0 ntpd
S        0     0 ?           -1  2200  2200     1  2200  0.0 acpid
S       81    81 ?           -1  2208  2208     1  2208  0.0 dbus-daemon
S       82    82 ?           -1  2213  2213     1  2213  0.0 hald
S        0     0 ?           -1  2213  2213  2213  2214  0.0 hald-runner
S       82    82 ?           -1  2213  2213  2214  2223  0.0 hald-addon-acpi
S        0     0 ?           -1  2229  2229     1  2230  0.0 crond
S        2     0 ?           -1  2232  2232     1  2232  0.0 atd
S        0     0 ?           -1  2235  2235     1  2235  0.0 saslauthd
S        0     0 ?           -1  2235  2235  2235  2236  0.0 saslauthd
S        0     0 ?           -1  2235  2235  2235  2237  0.0 saslauthd
S        0     0 ?           -1  2235  2235  2235  2238  0.0 saslauthd
S        0     0 ?           -1  2235  2235  2235  2239  0.0 saslauthd
S        0     0 ?           -1  2376  2376     1  2377  0.0 python
S        0     0 ?           -1  2379  2379     1  2379  0.0 svnserve
S     1003  1003 ?           -1  2381  2381     1  2381  0.0 mailmanctl
S        0     0 vc/1      2382  2382  2382     1  2382  0.0 agetty
S        0     0 vc/3      2384  2384  2384     1  2384  0.0 agetty
S        0     0 vc/4      2385  2385  2385     1  2385  0.0 agetty
S        0     0 vc/5      2386  2386  2386     1  2386  0.0 agetty
S        0     0 vc/6      2387  2387  2387     1  2387  0.0 agetty
S     1003  1003 ?           -1  2381  2381  2381  2398  0.0 python
S     1003  1003 ?           -1  2381  2381  2381  2399  0.0 python
S     1003  1003 ?           -1  2381  2381  2381  2400  0.0 python
S     1003  1003 ?           -1  2381  2381  2381  2401  0.0 python
S     1003  1003 ?           -1  2381  2381  2381  2402  0.0 python
S     1003  1003 ?           -1  2381  2381  2381  2403  0.0 python
S     1003  1003 ?           -1  2381  2381  2381  2404  0.0 python
S     1003  1003 ?           -1  2381  2381  2381  2405  0.0 python
S        0     0 vc/2      2422  2422  2422     1  2422  0.0 agetty
SW       0     0 ?           -1     1     1     6  3220  0.0 pdflush
S        0     0 ?           -1  3339  3339     1  3339  0.0 inetd
S        0     0 ?           -1  3508  3508     1  3508  0.0 httpd
S       99    99 ?           -1  7245  7245     1  7245  0.0 in.identd
S       80    80 ?           -1  3508  3508  3508  7754  0.1 httpd
S       80    80 ?           -1  3508  3508  3508  7755  0.0 httpd
S     1000  1000 ?           -1  8112  8112  3339  8112  0.0 imapd
S     1002  1002 ?           -1  8113  8113  3339  8113  0.0 imapd
S     1004  1004 ?           -1  8114  8114  3339  8114  0.0 imapd
S     1007  1007 ?           -1  8115  8115  3339  8115  0.0 imapd
S        0     0 ?           -1  8281  8281 25049  8281  0.0 sshd
S     1000  1000 ?           -1  8281  8281  8281  8292  0.0 sshd
S     1000  1000 pts/2    11949  8293  8293  8292  8293  0.0 bash
S        0     0 ?           -1  8760  9062     1  9063  0.0 ddclient
S        0     0 ?           -1  8760  9079     1  9084  0.0 pppoe-connect
S        0     0 ?           -1  9345  9345     1  9345  0.0 sendmail
S       25    25 ?           -1  9348  9348     1  9348  0.0 sendmail
S        0     0 pts/2    11949  8293 10005  8293 10005  0.0 bash
S        0     0 ?           -1 10377 10377 25049 10377  0.0 sshd
S     1000  1000 ?           -1 10377 10377 10377 10383  0.0 sshd
S     1000  1000 pts/4    10420 10384 10384 10383 10384  0.0 bash
S        0     0 pts/4    10420 10384 10420 10384 10420  0.0 bash
S       80    80 ?           -1  3508  3508  3508 10598  0.3 httpd
S       80    80 ?           -1  3508  3508  3508 10599  0.2 httpd
S       80    80 ?           -1  3508  3508  3508 11510  0.0 httpd
S       80    80 ?           -1  3508  3508  3508 11511  0.0 httpd
S       80    80 ?           -1  3508  3508  3508 11512  0.0 httpd
R        0     0 pts/2    11949  8293 11949 10005 11949  0.0 ps
S        0     0 ?           -1  2379  2379  2379 15669  0.0 svnserve
S       80    80 ?           -1  3508  3508  3508 19216  0.1 httpd
S       80    80 ?           -1  3508  3508  3508 22340  0.1 httpd
S        0     0 ?           -1 25049 25049     1 25049  0.0 sshd
S        0     0 ?           -1 25295 25295     1 25295  0.0 syslogd
S        0     0 ?           -1 25316 25334     1 25345  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25349  0.1 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25380  0.5 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25383  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25385  0.4 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25441  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25444  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25446  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25627  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25632  0.4 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25732  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25740  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25741  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25836  0.1 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25846  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25947  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25949  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 25950  0.4 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26052  0.1 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26054  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26157  0.2 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26158  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26262  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26480  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26586  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26682  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26689  0.1 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26793  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26795  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26796  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26894  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26900  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26901  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26902  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 26998  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 27003  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 27005  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 27007  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 27107  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 27118  0.0 ftp_scanner
S        0     0 ?           -1 25316 25334     1 27221  0.3 ftp_scanner
S        0     0 ?           -1 25316 25334     1 27222  0.0 ftp_scanner
S        0     0 ?           -1 28433 28433  9084 28433  0.0 pppd
S       99    99 ?           -1 28433 28433 28433 28434  6.5 pppoe
S       80    80 ?           -1  3508  3508  3508 29836  0.1 httpd
S        0     0 ?           -1 25073 30993     1 31000  0.0 mysqld_safe
S       27    27 ?           -1 25073 30993 31000 31033  0.0 mysqld
SW       0     0 ?           -1     1     1     6 31163  0.0 pdflush

Ayayay. I do not (for those of you who think too much) habitually run anything called ftp_scanner. I found in the ps man page an example to help me show the parent PIDs, and sure enough – all the ftp scanners had 1 (a.k.a. init) for the parent. Which means that the hacker had root access, and likely the box has been rooted.

I’ve hesitated for 10 minutes. The ftp scanner had to stop, rootkit had to be removed, the hacker locked out. But littlesvr.ca is not a toy, half my life is on it – all my email, all my computer work (svn), revenue-generating pages such as ISO Master, and less popular pages that would lose the precious little search engine ranking they had if they went offline. But it had to be done. I braced myself for a 36 hour shift, and:

root@littlesvr:~# halt

By this time my brain was overwhelmed. Too much excitement at once. The server had to be cleaned up and brought back to life in a hurry, but first I had to know how it got compromised to begin with – no point in resuscitating it only to have it hacked again two days later.

But this is a long story, and I have yet to see the ending. This post will be the first of a short series, so come back later if you want to know more.