/me@(voip.)?|www./ SamLown.com

Recently, my server has been in the depths of battle from pesky hackers entering the server through buggy programs running on my apache web server. Turns out that the culprit for the remote injection exploit was a bug in cacti that allowed the hackers or much more likely, automated scripts, to gain access to the www-data user on my server. Having someone penetrate ones server brings home the point that no important documents on the web server should be allowed to be written by the apache server user, and much care should be taken where the web server can be write to!

Finding out which program was leaking however was not so easy, so I’ve decided to document the commands I used here for future reference and the hope this information is perhaps useful to someone else.

Firstly, the most basic command in any Unix administrators toolbox is `ps` and gives you pretty good idea as to what is going on inside your server. The following command will generate a list of all the applications currently running:

 ps axh

Look for strange programs normally at the end of the list running as the web user other than of course the apache server. In debian based systems apache normally runs under the name “apache2” so if you have a malicious daemon running as “httpd” it sticks out like a sour thumb. `ps` while the most basic, its also the most obvious command that hackers will want to hide themselves from, which leads to next command:

lsof -i TCP -n

This will provide a list of all the processes on your machine running which have opened networking ports. “lsof” stands for LiSt Open Files and should be available with all distributions. Most exploits try to access the outside world and listen for their masters commands, and as such will appear here in one way or another. Its also worth swapping the “TCP” part of the command for “UDP” or just “4” for a more thorough check. It is also possible to filter by user name with the `-u www-data` option. Check the man page for more `lsof` options.

When the the culprit has been found, don’t run `‘kill -9’` just yet! We need to do a bit of forensics to find out where its running from and hopefully its source so the hole can be blocked. The most basic list of information can be found here:

/proc/-processId-

`-processId-` is the process id of dodgy program. In the directory, run ‘cat’ on the cmdline file for example to find out how it was called. Don’t expect to find to much information here though, as typically the scripts will be called with minimal options and from a temporary location like `/tmp` or `/var/tmp`. You may find however that the scripts are called from another web user writeable folder, so its worth checking.

The following command provides a list of all the files the program has requested to open on the system and is fantastic for finding out exactly what the script is accessing.

lsof -c process -n

Change `process` for the name of the malicious script. In the list of details for the cacti exploit I found an entry for `cacti/log/rra/suntzu.log` so it was clear where to look for the exploit. After a quick search on google cacti was confirmed to be at fault and after short investigation I was able to tap the hole and upgrade the application. Incidentally, the actual script used to hack was also found by google here.

Although in this case there were no left over files, it seems it deleted itself after execution, any left overs should be deleted or quarantined should you want to investigate them some more. Its also possible to leave them where you found them, but remove all access permissions to them from the web user, at least this way if an automated script tries again, it’ll have to be clever enough to use another location.

I hope this is useful to someone other than myself, and please let me know if you have any other suggests!