security - Has my Linux server been compromised? How do I tell?

Running (X)Ubuntu 10.04.2 LTS behind a router.

I just received an email from my root account on that machine, with the following subject:

*** SECURITY information for :

The message body contained this warning:

: jun 1 22:15:17 : : 3 incorrect password attempts ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/sh /tmp/tmpPHBmTO

I can see no /tmp/tmpPHBmTO file, though there is a file named /tmp/tmpwoSrWW with a timestamp dating from 2011-06-01 22:14, so just before the mentioned date/time. It's a binary file, and the content doesn't look familiar to me. Also, that file only has -rw------- permissions.

As I read it, this means that someone (or something) has (had) access to my machine. Apparently not root access (yet), but still, enough to write files to my /tmp directory at the very least.

Does someone have any pointers as to where I could look for more information: who could have done this, and how they could have done this?

My router is configured to allow access to forward traffic for SSH, HTTP (nginx acting as reverse proxy for one of several other services), SMTP, POP (postfix) and IMAP (dovecot), and also port 51413 (Transmission).

Answer

I realize it's been two years since the original question was asked, but in case somebody else comes here through Google like me: I saw that behavior caused by a Dropbox daemon running with a non-root account on a server which didn't run any X server. I managed to copy the files before they were deleted. For some reason the daemon wants to reset permissions on its data directory (never mind that it wouldn't even have needed root to do that) and kill some process. I can only speculate as to why, maybe it did some kind of auto update and tried to reload itself or something like that.

The file /tmp/tmpe1AGcd contains:

#!/bin/bash
sudo -K
zenity --entry --title="Dropbox" --text="Dropbox needs your permission to save settings to your computer.

Type your Linux password to allow Dropbox to make changes." --entry-text "" --hide-text | sudo -S /bin/sh /tmp/tmpAH5mxL
if [ "$?" != 0 ]; then
zenity --error --text="Sorry, wrong password"
exit 1

fi

Since the machine is running headless and the binary "zenity" isn't even installed, sudo receives empty password attempts and fails when trying to execute /tmp/tmpAH5mxL, containing:

#!/bin/bash
chown -R 1000 "/home//.dropbox"
chmod -R u+rwX "/home//.dropbox"
kill -s USR2 5364

The resulting message I got was the same you did:

 : Jul  4 16:32:24 : : 3 incorrect password attempts ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/sh /tmp/tmpAH5mxL