ubuntu - My linux server was hacked. How do I find out how and when it was done?

I have a home server running a desktop ubuntu distribution. I found this in my crontab

* * * * * /home/username/ /.access.log/y2kupdate >/dev/null 2>&1

and when looking in that directory (the space after username/ is a directory name) I found a lot of scripts which are obviously doing something they shouldn't.

Before I wipe that computer and reinstall stuff, I would like to find out what caused the security breach and when it was done. So I don't open the same hole again.

What log files should I look in? The only servers that I am aware of that are running on the computer is sshd and lighttpd.

What should I do to detect if things like this happens again?

Answer

First, make sure the computer is disconnected from any networks.
Second, make sure you get any important data off the drives before booting the hacked OS again.

Start with checking out the time stamps on the files in question. Often they are accurate.
Cross reference those with the httpd log and the auth log if they weren't wiped. If one other the other was wiped, you can bet that was the means of entry. If they're still in tact, you might be able to glean more information on how they got in from the log.

If they're all wiped, you're pretty screwed. It would likely take more time to figure out what happened than it's worth.

You mentioned those two services were running, was there a good firewall in place to prevent everything else from being accessed? Did you allow SSH on port 22; is your login reasonably easy to guess; did you allow password logins; did you have any sort of real rate limiting for password logins? Did you have any additional software installed with lighttpd; perl; php; cgi; a CMS or similar? Were you running updated version of all the software; do you subscribe to security notifications for all the software you run and carefully evaluate all notifications to see if they apply to software you run/expose to the public?