apache 2.2 - Server problems, running out of RAM, really high load average

I desperately need help in figuring out how to troubleshoot this problem I'm having. I run a fairly mission critical web server (Debian 7.5, 512MB RAM, 512MB swap, Apache, MySQL). It runs a couple WordPress sites on it. Today I found the websites responding quite slowly, and ssh'd in to find the load average was just above 10.0, and RAM use was at 100% and swap was close to the 512MB limit.

I have no idea how to figure out what's going on. Is Apache or MySQL not tuned properly? Maybe someone is attacking the server with repeated hits (how would I know?). I installed htop but even if I saw that Apache or MySQL was eating up a ton of resources, how would I figure out why?

Any points in the right direction would be massively appreciated. I'm at a loss here and I have to keep this server stable.

Side note: The server was up for 30 days, so maybe this was some sort of slow leak. Now that I've rebooted, load average is at 0.45 1.10 0.88, RAM is 165/512MB and swap is 68/512MB.

UPDATE: Still having issues. I captured a screenshot of htop below.

Answer

Congratulations, you've managed to use nearly all of your swap space.

The first obvious problem here is that you went very deep into swap. This is probably what's causing the system to thrash so hard (lots of time spent in system, I/O wait and software interrupts).

First thing to do is to reduce the number of Apache processes that are running. You don't need that many for a small site, and it's just going to throw you deep into swap and kill your performance...which is what already happened. I would recommend you start very small and increase when it becomes necessary. An example:

StartServers            1
MinSpareServers         1
MaxSpareServers         2
MaxClients              5

This limits you to only serving 5 simultaneous requests (everyone else has to wait in line). If at this point you get warnings from Apache about running out of servers, and you still have RAM to spare, then you can increase them, but you are eventually going to run into a point where your VPS simply hasn't got enough RAM to handle all your traffic. At that point you should upgrade the VPS.