linux - How to find the cause for growing server load

I'm having load problems with my server and even though I'm a somewhat experienced Linux admin I'm out of ideas now.

The problem is a slowly but steadily increasing load on the server without any apparent cause.

The Server is a AMD Athlon(tm) 64 X2 Dual Core Processor 6000+ with 6GB RAM. It is running Debian Stable with Linux gir 2.6.26-2-amd64 #1 SMP Wed Aug 19 22:33:18 UTC 2009 x86_64 GNU/Linux.

The server basically runs Lighttpd, several FastCGI PHP processes and a MySQL database. Typical webserver tasks.

The CPU is never really fully used up and memory is mainly used for buffers and cache which is fine. I tried to restart the various services to see if one of them would decrease the load again, but without luck.

Here are graphics showing load, CPU and IOStat:

So, question is: What could cause a slowly but ever increasing load? And how do I find out what's responsible?

Update: I forgot to mention, when I reboot the server, the load will be down to around 0.3 to 0.6 and will start to climb up again slowly over the next weeks.

Answer

I found an excellent hint in answer to a different question.

Looking for processes in state 'D' shows four PHP processes that seem to hang for quite a while corresponding to the "steps" in the load curve:

#> ps aux | awk '$8 ~ /D/  { print $0 }'
wiki      6651  0.0  0.0      0     0 ?        D    Oct04   0:41 [php-cgi]
bugs      6731  0.0  0.0      0     0 ?        D    Oct27   0:14 [php-cgi]
manpages  7536  0.0  0.0      0     0 ?        D    Oct30   0:21 [php5-cgi]
wiki     23847  0.0  0.0      0     0 ?        D    Oct06   1:32 [php-cgi]

So these seem to be the problem. I now need to find out while those processes hang and how to fix it. Thanks everyone.