linux - Stuck trying to track down the culprit for high CPU usage on a LAMP server

I'm running Apache on RedHat Enterprise 5.5 with PHP 5.1 from the Rackspace IUS community repository.

Occasionally, I have server spikes of 8+ load. top shows that httpd.worker is consuming 120-150% CPU but the hits don't appear to be coming that fast. I do notice that our craftysyntax URL seems to keep coming up when that happens.

I also have mod_security installed with one of the core rules sets.

I'm scrathcing my head about how to figure out what's causing the high CPU usage. My prime suspects are PHP and mod_security.

I have the following oprofile output form during a load spike, but it doesn't tell me which process is calling libpcre:

% opreport
CPU: Intel Architectural Perfmon, speed 2793.09 MHz (estimated)

Counted CPU_CLK_UNHALTED events (Clock cycles when not halted)
with a unit mask of 0x00 (No unit mask) count 100000

CPU_CLK_UNHALT...|
  samples|      %|

------------------
  7061182 91.9105 libpcre.so.0.0.1
   206901  2.6931 php-cgi
   142239  1.8514 mod_security2.so
   138121  1.7978 vmlinux
    53809  0.7004 libc-2.5.so
    20909  0.2722 libapr-1.so.0.2.7
    16585  0.2159 oprofiled
     9230  0.1201 oprofile

FYI, ldd shows that mod_security is linked to libpcre, but php-cgi is not (weird).

Answer

Attach ltrace the pid of the live process that's causing the issue. It looks like it's some bad regex, or a regex that gets called a lot. In either case, gotta localize it first. Remember to follow the forks.