I've got a server running several cherrypy apps on apache2 under mod_wsgi. We're seeing constantly fluctuating load average on a box that is not serving many requests. As far as I can tell, the box is under no real CPU load, has plenty of memory, there is very little network traffic and no disk I/O occurring. We are running 13 mod_wsgi daemon processes with 5 threads per process serving 5 different applications. These are very lightweight backend service applications that don't do much processing at all. I've checked just about everything I can think of as a cause of the load flapping and was wondering if anyone here has had experience with a similar problem. Any comments greatly appreciated.
Here's a trace of load averages over the course of about 5 minutes on a staging box serving 10s of requests per minute:
~ $ sar -q 5
Linux 2.6.32-305-ec2 01/27/2011 _i686_ (1 CPU)
04:18:37 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
04:18:42 AM 0 257 1.52 1.90 1.89
04:18:47 AM 0 257 1.40 1.87 1.88
04:18:52 AM 0 257 1.28 1.84 1.87
04:18:57 AM 0 257 1.18 1.81 1.86
04:19:02 AM 0 257 1.17 1.79 1.85
04:19:07 AM 0 257 1.15 1.78 1.85
04:19:12 AM 0 257 1.14 1.77 1.84
04:19:17 AM 0 257 1.05 1.74 1.83
04:19:22 AM 0 257 0.96 1.71 1.82
04:19:27 AM 0 257 0.89 1.68 1.81
04:19:32 AM 0 256 0.82 1.65 1.80
04:19:37 AM 0 256 0.75 1.62 1.79
04:19:42 AM 0 256 0.69 1.60 1.78
04:19:47 AM 0 256 0.95 1.64 1.79
04:19:52 AM 0 256 1.20 1.67 1.81
04:19:57 AM 0 256 1.42 1.71 1.82
04:20:02 AM 0 256 1.31 1.68 1.81
04:20:07 AM 0 256 2.00 1.82 1.85
04:20:12 AM 0 256 2.64 1.96 1.89
04:20:17 AM 0 256 3.23 2.09 1.94
04:20:22 AM 0 256 2.97 2.06 1.93
04:20:27 AM 0 256 2.74 2.02 1.92
04:20:32 AM 0 256 2.52 1.99 1.91
04:20:37 AM 0 256 2.31 1.95 1.90
04:20:42 AM 0 256 2.13 1.92 1.89
04:20:47 AM 0 256 1.96 1.89 1.88
04:20:52 AM 0 256 1.80 1.86 1.87
04:20:57 AM 0 256 1.66 1.83 1.85
04:21:02 AM 0 256 1.52 1.80 1.84
04:21:07 AM 0 256 1.40 1.77 1.83
04:21:12 AM 0 256 1.29 1.74 1.82
04:21:17 AM 0 256 1.19 1.71 1.81
04:21:22 AM 0 256 1.09 1.68 1.80
04:21:27 AM 0 256 1.00 1.65 1.79
04:21:32 AM 0 256 0.92 1.62 1.78
04:21:37 AM 0 256 0.85 1.59 1.77
04:21:42 AM 0 256 0.78 1.57 1.77
04:21:47 AM 0 256 0.72 1.54 1.76
04:21:52 AM 0 256 0.98 1.58 1.77
04:21:57 AM 0 256 1.22 1.62 1.78
04:22:02 AM 0 256 1.44 1.66 1.79
04:22:07 AM 0 256 2.13 1.80 1.83
04:22:12 AM 0 256 2.76 1.93 1.88
04:22:17 AM 0 256 3.34 2.07 1.92
04:22:22 AM 0 256 3.87 2.20 1.96
04:22:27 AM 0 256 3.56 2.16 1.95
04:22:32 AM 0 256 3.28 2.13 1.94
04:22:37 AM 0 256 3.01 2.09 1.93
04:22:42 AM 0 256 2.77 2.06 1.92
04:22:47 AM 0 256 2.55 2.02 1.91
04:22:52 AM 0 256 2.34 1.99 1.90
04:22:57 AM 0 256 2.16 1.95 1.89
04:23:02 AM 0 256 1.98 1.92 1.88
04:23:07 AM 0 256 1.82 1.89 1.87
04:23:12 AM 0 256 1.68 1.86 1.86
and a top profile:
top - 04:38:57 up 1:17, 1 user, load average: 2.55, 3.03, 2.46
Tasks: 78 total, 1 running, 77 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1741016k total, 946844k used, 794172k free, 63712k buffers
Swap: 917496k total, 0k used, 917496k free, 646064k cached
per request, apache conf of one service (they all pretty much look like this).
Listen 12800
WSGIScriptAlias / /var/www/services/tracking/tracking.wsgi
WSGIDaemonProcess tracking user=www-data group=www-data processes=3 threads=5 maximum-requests=1000 umask=0007
WSGIProcessGroup tracking
WSGIApplicationGroup tracking
WSGIPassAuthorization On
ErrorLog /var/log/apache2/tracking.error.log
CustomLog /var/log/apache2/tracking.access.log combined
LogLevel warn
We haven't really done any specific parameter tuning for mod_wsgi beyond what you see in this conf.
No comments:
Post a Comment