log files - Is Apache stalled? /server-status shows over 240 requests like "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"

Some details:

• Webserver: Apache/2.2.13 (FreeBSD) mod_ssl/2.2.13 OpenSSL/0.9.8e


• OS: FreeBSD 7.2-RELEASE


• This is a FreeBSD Jail.



• I believe I use the Apache 'prefork' MPM (I run the default for FreeBSD).


• I use the default values for MaxClients (256)

I have enabled mod_status, with "ExtendedStatus On". When I view /server-status , I see a handful of regular requests. I also see over 240 requests from the 'localhost', like these.

37-0    -   0/0/1   .   0.00    1510    0   0.0 0.00    0.00    127.0.0.2   www.example.gov OPTIONS * HTTP/1.0
38-0    -   0/0/1   .   0.00    1509    0   0.0 0.00    0.00    127.0.0.2   www.example.gov OPTIONS * HTTP/1.0
39-0    -   0/0/3   .   0.00    1482    0   0.0 0.00    0.00    127.0.0.2   www.example.gov OPTIONS * HTTP/1.0
40-0    -   0/0/6   .   0.00    1445    0   0.0 0.00    0.00    127.0.0.2   www.example.gov OPTIONS * HTTP/1.0

I also see about 2417 requests yesterday from the localhost, like these:

Apr 14 11:16:40 192.168.16.127 httpd[431]: www.example.gov 127.0.0.2 - - [15/Apr/2010:11:16:40 -0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache (internal dummy connection)"

The page at http://wiki.apache.org/httpd/InternalDummyConnection says "These requests are perfectly normal and you do not, in general, need to worry about them", but I'm not so sure.

Why are there over 230 of these? Are these active connections? If I have "MaxClients 256", and over 230 of these connections, it seems that my webserver is dangerously close to running out of available connections. It also seems like Apache should only need a handful of these "internal dummy connections"

We actually had two unexplained outages last night, and I am wondering if these "internal dummy connection" caused us to run out of available connections.

UPDATE 2010/04/16

It is 8 hours later. The /server-status page still shows that there are 243 lines which say "www.example.gov OPTION *". I believe these connections are not active. The server is mostly idle (1 requests currently being processed, 9 idle workers). There are only 18 active httpd processes on the Unix host.

If these connections are not active, why do they show up under /server-status? I would have expected them to expire a few minutes after they were initialized.

Answer

Well, this had a surprise answer. This was caused by a filesystem problem when we took UFS filesystem snapshots at midnight.

This seems to be caused by a FreeBSD UFS bug. We use FreeBSD Jails on a FreeBSD Host, with the default UFS filesystem. The UFS filesystem is large -- 1.8TB.

Once per night, we run a backup using 'dump(8)'. dump(8) was creating a snapshot of the filesystem before backing it up, and this froze the filesystem. Dump is supposed to work with filesystems less then 2TB, but it failed in our case. This guy had the same problem.

(I moved my answer from the question section down here to the answer section. stefan, 20100608)