Apache load balancer limits with Tomcat over AJP

I have Apache acting as a load balancer in front of 3 Tomcat servers. Occasionally, Apache returns 503 responses, which I would like to remove completely. All 4 servers are not under significant load in terms of CPU, memory, or disk, so I am a little unsure what is reaching it's limits or why. 503s are returned when all workers are in error state - whatever that means. Here are the details:

Apache config:

  StartServers           30

  MinSpareServers        30
  MaxSpareServers        60
  MaxClients            200
  MaxRequestsPerChild  1000


...


  AddDefaultCharset Off

  Order deny,allow
  Allow from all


# Tomcat HA cluster

  BalancerMember ajp://10.176.201.9:8009 keepalive=On retry=1 timeout=1 ping=1
  BalancerMember ajp://10.176.201.10:8009 keepalive=On retry=1 timeout=1 ping=1
  BalancerMember ajp://10.176.219.168:8009 keepalive=On retry=1 timeout=1 ping=1



# Passes thru track. or api.
ProxyPreserveHost On
ProxyStatus On

# Original tracker
ProxyPass /m  balancer://mycluster/m
ProxyPassReverse /m balancer://mycluster/m

Tomcat config:

  
  
  
  

  
    
               connectionTimeout="20000" 
               redirectPort="8443" />

    

    
                unpackWARs="true" autoDeploy="true"
          xmlValidation="false" xmlNamespaceAware="false">
    

  

Apache error log:

[Mon Mar 22 18:39:47 2010] [error] (70007)The timeout specified has expired: proxy: AJP: attempt to connect to 10.176.201.10:8009 (10.176.201.10) failed
[Mon Mar 22 18:39:47 2010] [error] ap_proxy_connect_backend disabling worker for (10.176.201.10)
[Mon Mar 22 18:39:47 2010] [error] proxy: AJP: failed to make connection to backend: 10.176.201.10

[Mon Mar 22 18:39:47 2010] [error] (70007)The timeout specified has expired: proxy: AJP: attempt to connect to 10.176.201.9:8009 (10.176.201.9) failed
[Mon Mar 22 18:39:47 2010] [error] ap_proxy_connect_backend disabling worker for (10.176.201.9)
[Mon Mar 22 18:39:47 2010] [error] proxy: AJP: failed to make connection to backend: 10.176.201.9
[Mon Mar 22 18:39:47 2010] [error] (70007)The timeout specified has expired: proxy: AJP: attempt to connect to 10.176.219.168:8009 (10.176.219.168) failed
[Mon Mar 22 18:39:47 2010] [error] ap_proxy_connect_backend disabling worker for (10.176.219.168)
[Mon Mar 22 18:39:47 2010] [error] proxy: AJP: failed to make connection to backend: 10.176.219.168
[Mon Mar 22 18:39:47 2010] [error] proxy: BALANCER: (balancer://mycluster). All workers are in error state
[Mon Mar 22 18:39:47 2010] [error] proxy: BALANCER: (balancer://mycluster). All workers are in error state
[Mon Mar 22 18:39:47 2010] [error] proxy: BALANCER: (balancer://mycluster). All workers are in error state
[Mon Mar 22 18:39:47 2010] [error] proxy: BALANCER: (balancer://mycluster). All workers are in error state

[Mon Mar 22 18:39:47 2010] [error] proxy: BALANCER: (balancer://mycluster). All workers are in error state
[Mon Mar 22 18:39:47 2010] [error] proxy: BALANCER: (balancer://mycluster). All workers are in error state

Load balancer top info:

top - 23:44:11 up 210 days,  4:32,  1 user,  load average: 0.10, 0.11, 0.09
Tasks: 135 total,   2 running, 133 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.2%sy,  0.0%ni, 99.2%id,  0.1%wa,  0.0%hi,  0.1%si,  0.3%st

Mem:    524508k total,   517132k used,     7376k free,     9124k buffers
Swap:  1048568k total,      352k used,  1048216k free,   334720k cached

Tomcat top info:

top - 23:47:12 up 210 days,  3:07,  1 user,  load average: 0.02, 0.04, 0.00
Tasks:  63 total,   1 running,  62 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.0%sy,  0.0%ni, 99.8%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:   2097372k total,  2080888k used,    16484k free,    21464k buffers
Swap:  4194296k total,      380k used,  4193916k free,  1520912k cached

Catalina.out does not have any error messages in it.

According to Apache's server status, it seems to be maxing out at 143 requests/sec. I believe the servers can handle substantially more load than they are, so any hints about low default limits or other reasons why this setup would be maxing out would be greatly appreciated.

Answer

Solution for this Problem is pretty simple:

add to Proxypass:

BalancerMember ajp://10.176.201.9:8009 keepalive=On ttl=60

add to Tomcats Server.xml:

Connector port="8009" protocol="AJP/1.3" redirectPort="8443 connectionTimeout="60000"

After these changes everything should be work fine :-)