Wednesday, February 27, 2019

Apache hangs after every 2-3 weeks, with closed_wait connections increasing until it is restarted

We have a cluster setup of 3 server, on all 3 servers weblogic is running to host website. On the top of weblogic using Apache as load balancer.



Last year I had upgraded Apache http server from 2.4.9 to 2.4.29. Before upgrade things were working fine but post upgrade after every 2-4 weeks Apache hangs. Website becomes inaccessible, on checking the server we don't see any error in fact the moment when apache hangs it stops logging too.



We monitored and found no abnormal memory usage, no suspicious type of request to website, no suspicious process running before the issue occurs.



After exploring over the internet, I found that problem could be related to MPM configuration. Apache was using event MPM but I found that event MPM configuration was not declared in apache configuration, I think during upgrade default configuration was replaced by old configuration.



So we have modified MPM configuration to use below event mpm configuration and this change worked. We did not face this issue for next 6 months.





StartServers 4
MinSpareThreads 256
MaxSpareThreads 512
ThreadsPerChild 64
MaxRequestWorkers 2048
MaxConnectionsPerChild 0



We thought it is fixed but now after 6 months Apache hangs again, only symptoms we can see increasing closed_wait connections from 20 to 550 in next 1 hours, no accessibility of website, apache stopped logging to it's ssl and error logs, we don't see any abnormal behaviour in different running process data, memory usage, also don't see any suspicious request before issue occurred.

No comments:

Post a Comment

linux - How to SSH to ec2 instance in VPC private subnet via NAT server

I have created a VPC in aws with a public subnet and a private subnet. The private subnet does not have direct access to external network. S...