Sunday, December 16, 2018

apache 2.2 - How to minimise effect of mischievous, persistent POST requests

For a few months now one of our shared hosting servers has been persistently and constantly hammered by "POST /" requests from what must be hundreds of thousands of individual IPs. On a number of occasions this has overwhelmed the server and led to a denial of service-type outage. The target domain is pretty boring (a small Estate Agent) so whilst this appears to be malicious I can't understand motive of this long-running and 99% unsuccessful attack.



A typical request (taken from TCPDUMP) looks something like this:



POST / HTTP/1.1
Accept: */*
Accept-Language: en-us
Content-Type: application/octet-stream

Content-Length: 570
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: xxx.co.uk
Connection: Keep-Alive
Cache-Control: no-cache

2+cIPSyYVJFqB9xPFzWoLj9seNyEKIcuIJz/yfkc9tVP+orXgjDk8ywglufcXsMz
bVP4XLcowz/fQtsn2kceQEj/EaEWx/GEbcC3eTbCbTube0suAfEGje3qISKQJ+ka
HaChqun3whii3OTh7vCayGV72lh4raLRandKC5g/73wgQ9Jzh2OLIzNvsiEMSJco
yG+4i35XJMvX7ovx8qJkyByHUIeE5G5M2Kp97O4sOT4jTAK2y/KAMjf6oFgtAJhI

K4/HdcnyfNdI3/4RJXlrSfhUQAc+qhGMEL7AZdtzgRub7lnu+hbuPGZvS3rF1MvL
WK1q4mrnZr0Q3m0bWkzsMZCndQ7fqOBafchjprhn4JKPsjO+upRm2m+irvmJjqnl
sDiR3fnD6pzbWyLTm2qonMJPCll3p6zg06gEfIaW04t9r89/PdHgz8AU8nzO4BX8
qwTG6dSjgbowHyJQmud8Ro+ZT+gHfw/YQUrBqKm7RoFmfJzUoOCKaP1LTwHfI1Gc
E+L8bwQV6ztKBwVn2NqbE83SAXYr9E0QkpaxGg==


We haven't been able to determine what's in the POST request as it looks like garbage, but I'm not sure its relevant. It's not base64 encoded.



To reduce the amount of bandwidth being used up by responses to this request we have banned the use of POST requests in the Apache2 configuration:






Order deny,allow
Deny from all




This restricts the response size to just a simple 403 Forbidden message, rather than the client's usual homepage.




To try and block the IPs doing this we've tried piping the access log, filtering for the POST request, and feeding this directly into iptables:



tail -f /var/www/vhosts/xxx.co.uk/statistics/logs/access_log | grep "POST / " | awk '{print $1}' | xargs -I{} iptables -A INPUT -s {} -j DROP


This works well and reduces the effect of the problem, but it is relentless and we usually have to clear the iptables rule set when it reaches 50-60k due to iptables/kernel problems. It's not a solution as I can't just leave this running for a few weeks until whoever is responsible gets the message and gives up.



We've turned off KeepAlive for this particular VirtualHost too to keep the number of occupied Apache workers to a minimum which has helped, but it's not a solution.




Does anybody have any better ideas on how to blackhole these requests, on a scale of hundreds of thousands of remote IPs, or to reduce the impact on Apache to the absolute minimum? The best I can do at the moment is configuring it to send a 403 Forbidden, combined with IP-blocking for a few hours...



Thanks!

No comments:

Post a Comment

linux - How to SSH to ec2 instance in VPC private subnet via NAT server

I have created a VPC in aws with a public subnet and a private subnet. The private subnet does not have direct access to external network. S...