apache 2.2 - Why is iptables not blocking an IP address? (LB/proxy version)

WARNING: Long. Lots of info here.

3 years ago someone asked Why is iptables not blocking an IP address? and it turned out the reason was because the servers were behind CloudFlare which made it impossible to block IP addresses directly they way they wanted to unless you use it differently. Any reverse proxy or load balancer would cause the same thing.

Similarly we have setup fail2ban with a rule to ban any bots which attempt to brute-force their way into the administrative login or spam xmlrpc. The site is sitting behind a load balancer so obviously we can't directly ban the IP address but iptables is supposed to be accepting the connection and pattern matching the packet data to ban specific traffic.

This is fail2ban jail.conf config:

[wp-auth] 
enabled = true
filter = wp-auth
action = iptables-proxy[name = lb, port = http, protocol = tcp]

         sendmail-whois[name=LoginDetect, dest=ITemail@ourdomain.com, sender=acceptablebotbot@ourdomain.com, sendername="Fail2Ban"]
logpath = /obfuscated/path/to/site/transfer_log
bantime = 604800
maxretry = 4
findtime = 120

This is the simply pattern match for wp-login requests:

[Definition]

failregex = ^ .* "POST /wp-login.php
ignoreip = # our ip address

This is our fail2ban iptables action which is supposed to be able to block these bots but for the most part doesn't seem to. It is from the CentOS site Tips section for fail2ban behind a proxy. For the sake of brevity I've left only the section header comments in place.

# Fail2Ban configuration file
#
# Author: Centos.Tips
#


[INCLUDES]

before = iptables-blocktype.conf

[Definition]

# Option:  actionstart
actionstart = iptables -N fail2ban-
              iptables -A fail2ban- -j RETURN

              iptables -I  -p  --dport  -j fail2ban-

# Option:  actionstop
actionstop = iptables -D  -p  --dport  -j fail2ban-
             iptables -F fail2ban-
             iptables -X fail2ban-

# Option:  actioncheck
actioncheck = iptables -n -L  | grep -q 'fail2ban-[ \t]'


# Option:  actionban
actionban = iptables -I fail2ban- 1 -p tcp --dport 80 -m string --algo bm --string 'X-Forwarded-For: ' -j DROP

# Option:  actionunban
actionunban = iptables -D fail2ban- -p tcp --dport 80 -m string --algo bm --string 'X-Forwarded-For: ' -j DROP

[Init]
# Default name of the chain
name = default


# Option:  port
port = http

# Option:  protocol
protocol = tcp

# Option:  chain
chain = INPUT    

So as I mentioned the site is on a pair of servers behind an elastic load balancer and seems to work in test. We can add any of our own IP addresses and we cannot reach the site. Despite this bots seem to be able to get through.

[root:~/] iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N fail2ban-SSH
-N fail2ban-lb
-A INPUT -p tcp -m tcp --dport 80 -j fail2ban-lb
-A INPUT -p tcp -m tcp --dport 22 -j fail2ban-SSH

-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5666 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 3306 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 24007:24020 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

-A fail2ban-SSH -j RETURN
-A fail2ban-lb -p tcp -m tcp --dport 80 -m string --string "X-Forwarded-For: 91.200.12.33" --algo bm --to 65535 -j DROP
-A fail2ban-lb -p tcp -m tcp --dport 80 -m string --string "X-Forwarded-For: 91.134.50.10" --algo bm --to 65535 -j DROP
-A fail2ban-lb -p tcp -m tcp --dport 80 -m string --string "X-Forwarded-For: 160.202.163.125" --algo bm --to 65535 -j DROP
-A fail2ban-lb -p tcp -m tcp --dport 80 -m string --string "X-Forwarded-For: 162.243.68.232" --algo bm --to 65535 -j DROP
-A fail2ban-lb -j RETURN

Port 80 is the only port open to all. All others are ACL'd via AWS Security Groups. IPtables appears to be processing in the correct order and should therefore be blocking these IPs based on their X-Forwarded-For header. There is a Firefox plugin which allows you to send these headers with initial requests and we get blocked as a result with any of these bot IPs as well.

The source IP address does not appear to be forging the X-Forwarded-For header as we've been playing with as the ELB rewrites them anyway. tcpdump does not show any extra information on the packet at the server level.

22:07:14.309998 IP ip-10-198-178-233.ec2.internal.11054 > ec2-10.4.8.71.http: Flags [P.], seq 2545:3054, ack 19506, win 166, options [nop,nop,TS val     592575835 ecr 2772410449], length 509
E..1..@.@..9
...
f.p+..P.Nz.
20............
#Q.[.?.QPOST /wp-login.php HTTP/1.1
host: www.thiswebsite.com
Accept: */*

Accept-Language: zh-cn
Cache-Control: no-cache
Content-Type: application/x-www-form-urlencoded
Referer: http://www.thiswebsite.com/wp-login.php
User-Agent: Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; 125LA; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022)
X-Forwarded-For: 91.200.12.33
X-Forwarded-Port: 80
X-Forwarded-Proto: http
Content-Length: 21
Connection: keep-alive

These requests are all being logged in the transfer_log.
When we do the same thing and forge the X-Forwarded-For we get caught by iptables before ever reaching Apache. tcpdump also shows our extra IPs.

20:10:25.378873 IP ip-10-198-178-233.ec2.internal.11054 > ec2-10.4.8.71.http: Flags [P.], seq 3157:3860, ack 124583, win 267, options [nop,nop,TS     val 526293643 ecr 2507283790], length 703
E...Tf@.@.[.
...
f.p,O.P...GU........m.....
.^...r.QPOST /wp-login.php HTTP/1.1

host: www.thiswebsite.com
Accept: /
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.5
Cache-Control: no-cache
Cookie: __utma=190528439.16251225.1476378792.1478280188.1478289736.3; __utmz=190528439.1476378792.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none);     _icl_current_language=en; __utmc=190528439; __utmb=190528439.2.10.1478289736; __utmt=1
Pragma: no-cache
Referer: http://www.thiswebsite.com/
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0
X-Forwarded-For: 91.200.12.33, 

X-Forwarded-Port: 80
X-Forwarded-Proto: http
Connection: keep-alive

I also have the ELB access log here which I expect to see an entry for, just not the Apache transfer logs.

2016-11-07T22:07:14.309917Z mLB 91.200.12.33:60407 10.4.8.71:80 0.000079 1.99244 0.000091 200 200 21 3245 "POST http://www.thiswebsite.com:80/wp-login.php HTTP/1.1" "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; 125LA; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022)" - -

So the IP address (at least according to the ELB) does not appear to be forced at the X-Forwarded-For level. Why is traffic from it not being blocked? The IP address also shows up constantly in the fail2ban log with the usual:

fail2ban.actions[11535]: INFO [wp-auth] 91.200.12.33 already banned