Wednesday, December 28, 2016

linux - IPTables + Limit module: Why doesn't limit-burst get completely used?

Long time reader, first time poster.. yada yada yada..



Anyways, I am hopeful someone out there has some extensive iptables/netfilter LIMIT or HASHLIMIT module experience and explain the behavior I'm witnessing.



Background:

We have a webserver and want to limit how many connections a customer can have over the course of a month (HTTP keepalives are off, btw). So, I am trying to use the iptables LIMIT module to limit the number of their new connections to a set number per month (let's say 500). iptables LIMIT module uses a "token bucket" algorithm so I should be able to set the limit-burst (bucket size) to 500 and the limit (refill rate) to 500 divided by 28 days or about 18/day. This will make sure the bucket gets refilled in a month's time (4 weeks) if it is every completely emptied. (I understand this will actually grant more than exactly 500 but it should be close enough for our needs).



Here are my iptables rules (We group IPs using ipset. LimBurTest4 contains my source testing machines)



Chain INPUT (policy DROP 2316 packets, 186K bytes)
pkts bytes target prot opt in out source destination
2952K 626M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Accept outgoing return traffic */
379 13702 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 8 state NEW limit: avg 1/sec burst 1
377 30868 DROP icmp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x00 /* Block NULL packets */

73 14728 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:!0x17/0x02 state NEW /* Block SYN flood */
0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x3F/0x3F /* Block XMAS packets */
24 120 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 match-set LimBurTest4 src tcp dpt:443 limit: avg 18/day burst 500 /* LimitBurst Test */
76 30180 LOG tcp -- * * 0.0.0.0/0 0.0.0.0/0 match-set LimBurTest4 src tcp dpt:443 /* LimitBurst Testing */ LOG flags 0 level 4 prefix "LimBurTest Over Quota "
2522 138K REJECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 /* Reject queries */ reject-with tcp-reset


I add the LIMIT rule like this:



iptables -I INPUT 7 -m set --match-set LimBurTest4 src -p tcp --dport 443 -m limit --limit 18/day --limit-burst 500 -m comment --comment "Limit Burst Test" -j ACCEPT



Testing:
Then I created a simple shell script to do requests to my webserver one after the other using curl. Each successful request takes about 0.20 ms. Output of the script looks like this:



./limBurTest.sh
1 - 200 - 0.257 ms
2 - 200 - 0.193 ms
3 - 200 - 0.155 ms
etc.

etc.


Outcome:
My expectation of this configuration would be to quickly (in a couple seconds) use up all 500 connections before I start seeing rejected connections. However, that is not happening at all. Instead, my test script made 24 successful connections and then the rest were rejected. For example, in the above iptables output, I ran my shell script in a loop 100 times and you can see 24 ACCEPT rule matches and 76 LOG rule matches after the ACCEPT rule. I have tested this on CentOS 6.8 and Ubuntu 16.04 and both behave this way, but this seems contrary to the documentation. Why can I not use up all 500 connections specified by limit-burst?



And, yes, of course, I have done extensive googling and have seen lots of examples of people saying that the LIMIT module is supposed to work exactly as I've described. And I've read the netfilter docs numerous times.



What am I missing here?




Thank you in advance.

No comments:

Post a Comment

linux - How to SSH to ec2 instance in VPC private subnet via NAT server

I have created a VPC in aws with a public subnet and a private subnet. The private subnet does not have direct access to external network. S...