This is a proposed Canonical Question about understanding and debugging the software firewall on Linux systems.
In response to EEAA's answer and @Shog's comment that we need a suitable canonical Q&A for closing common relatively simple questions about iptables.
What is a structured method to debug problems with the Linux software firewall, the netfilter packet filtering framework, commonly referred to by the userland interface iptables?
What are common pitfalls, recurring questions and simple or slightly more obscure things to check that an occasional firewall administrator might overlook or otherwise benefit from knowing?
Even when you use tooling such as UFW, FirewallD (aka firewall-cmd), Shorewall or similar you might benefit from looking under the hood without the abstraction layer those tools offer.
This question is not intended as a How-To for building firewalls: check the product documentation for that and
for instance contribute recipes to iptables Trips & Tricks or search the tagged iptables ufw firewalld firewall-cmd questions
for existing frequent and well regarded high-scoring Q&A's.
Answer
Viewing and modifying the firewall configuration requires administrator privileges (root) as does
opening services in the restricted port number range. That means that you should either be logged in
as root or alternatively use sudo to run the command as root. I'll try to mark such commands with the optional [sudo].
Contents:
- Order matters or the difference between
-Iand-A - Display the current firewall configuration
- Interpreting the ouput of
iptables -L -v -n - Know your environment
- The INPUT and FORWARD chains
- Kernel modules
The thing to remember is that firewall rules are checked in the order they are listed. The kernel will stop processing the chain when a rule is triggered that will either allow or dis-allow a packet or connection.
I think the most common mistake for novice firewall administrators is that they follow the correct instructions to open a new port, such as the one below:
[sudo] iptables -A INPUT -i eth0 -p tcp --dport 8080 -j ACCEPT
and then discover that it won't take effect.
The reason for that is that the -A option adds that new rule, after all existing rules
and since very often the final rule in the existing firewall was one that blocks all traffic that isn't explicitely allowed, resulting in
...
7 2515K 327M REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
8 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080
Or equivalent in iptables-save:
...
iptables -A INPUT -j REJECT
iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
and the new rule opening TCP port 8080 will never be reached. (as evidenced by the counters stubbornly remaining at 0 packets and zero bytes).
By inserting the rule with -I the new rule would have been the first in the chain and will work.
My recommendation for the firewall administrator is to look at the actual configuration the Linux kernel is running, rather
than trying to diagnose firewall issues from userfriendly tools. Often once you understand the underlying issues you can
easily resolve them in a matter supported by those tools.
The command [sudo] iptables -L -v -n is your friend (although some people like iptables-save better). Often when discussing configurations it is useful to use the --line-numbers option as well
to number lines. Refering to rule #X makes discussing them somewhat easier.
Note: NAT rules are included in the iptables-save output but have to listed separately by adding the -t nat option i.e, [sudo] iptables -L -v -n -t nat --line-numbers.
Running the command multiple times and checking for incrementing counters can be a useful tool to see if a new rule actually gets triggered.
[root@host ~]# iptables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
1 784K 65M fail2ban-SSH tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
2 2789K 866M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
3 15 1384 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
4 44295 2346K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
5 40120 2370K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80
6 16409 688K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:443
7 2515K 327M REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
1 0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT 25 packets, 1634 bytes)
num pkts bytes target prot opt in out source destination
Chain fail2ban-SSH (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 REJECT all -- * * 117.239.37.150 0.0.0.0/0 reject-with icmp-port-unreachable
2 4 412 REJECT all -- * * 117.253.208.237 0.0.0.0/0 reject-with icmp-port-unreachable
Alternatively the output of iptables-save gives a script that can regenerate the above firewall configuration:
[root@host ~]# iptables-save
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [441:59938]
:fail2ban-SSH - [0:0]
-A INPUT -p tcp -m tcp --dport 22 -j fail2ban-SSH
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A fail2ban-SSH -s 117.239.37.150/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-SSH -s 117.253.208.237/32 -j REJECT --reject-with icmp-port-unreachable
COMMIT
It is a matter of preference what you'll find easier to understand.
The Policy sets the default action the chain uses when no explicit rule matches. In the INPUT chain that is set to ACCEPT all traffic.
The first rule in the INPUT chain is immediately an interesting one, it sends all traffic (source 0.0.0.0/0 and destination 0.0.0.0/0) destined for TCP port 22 (tcp dpt:22) the default port for SSH to a custom target (fail2ban-SSH).
As the name indicates this rule is maintained by fail2ban (a security product that among other things scans system log files for possible abuse and blocks the IP-address of the abuser).
That rule would have been created by an iptables commandline similar to iptables -I INPUT -p tcp -m tcp --dport 22 -j fail2ban-SSH or is found in the output of
iptables-save as -A INPUT -p tcp -m tcp --dport 22 -j fail2ban-SSH. Often you'll find either of those notations in documentation.
The counters indicate that this rule has matched 784'000 packets and 65 Megabytes of data.
Traffic that matches to this first rule is then processed by the fail2ban-SSH chain that, as a non-standard chain, gets listed below the OUTPUT chain.
That chain consists of two rules, one for each abuser (source ip-address 117.253.221.166 or 58.218.211.166) that is blocked (with a reject-with icm-port-unreachable).
-A fail2ban-SSH -s 117.253.221.166/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-SSH -s 58.218.211.166/32 -j REJECT --reject-with icmp-port-unreachable
SSH packets that aren't from those blocked hosts are neither allowed nor dis-allowed yet and will now that the custom chain is completed will be checked against the second rule in the INPUT chain.
All packets that weren't destined for port 22 passed the first rule in the INPUT chain and will also be evaluated in INPUT rule #2.
The INPUT rule number 2 makes this is intended to be a statefull firewall, which tracks connections. That has some advantages, only the packets for new connections need to be checked against
the full rule-set, but once allowed additional packets belonging to an established or related connection are accepted without further checking.
Input rule #2 matches all open and related connections and packets matching that rule will not need to be evaluated further.
Note: rule changes in the configuration of a stateful firewall will only impact new connections, not established connections.
In contrast a simple packet filter tests every packet against the full rule-set, without tracking connection state. In such a firewall no state keywords would be used.
INPUT rule #3 is quite boring, all traffic connecting to the loopback (lo or 127.0.0.1) interface is allowed.
INPUT rules 4, 5 and 6 are used to open TCP ports 22, 80 and 443 (the default ports for resp. SSH, HTTP and HTTPS) by granting access to NEW connections
(existing connections are already allowed by INPUT rule 2).
In a stateless firewall those rules would appear without the state attributes:
4 44295 2346K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0
5 40120 2370K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0
6 16409 688K ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0
or
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
The final INPUT rule, #7 is a rule that blocks all traffic that was NOT granted access in INPUT rules 1-7. A fairly common convention: everything not allowed is denied. In theory this rule could have been omitted by setting the default POLICY to REJECT.
Always investigate the whole chain.
4.1 . The settings in a software firewall won't effect security settings maintained elsewhere in the network,
i.e. despite opening up a network service with iptables the unmodified access control lists on routers or other firewalls in your network may still block traffic...
4.2 . When no service is listening you won't be able to connect and get a connection refused error, regardless of firewall settings. Therefore:
- Confirm that a service is listening (on the correct network interface/ip-address) and using the port numbers you expect with
[sudo] netstat -plnutor alternatively usess -tnlp. - If your services are not yet supposed to be running, emulate a simple listener with for instance netcat:
[sudo] nc -l -p 123oropenssl s_server -accept 1234 [options]if you need a TLS/SSL listener (checkman s_serverfor options). - Verify that you can connect from the server itself i.e.
telnetor123 echo "Hello" | ncor when testing TLS/SSL secured service123 openssl s_client -connect, before trying the same from a remote host.:1234
4.3 . Understand the protocols used by your services. You can't properly enable/disable services you don't sufficiently understand. For instance:
- is TCP or UDP used or both (as with DNS)?
- is the service using a fixed default port (for instance something like TCP port 80 for a webserver)?
- alternatively is a dynamic port number chosen that can vary (i.e. RPC services like classic NFS that register with Portmap)?
- infamous FTP even uses two ports, both a fixed and a dynamic port number when configured to use passive mode...
- the service, port and protocol descriptions in
/etc/servicesdo not necessarily match with the actual service using a port.
4.4 . The kernel packet filter is not the only thing that may restrict network connectivity:
- SELinux might also be restricting network services.
getenforcewill confirm if SELinux is running. - Although becoming slightly obscure TCP Wrappers are still a powerful tool to enforce network security. Check with
ldd /path/to/service |grep libwrapand the/hosts.[allow|deny]control files.
The concept of chains is more thoroughly explained here but the short of it is:
The INPUT chain is where you open and/or close network ports for services running locally, on the host where you issue the iptables commands.
The FORWARD chain is where you apply rules to filter traffic that gets forwarded by the kernel to other systems,
actual systems but also Docker containers and Virtual guest Servers servers when your Linux machine is acting as a bridge, router, hypervisor and/or does network address translation and port forwarding.
A common misconception is that since a docker container or KVM guest runs locally, the filter rules that apply should be in the INPUT chain, but that is usually not the case.
Since the packet filter runs within the Linux kernel it can also be compiled as dynamic module, multiple modules actually. Most distributions include netfilter as modules and the
required netfilter modules will get loaded into the kernel as needed,
but for some modules a firewall administrator will need to manually ensure they get loaded. This primarily concerns the connection tracking modules, such as nf_conntrack_ftp which can be loaded with insmod.
The modules currently loaded into the running kernel can be displayed with lsmod.
The method to ensure modules are loaded persistently across reboots depends on the Linux distribution.
No comments:
Post a Comment