System: fail2ban and iptables

Around the beginning of 2005 we saw an increase in brute-force ssh attacks - people or robots trying different combinations of
username and password to log into remote servers. A quick search on this topic returns many references to iptables and ipchains
but noone really explained how they work.

Having just gone through this learning curve myself, and found a satisfactory solution in the fail2ban package, I'm going to try and
explain how to achieve the simple goal of banning IP addresses that make repeated failed ssh login attempts.

If you want more technical information regarding firewalls and iptables in particular, see the References section at the bottom of this
page.

Note: The following relates to the default config file provided with fail2ban 0.6.0 in the Debian unstable distribution and should be used for information
purposes only. We take no responsibility for any consequences of following the instructions on this page.

1. What is/are iptables?

iptables is the firewall administration program for the Netfilter firewall mechanism which is built into the Linux kernel.

To view your current 'firewall chains' you need to run the following command as root:

# iptables -L

This will show a list of 'chains' called INPUT, FORWARD and OUTPUT - these are always present - followed by any custom chains. The
default (blank) settings will look something like:

Chain INPUT (policy ACCEPT)

target prot opt source destination

Chain FORWARD (policy ACCEPT)

target prot opt source destination

Chain OUTPUT (policy ACCEPT)

target prot opt source destination

If that's what you're seeing then there are no existing firewall rules and it's probably safe to proceed. Otherwise you may want to
check with the sysadmin who installed any existing firewall rules before making changes.

Note: you can also use iptables -L INPUT to list just the INPUT chain. Other useful options are to add -n to display values in numerical format or -v for
verbose mode. Finally iptables -h will display all command-line options.

2. Installing and configuring fail2ban

The fail2ban package is available under Debian/unstable and also as a download for other Linux systems. See the Fail2Ban website
linked under Resources at the bottom of the page for details. To install on Debian:

# apt-get -t unstable install fail2ban

If you run this command then fail2ban will be installed and already running as a daemon. However you might want to edit the
configuration file and stop/start the daemon to get it running how you want. The configuration file can be found at /etc/fail2ban.
conf.

The configuration file is broken up into sections. Most entries don't need to be changed but there are a few that you might want to
edit. The DEFAULT settings apply to all sections:

[DEFAULT]

maxfailures = 3
bantime = 900
findtime = 600

This says that after three failed access attempts are detected from a single IP address within 600 seconds or 10 minutes (findtime),
then that address will be automatically blocked for 900 seconds (bantime). In most cases you won't see them again as the bots will
move on to another server.

[MAIL]

enabled = true
to = root@localhost
Needs to be set to true for emails to be sent - to the root user in this case. Only do this if you're actually going to check the root
mailbox otherwise you're just wasting disc space.

[Apache]

enabled = false

For now we're not going to worry about monitoring the Apache log files.

[SSH]

enabled = true
logfile = /var/log/auth.log

fail2ban will monitor the auth.log file for failed access attempts. As soon as the daemon is running your ssh port (22) will be
protected from brute-force attacks - preventing more than a small number of attempts at one time.

To check if it's running:

# /etc/init.d/fail2ban status
Status of fail2ban: fail2ban is running.

Note: these commands will differ for different flavours of Linux.

To stop/start the daemon after making configuration changes:

# /etc/init.d/fail2ban stop
Stopping fail2ban: .done
# /etc/init.d/fail2ban start
Starting fail2ban: .done

Actions taken by the daemon are logged by default in /var/log/fail2ban.log and you can change the verbosity in the conf file to
one of: 0 - WARN, 1 - INFO or 2 - DEBUG.

3. What does fail2ban do with iptables?

This code runs when the daemon is started and adds new firewall rules using iptables:

fwstart = iptables -N fail2ban-ssh

iptables -A fail2ban-ssh -j RETURN
iptables -I INPUT -p tcp --dport ssh -j fail2ban-ssh

Translation:

● create a NEW chain called fail2ban-ssh;

● APPEND a RETURN command to the end of the fail2ban-ssh chain (same as return in PHP or other languages);
● INSERT a rule at the start of the built-in INPUT chain that redirects SSH packets to the fail2ban-ssh chain.

This adds the following to the output of iptables -L:

Chain INPUT (policy ACCEPT)

target prot opt source destination
fail2ban-ssh tcp -- anywhere anywhere tcp dpt:ssh

Chain fail2ban-ssh (1 references)

target prot opt source destination
RETURN all -- anywhere anywhere

Because there are no actual rules in the fail2ban-ssh chain, connection attempts using SSH are simply redirected from the INPUT
chain to the fail2ban-ssh chain and then sent straight back. What it does mean however is that we (or the fail2ban daemon to be
precise) can insert new rules at any time and they will be applied to all incoming SSH traffic.

The fwend commands simply reverse this process, removing the JUMP command and and then FLUSHing and DELETEing the fail2ban-
ssh chain:

fwend = iptables -D INPUT -p tcp --dport ssh -j fail2ban-ssh

iptables -F fail2ban-ssh
iptables -X fail2ban-ssh

Note: You should always stop fail2ban before editing the config file - so that it cleans up existing rules.
To ban an IP address the following command is run, with <ip> replaced by the actual IP address or hostname captured by the
failregex regular expression (see below):

fwban = iptables -I fail2ban-ssh 1 -s <ip> -j DROP

Translation:

● Insert a rule as line 1 of the fail2ban-ssh chain to DROP all packets from the listed IP address or hostname.

When a fwban event is triggered, the output of iptables

-L fail2ban-ssh --line-numbers becomes:

Chain fail2ban-ssh (1 references)

num target prot opt source destination
1 DROP all -- <host> anywhere
2 RETURN all -- anywhere anywhere

where <host> is the problem ip address or domain. While this rule is in effect all ssh requests from <host> will be silently dropped. If
a second fwban even occurs while the first is in place it will appear as line 1 and the other commands moved down the chain.

The fwunban command simply removes the DROP command from the chain. It's called after 10 minutes or whatever time you have set
for the bantime in fail2ban.conf:

fwunban = iptables -D fail2ban-ssh -s <ip> -j DROP

4. The failregex regular expression

The most complicated part of the configuration file is the regular expression that is applied to auth.log to determine what to count as
an access failure.

failregex = : (?:(?:Authentication failure|Failed [-/\w+]+) for(?: illegal user)?|Illegal user|Did

not receive identification) .* from (?P<host>\S*)

Note: the regular expression used here has been optimised for the auth.log in Debian Linux (Sarge) and may need to be modified slightly for other
platforms.

In short, this matches any/all of the following auth.log entries and triggers a fwban event with the ip address or domain (host) that's
been captured by the regular expression:

Authentication failure for username from <host>

Failed password for illegal user username from <host>
Failed password for username from <host>
Failed keyboard-interactive/pam for username from <host>
Illegal user username from <host>
Did not receive identification string from <host>

For anyone trying to reverse-engineer the regular expression:

● (?:.*) is a non-grouping version of regular parentheses; and

● (?P<name>.*) is similar to regular parentheses, but the substring matched by the group is accessible via the symbolic group
name name.

The single matched variable <host> will be added to the fail2ban-ssh chain if maxfailures or more matches occur within findtime.
It's that simple.

You will see a lot of different (and usually simpler) failregex patterns that don't contain the <host> capture. The reason for having
the named capture in the regexp is to counter a bug that allows malicious users to trick your server into blocking non-hostile
addresses:

"fail2ban's approach to identifying an IP address in a login failure line is to scan the line for all IP addresses.

Since it is possible to generate false logins from accounts such as 10.2.28.2, it is possible to force fail2ban to block access to
addresses which are not attempting to connect to the system. For each IP address available to the attacker, a desired ip
address may be blocked."

5. Customising the Config file

From reading more about iptables in the Linux Firewalls book (very informative but tough reading) and the online resources listed
below it appears that some of the iptables commands contain options that are redundant:
 ● the RETURN command in fwstart is not necessary as a user-defined chain will automatically return to the parent chain after
processing all commands (unless another JUMP is encountered). The only advantage I see in including the RETURN command is
that iptables with the -v option can then show you both the number of packets/bytes sent to and returned from fail2ban-ssh -
the difference telling you how many were dropped.
● the rule number 1 in fwban is also redundant as commands will by default be INSERTed at the top of the chain.

The fwstart command then becomes just:

fwstart = iptables -N fail2ban-ssh

iptables -I INPUT -p tcp --dport ssh -j fail2ban-ssh

It's also possible to use the LOG feature in iptables rather than relying on the fail2ban program. This gives the advantage of recording
the details of packets that are DROPped while a ban rule is in place.

This involves a change to the fwban command:

fwban = iptables -A fail2ban-ssh -s <ip> -j LOG --log-prefix "Fail2Ban: " --log-ip-options --log-
tcp-sequence --log-tcp-options
iptables -A fail2ban-ssh -s <ip> -j DROP

Note: with the RETURN command removed we can use APPEND instead of INSERT here.

Translation:

● add a DROP command for <ip> at the start of the chain;

● add a LOG command with all options enabled at the start of the chain (before the DROP command).

and to fwunban:

fwunban = iptables -D fail2ban-ssh -s <ip> -j DROP

iptables -D fail2ban-ssh -s <ip> -j LOG --log-prefix "Fail2Ban: " --log-ip-options --log-
tcp-sequence --log-tcp-options

Note: DROP commands must exactly match the INSERT command syntax (or specify a rule number). The order isn't really important but you'll notice that
we add the LOG command first and remove it last.

Now your logfiles will record all details of the dropped packages - typically in kern.log or syslog depending on your server
configuration. Be aware that this can cause those files to grow quite quickly so only add the LOG commands if you're going to use the
data collected.

6. Does it work?

Does it ever!?!

Here you can see the results for a single server - you can tell when fail2ban was installed and why it was necessary:

$ zcat /var/log/auth.log* | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort | uniq -
c

408 Dec 11
77 Dec 12
12153 Dec 13
9390 Dec 14
1033 Dec 18
3743 Dec 19
4167 Dec 20
2789 Dec 25
9200 Dec 26
15742 Dec 27
281 Dec 28
7 Dec 29
17 Dec 30
3 Dec 31
4 Jan 1
5 Jan 2
4 Jan 4
9 Jan 5
11 Jan 6
9 Jan 7

There's word of some big changes in the fail2ban package including 'better configuration files' and 'templates for common services'
among other things but it's not clear when that might make it into package form.

Note: if you just want to rate-limit SSH connections without installing new software then you can use the recent module but be aware that this can use a lot
of system resources on busy networks as it has to store state information for all connections.
7. Related Articles

● fail2ban and iptables

● fail2ban and sendmail
● Monitoring the fail2ban log

8. References

● Fail2Ban
● iptables Tutorial
● Manpage of IPTABLES
● Linux Firewalls, Third Edition - ISBN: 0672327716
● Python: Regular Expression HOWTO
System: fail2ban and sendmail
Following on from the article on fail2ban and iptables this article looks at changing the configuration file so that we can monitor the
mail.log file and take action against suspect connections over smtp (port 25).

Again, some of the files/methods described here may be specific to Debian (Sarge).

1. Changes to fail2ban.conf

To start with we need to create (and enable) a new section in /etc/fail2ban.conf giving instructions on where to find the logfile and
what do look for:

[SMTP]
enabled = true

logfile = /var/log/mail/mail.log

When the daemon starts we need to go through a similar process as for blocking SSH break-in attempts. The fwstart commands
create a new chain called fail2ban-mail and add a JUMP command to the INPUT chain. The fwend commands have the effect of
removing both the JUMP command and the new chain:

The fwcheck command simply lets fail2ban know how to check whether the fail2ban-mail chain exists.

fwstart = iptables -N fail2ban-mail

iptables -A INPUT -p tcp --dport 25 -j fail2ban-mail

fwend = iptables -D INPUT -p tcp --dport 25 -j fail2ban-mail

iptables -F fail2ban-mail
iptables -X fail2ban-mail

fwcheck = iptables -L INPUT | grep -q fail2ban-mail

Now to the 'active' components. When a match is found for the failregex expression (see below). The fwban command adds a rule to
the fail2ban-mail chain instructing iptables to REJECT all packets from the relevant host. The fwunban command removes this rule.
The timeregex and timepattern are the same as for auth.log:

fwban = iptables -A fail2ban-mail -p tcp -s <ip> -j REJECT --reject-with tcp-reset

fwunban = iptables -D fail2ban-mail -p tcp -s <ip> -j REJECT --reject-with tcp-reset

timeregex = \S{3}\s{1,2}\d{1,2} \d{2}:\d{2}:\d{2}

timepattern = %%b %%d %%H:%%M:%%S

Finally we come to the regular expression. There are a number of mail.log entries that you might want to monitor but it's tricky to
cover them all with a single regexp. The regexp presented here:

failregex = [[](?P<host>\S*)[]] (?:did not issue|[(]may be forged[)])

will match lines in the mail log matching either of the following:

... [<host>] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA ...
... [<host>] (may be forged) ...

All you have to do now is stop/start the daemon as described in the previous article.

2. Changes for fail2ban 0.8

After a long wait, fail2ban now supports the use of multiple regular expressions in a single rule, making all our lives that much easier.
There is also a whole new configuration system which I won't go into here as it's already quite well documented.

Here is a sample failregex that you might want to use for the sendmail filter:

failregex = \[<HOST>\] .*to MTA

\[<HOST>\] \(may be forged\)
\[<HOST>\], reject.*\.\.\. Relaying denied

Note the use of the new <HOST> predefined entity, which matches either a hostname or an IPv4 address.

These regular expressions will match and block any/all of the following:
 ... lost input channel from [<HOST>] to MTA after data ...
... [<HOST>] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA ...
... [<HOST>] (may be forged) ...
... [<HOST>], reject.*... Relaying denied ...

There are also some helpful command-line tools for testing your regular expressions:

# fail2ban-regex /var/log/mail.log "\[<HOST>\], reject.*\.\.\. Relaying denied"

You can start/stop individual jails while fail2ban is still running:

# fail2ban-client reload sendmail

And you can query most aspects of the configuration:

# fail2ban-client get sendmail failregex

The following regular expression are defined:
|- [0]: \[(?:::f{4,6}:)?(?P<host>[\w\-.^_]+)\] .*to MTA
|- [1]: \[(?:::f{4,6}:)?(?P<host>[\w\-.^_]+)\] \(may be forged\)
`- [2]: \[(?:::f{4,6}:)?(?P<host>[\w\-.^_]+)\], reject.*\.\.\. Relaying denied

3. Related Articles

● fail2ban and iptables

● fail2ban and sendmail
● Monitoring the fail2ban log
System: Monitoring the fail2ban log
Following on from the article on fail2ban and iptables this article looks at the fail2ban logfile and ways to analyse it using simple
command-line tools such as awk and grep.

1. Format of the Logfile

At the simplest logging level, entries will appear in /var/log/fail2ban.log as follows (fail2ban version 0.8.3):

...
2006-02-13 15:52:30,388 fail2ban.actions: WARNING [sendmail] Ban XXX.66.82.116
2006-02-13 15:59:29,295 fail2ban.actions: WARNING [sendmail] Ban XXX.27.118.100
2006-02-13 16:07:31,183 fail2ban.actions: WARNING [sendmail] Unban XXX.66.82.116
2006-02-13 16:14:29,530 fail2ban.actions: WARNING [sendmail] Unban XXX.27.118.100
2006-02-13 16:56:27,086 fail2ban.actions: WARNING [ssh] Ban XXX.136.60.164
2006-02-13 17:11:27,833 fail2ban.actions: WARNING [ssh] Unban XXX.136.60.164

This is all very interesting, but what if you want to see a summary report so that you can try to identify IP addresses that regularly
trigger Fail2Ban - so that you can send a report to their ISP or block them using a firewall script for example?

2. Generating Simple Reports

All of the following commands can be run at the command-line or via a script. They are written for Linux/UNIX systems but may work
on other platforms.

Grouping by IP address:

awk '($(NF-1) = /Ban/){print $NF}' /var/log/fail2ban.log | sort | uniq -c | sort

Note: the variable NF equals the number of fields in each row of the logfile. So $NF is the value of the last field.

Sample output:

...
4 XXX.124.81.130
5 XXX.248.175.246
8 XXX.29.45.142

Remember that each time an IP address gets banned it's because they've been caught at least maxfailure times, so a total of 8
represents maybe 30 matches in the relevant logfile. Once they reach 10-20 you might consider them as candidates for reporting, or a
more permanent solution (see below).

To run this report for all logfiles only a slight change is needed:

zgrep -h "Ban " /var/log/fail2ban.log* | awk '{print $NF}' | sort | uniq -c

or, even better, we can truncate the IP addresses to identify the most problematic subnets:

zgrep -h "Ban " /var/log/fail2ban.log* | awk '{print $NF}' | awk -F\. '{print $1"."$2"."}' | sort |
uniq -c | sort | tail

This is the best report for identifying problem subnets. The output will be the first two bytes of the most 'caught' subnets:

...
75 83.110.
90 219.95.
154 210.213.

Let's take the last one on the list (highlighted) and see what it's been up to:

zgrep -c 210.213. /var/log/fail2ban.log*

The output shows how many times those numbers appear in each logfile:

fail2ban.log:39
fail2ban.log.1.gz:129
fail2ban.log.2.gz:55
fail2ban.log.3.gz:78
fail2ban.log.4.gz:22
and which specific IP addresses are involved:

zcat /var/log/fail2ban.log* | awk '(NF == 6 && $NF ~ /^210\.213\./){print $NF}' | sort | uniq -c

The output of this will be a list of the IP addresses starting with 210.213. If they look like they're part of a subnet (or multiple
subnets) you can copy the lowest and highest numbers in our Subnet Calculator to give you the subnet code which you can then add
to your firewall rules (see below for details).

Grouping by IP address and Hostname:

The command for including hostnames in the list is a bit more complicated. You also need to insert the correct path for the logresolve
program which converts IP addresses to hostnames (the path may be something like /usr/sbin/logresolve but it varies between
systems):

awk '($(NF-1) = /Ban/){print $NF,"("$NF")"}' /var/log/fail2ban.log | sort | logresolve | uniq -c |

sort

Note: The logresolve command can take some time, especially if there are a lot of IP addresses to be processed.

The output is similar to what we've seen previously, but also includes the hostname which makes it easier to identify the ISP and/or
country of origin and to see which entries might be related:

...
4 XXX.net.pk (XXX.83.169.221)
5 XXX.248.175.246 (XXX.248.175.246)
8 XXX.example.com.au (XXX.29.45.142)

You can of course just run dig, nslookup or logresolve manually on the addresses that you want to identify.

Group by IP address and Fail2Ban section:

grep "Ban " /var/log/fail2ban.log | awk -F[\ \:] '{print $10,$8}' | sort | uniq -c | sort

This shows us which services each IP address has been trying to access/exploit:

...
4 XXX.124.81.130 [sendmail]
5 XXX.248.175.246 [sendmail]
8 XXX.29.45.142 [sendmail]

Now you know which logfiles to look in to see what they were doing to get banned. In this case it's most likely passing forged mail
headers to sendmail which you can see in /var/log/mail/mail.log (or the relevant file on your system).

Reporting on 'today's activity:

Here's a report I find useful to run before midnight each day to generate a summary of the day's activity:

grep "Ban " /var/log/fail2ban.log | grep `date +%Y-%m-%d` | awk '{print $NF}' | sort | awk '{print
$1,"("$1")"}' | logresolve | uniq -c | sort

The output will be the same as the second report above, but limited to just today's activity rather than the whole logfile.

Grouping by Date and Fail2Ban section

This report scans all fail2ban logfiles and gives you a summary of how many ban events there were for each section on each day:

zgrep -h "Ban " /var/log/fail2ban.log* | awk '{print $5,$1}' | sort | uniq -c

This can give you an idea of longer-term trends and the effectiveness of your firewall rules. This method of examining all logfiles
rather than just the current one can also be applied to most of the reports above.

3. Banning an IP block or subnet

If it turns out that a significant portion of 'unwanted' traffic comes from a single ISP then you should try sending an email to their
abuse address, but don't be too hopeful of getting a response. If the abuse continues then it's time to get strict.

First have a look at the different IP addresses that are being caught. See if you can identify which ones come from the same subnet.
The whois reports often include this information, otherwise you can use our Subnet Calculator to help you along - just paste the
lowest and highest addresses into the form and it will give you the smallest subnet that covers them both.
Once you have this value (in the form XXX.XXX.XXX.XXX/XX) you can add a firewall rule using iptables to block them from the server
completely, or just from the port they're abusing. For a single address you don't need to worry about subnets and the address can be
used directly.

Block a subnet from accessing SSH:

iptables -I INPUT -p tcp -s XXX.XXX.XXX.XXX/XX --dport ssh -j REJECT --reject-with tcp-reset

Block a subnet from accessing SMTP:

iptables -I INPUT -p tcp -s XXX.XXX.XXX.XXX/XX --dport smtp -j REJECT --reject-with tcp-reset

and so on for any other services. These rules will be added to the start of your firewall or you can include a rule number to decide
where they appear. As always the DROP command is identical, just with -D in place of -I.

To see what effect these rules are having - the number of packets and bytes being blocked by each rule - use the following command
and look at the values in the first two columns.

iptables -vnL INPUT --line-numbers

At some point (hopefully) the source computer will be 'fixed' or in any case stop abusing your server. You should then remove the
firewall rules.

4. Monitoring the fail2ban log with fail2ban 0.8

This is something I've been meaning to investigate for some time now, and there have been a number of request for this ability. Can
we use fail2ban to block for a longer time (even permanently) addresses after they hav been blocked a number of times by the normal
fail2ban filter.

It seems that it is possible, though you may have to set up different jails for different ports. For example, for repeat offenders
according to the sendmail filter, add the following to /etc/fail2ban.jail.local:

[fail2ban-smtp]

enabled = true
port = smtp
filter = fail2ban-smtp
logpath = /var/log/fail2ban.log
maxretry = 3
findtime = 21600
bantime = 86400

And then create a file /etc/fail2ban/filter.d/fail2ban-smtp.conf with the following:

failregex = \[sendmail\] Ban <HOST>

ignoreregex =

Finally start the new jail:

# fail2ban-client add fail2ban-smtp

# fail2ban-client start fail2ban-smtp

With these settings, fail2ban will monitor it's own logfile and if a HOST is banned three times (maxretry) in six hours (findtime)
they will incur a new ban lasting a full 24 hours (bantime). If the bantime value is negative then the HOST in question will never be
unbanned.

Similar rules can be set up for other existing jails, and they can be combined if they share the same port.

Please note: this jail is still experimental so use it with care. Please let s know though the Feedback link below if you have any
questions or comments about using it on your server.

5. Related Articles

● fail2ban and iptables

● fail2ban and sendmail
● Monitoring the fail2ban log