snork.ca: Welcome to the brown age of computing! Mainly Sunny, 21.2°C [H: 27°C] - No precip

hMailServer + Rsyslog + Fail2ban 2021-12-06

I recently had to redo this when I moved to a new hMailServer box and bumped in to a couple of annoying things, so I updated the instructions to include fixes for the tabs showing up as #011 and enabling remote syslog. I should also note that currently I am not bothering with fail2ban and am just letting hMailServer deal with whatever increased traffic it sees as a result.

hMailServer + Rsyslog + Fail2ban 2020-08-20

To [possibly] save you a lot of trouble and reading, I should mention right up front that this involves running NXLog on my Windows/hMailServer which monitors the flat-text log files and passes them via Rsyslog to my Linux based router. The router machine receives the logs and saves them to its own flat-text file, which Fail2ban monitors and bans "bad" IP addresses with iptables/ipset rules. Your network may not have a Linux box between your Windows based server and the outside world, but it really shouldn't be that hard to sneak a Linux machine in between, or modify this for use with Linux/BSD based embedded routers.

For some years now I have been using hMailServer as my IMAP, POP3, and outbound SMTP software of choice. However, my inbound SMTP has mostly been handled by a Linux machine with postfix, postgrey, and spamassassin. This is because Windows has no firewall that can compare to iptables & ipset and no automated log monitor/ban tool like fail2ban. At the same time, Linux has no mail server software that can compare to hMailServer. On Linux I use Fail2ban to monitor my log files and immediately ban IP addresses that do shitty things to my mail server. I have tried to [loosely] duplicate this powerful traffic management strategy on Windows, but have never come close. Until just recently that is.

truckstop Stops spammers this fast.

NXLog is a pretty light application, and the community edition is open source. It has no GUI, installs itself as a service, and is configured via a text file. According to Task Manager on my Windows box it uses a cool 3,176k of memory. The default config file does not actually give any sample input/output streams but the documentation is pretty comprehensive and adding an input/output stream isn't very hard. In my case I put something like this in my config:

<Input hmslogs>
  Module im_file
  File 'C:\Apps\hMailServer\Logs\hmailserver_*.log'
  SavePos TRUE
  ReadFromLast TRUE
  PollInterval 1
  <Exec>
    $raw_event = replace($raw_event, "\t", " ");
# Exclude logs for webmail
    if $raw_event =~ /192\.168\.1\.101 connected to 192\.168\.1\.10:993\.\"$/ drop();
# Exclude logs for outbound connections
    if $raw_event =~ /^\"POP3C/ drop();
    if $raw_event =~ /^\"SMTPC/ drop();
    else $Message = $raw_event; $SyslogFacilityValue = 23;
  </Exec>
</Input>

<Output routerlogs>
  Module om_udp
  Host 192.168.1.1
  Port 514
  Exec to_syslog_bsd();
</Output>

<Route 1>
  Path hmslogs => routerlogs
</Route>

Basically the "Input" section defines the flat-text logs that it will read, the "output" section defines the syslog server it will send messages to, and the "Route" section simply ties them together. Obviously you could have numerous input, output, and route sections tied together any way you like. The "if $raw_event" lines are used to ignore certain log lines from being sent to the remote server. In my case I do not want to ban my webmail server or any outbound connections so I don't bother to send them.

Now, in order for my router to "receive" these messages I had to configure Rsyslog to accept them and save them to a file somewhere. To do that I edited my /etc/rsyslog.conf and added something like this:

local7.notice /var/log/hmail.log

I also had to uncomment the lines to listen for UDP based syslog from remote machines near the top of the /etc/rsyslog.conf like this:

module(load="imudp")
input(type="imudp" port="514")

After restarting Rsyslog the hmail.log starts getting populated with the entries that are sent to it via Rsyslog. It is important to note that the value "$SyslogFacilityValue" in your nxlog.conf (23 in my case) has to match the "local7.notice" value in your rsyslog.conf file. I remember using Kiwi syslog utilities to figure out a matching set, I assume there is an easier way to do that, and I suspect there is a mapping someone has made available somewhere. If you want to use something other than local7.notice then you may have to do some hunting. Anyways, I ran a tail -F /var/log/hmail.log and got this kind of stuff:

Aug 20 17:02:35 mail "TCPIP"#0112824#011"2020-08-20 17:02:34.406"#011"TCP - 54.240.13.35 connected to 192.168.1.101:25."
Aug 20 17:02:35 mail "SMTPD"#0112824#0119493#011"2020-08-20 17:02:34.406"#011"54.240.13.35"#011"SENT: 220 Dude, Wassup!"
Aug 20 17:02:35 mail "SMTPD"#0112096#0119493#011"2020-08-20 17:02:34.453"#011"54.240.13.35"#011"RECEIVED: EHLO a13-35.smtp-out.amazonses.com"
Aug 20 17:02:35 mail "SMTPD"#0112096#0119493#011"2020-08-20 17:02:34.453"#011"54.240.13.35"#011"SENT: 250-mail.snork.ca[nl]250-SIZE 40960000[nl]250 HELP"
Aug 20 17:02:35 mail "SMTPD"#0114152#0119493#011"2020-08-20 17:02:34.500"#011"54.240.13.35"#011"RECEIVED: MAIL FROM:<thing@bounces.amazon.ca>"
Aug 20 17:02:35 mail "TCPIP"#0114152#011"2020-08-20 17:02:34.718"#011"DNS lookup: 35.13.240.54.hostkarma.junkemailfilter.com, 2 addresses found: 127.0.0.3, 127.0.1.1, Match: False"
Aug 20 17:02:37 mail "TCPIP"#0114152#011"2020-08-20 17:02:37.370"#011"DNS lookup: 35.13.240.54.dnsbl.justspam.org, 0 addresses found: (none), Match: False"
Aug 20 17:02:43 mail "TCPIP"#0114152#011"2020-08-20 17:02:42.674"#011"DNS lookup: 35.13.240.54.zz.countries.nerd.dk, 1 addresses found: 127.0.3.72, Match: False"
Aug 20 17:02:43 mail "SMTPD"#0114152#0119493#011"2020-08-20 17:02:42.721"#011"54.240.13.35"#011"SENT: 250 OK"
Aug 20 17:02:43 mail "SMTPD"#0112824#0119493#011"2020-08-20 17:02:42.768"#011"54.240.13.35"#011"RECEIVED: RCPT TO:<doesnotexist@snork.ca>"
Aug 20 17:02:48 mail "TCPIP"#0112824#011"2020-08-20 17:02:47.089"#011"DNS MX lookup: bounces.amazon.ca"
Aug 20 17:02:51 mail "TCPIP"#0112824#011"2020-08-20 17:02:50.427"#011"DNS - MX Result: 1 IP addresses were found."
Aug 20 17:02:51 mail "SMTPD"#0112824#0119493#011"2020-08-20 17:02:50.459"#011"54.240.13.35"#011"SENT: 250 OK"
Aug 20 17:02:51 mail "SMTPD"#0112096#0119493#011"2020-08-20 17:02:50.505"#011"54.240.13.35"#011"RECEIVED: DATA"
Aug 20 17:02:51 mail "SMTPD"#0112096#0119493#011"2020-08-20 17:02:50.505"#011"54.240.13.35"#011"SENT: 354 OK, send."
Aug 20 17:02:52 mail "TCPIP"#0112700#011"2020-08-20 17:02:51.535"#011"Connecting to 127.0.0.1:783..."
Aug 20 17:03:06 mail "SMTPD"#0112700#0119493#011"2020-08-20 17:03:06.074"#011"54.240.13.35"#011"SENT: 250 Queued (15.232 seconds)"
Aug 20 17:03:07 mail "APPLICATION"#0112744#011"2020-08-20 17:03:06.183"#011"SMTPDeliverer - Message 81871: Delivering message from thing@bounces.amazon.ca to doesnotexist@snork.ca. File: C:\Path\{1234-ABCD}.eml"
Aug 20 17:03:07 mail "APPLICATION"#0112744#011"2020-08-20 17:03:06.605"#011"SMTPDeliverer - Message 81871: Message delivery thread completed."
Aug 20 17:03:28 mail "SMTPD"#0112096#0119493#011"2020-08-20 17:03:28.086"#011"54.240.13.35"#011"RECEIVED: QUIT"
Aug 20 17:03:28 mail "SMTPD"#0112096#0119493#011"2020-08-20 17:03:28.086"#011"54.240.13.35"#011"SENT: 221 goodbye"

Notice the tabs were sent as "#011". If you use the "replace" line as pasted in the nxlog config example abaove those will be converted to spaces before sending to the syslog server. Now... Fail2ban is kind of odd in that you need to configure a "jail" file which defines your filters and actions, then filter configs which look for lines in your log file, then an action config which actally does something when matches are found. So a quick /etc/fail2ban/jail.local definition might look like this:

[fu_hmail]
enabled = true
logpath = /var/log/hmail.log
maxretry = 1
bantime = 86400
filter = fu_hmail
banaction = fu_hmail

Which obviously tells it which log file to monitor, how long to ban for, and which filters and actions to use. So my accompanying /etc/fail2ban/filter.d/fu_hmail.conf filter file looks like this:

[Definition]
failregex = ^.* \"SMTPD\"#011\d*#011\d*#011\".*\"#011\"<HOST>\"#011\"RECEIVED: EHLO .*\.us\"$
  ^.* \"SMTPD\"#011\d*#011\d*#011\".*\"#011\"<HOST>\"#011\"RECEIVED: EHLO .*clickdimensions\.com\"$
  ^.* \"SMTPD\"#011\d*#011\d*#011\".*\"#011\"<HOST>\"#011\"RECEIVED: EHLO adams\.edu\"$

  ^.* \"SMTPD\"#011\d*#011\d*#011\".*\"#011\"<HOST>\"#011\"RECEIVED: Mail from:<bounce@aweber\.com>\"$
  ^.* \"SMTPD\"#011\d*#011\d*#011\".*\"#011\"<HOST>\"#011\"RECEIVED: MAIL FROM:<goof@gmail\.com>.*\"$
  ^.* \"SMTPD\"#011\d*#011\d*#011\".*\"#011\"<HOST>\"#011\"RECEIVED: Mail from:<spameri@tiscali\.it>\"$

  ^.* \"SMTPD\"#011\d*#011\d*#011\".*\"#011\"<HOST>\"#011\"SENT: 503 Bad sequence of commands\"$
  ^.* \"SMTPD\"#011\d*#011\d*#011\".*\"#011\"<HOST>\"#011\"SENT: 504 Authentication not enabled\.\"$

ignoreregex = ""

The first few lines would be used to block known crappy EHLO's, the second set are some known crappy sender addresses, and the last couple of lines are for machines that are clearly not playing by the rules when speaking to my server. Obviously you can write regular expression rules for whatever you want to find and dump on your own server. I realize that these examples are pretty general and could potentially ban a legitimate mail server, so maybe if you are just starting out you could make your ban time pretty low, and monitor for bans to ensure you're not banning good servers.

Now the last piece of Fail2ban is of course the /etc/fail2ban/action.d/fu_hmail.conf file which will "do something" when it sees those regexp's in the defined log file. Mine does this:

[Definition]
actionstart =
actionstop =
actioncheck =

actionban = ipset add hmailblock <ip>
  logger -p local7.notice -t fail2ban "<ip> BANNED!"

actionunban = ipset del hmailblock <ip>
  logger -p local7.notice -t fail2ban "<ip> BAN LIFTED!"

This will obviously add the offending IP address to an ipset called "hmailblock" on the router machine, and will also add a message to the mail logs saying that the IP has been banned. Clearly later (after about 86400 seconds) it will do the opposite, and log it. The last piece of this puzzle is to setup the firewall to block the addresses in that ipset. A couple of lines like these should do:

ipset create hmailblock hash:net -exist
iptables -A FORWARD -m set --match-set hmailblock src -j DROP

There is a lot of leeway in these instructions, and the information could [hopefully] be useful in many different network configurations and for many different desired reasons. If you have used a setup like this for something different I would love to hear about it and would add your information here if you wished to share your setup with others.

Made with Notepad++ & FastStone, without javascript, cookies, or the help of Clippy or ai. Hosted on Devuan with nginx & powered by NK shrooms.