So, I set up OpenSMTPD, spampd and Dovecot up on a Debian system and created a simple mail server. This is good, but I thought I could do better. OpenSMTPD has a (slightly) experimental API for hooking separate filter programs into the mail transaction (similar to using other programs to authenticate remote users, such as table-passwd from my previous post) which can perform actions at different points in the transaction and then tell OpenSMTPD to either accept or reject the mail. Seeing as I'm a complete n00b of a mail admin, I thought "hey, a not-quite-stable way to ~~rice up~~ enhance my OpenSMTPD? Sure, let's give it a shot!". I have two filters in use on my OpenSMTPD installation and both of them implement known spam mitigation techniques; I'll explain how they work (for the benefit of the uninitiated) and then show how they're implemented in my configuration.
Follow the instructions below at your own risk! I do not claim that this is a good idea nor can I ensure that your mail server won't get up and chase your cat as a result of following the instructions!
Some not-very-smart spam systems will simply send their message as quickly as possible as soon as they can open a connection without waiting for the server to send their first line. This is obviously rather rude and is a violation of the protocol; well behaved clients should wait for the server to send its first line before sending any lines of their own, and thereafter they should only send their following lines after the server has acknowledged their previous lines. The solution is to wait a couple of seconds after the connection is opened before sending the "220" banner. The spam systems won't wait for the banner and will send their entire message all at once, which will result in their connection getting closed by the smtpd. Well behaved clients will wait through the pause until the mail server sends the initial "220" line, after which the SMTP dialogue can continue as normal.
Greylisting is a very simple but very effective spam mitigation technique. The logic is that senders of legitimate mail will probably send their messages on a best effort basis: if they get a temporary error from the remote sender, they'll try again a few times before deciding that the mail is undeliverable. Spammers, on the other hand, want to send as many messages to as many recipients as possible as quickly as possible; the message is either accepted or rejected by the remote server and if it is rejected with a temporary failure message they'll move on to other victims. Greylisting, then, works as follows: for any incoming mail, if the sender has not sent any mail to us before, we reject the mail with a temporary failure notice and record the time that this sender was last seen (the sender has been "greylisted"). After a delay has elapsed, if the sender retries sending the mail, it will be accepted (i.e. they are now automatically whitelisted). From this point onwards, any future emails from that sender will not be delayed so long as their entry on the "auto-whitelist" does not expire. A simple spam system will attempt to send a message, but won't try to resend it if it's rejected; therefore the spam email is never delivered in the first place.
(If you don't quite understand the explanation of greylisting above, search around for it on the internet; plenty has been written about it.)
Note: I'll describe each of the filters separately before I describe how to use them together.
filter-pauseOpenSMTPD does not ship with any plugins, however there is an extras package (OpenSMTPD-extras, of course) which contains several compatible plugins to complement and extend OpenSMTPD's functionality, including a filter which implements the "pause" behaviour described above. (Note that Debian splits this source package up into two binary packages, opensmtpd-extras and opensmtpd-extras-experimental.) However, the version which ships with Debian Stretch (and Debian Testing!) is older and the filters which come with that version are not compatible with the version of OpenSMTPD in Debian's repositories.
Therefore, it is necessary to build a more recent version of opensmtpd-extras. Either download the 5th of July 2016 snapshot of opensmtpd-extras from here (or clone the upstream git repository and checkout the filter-pause branch) and configure and compile it with the same options as the Debian package. You should then copy the resulting filter-pause binary to the appropriate location (e.g. /usr/lib/$ARCH-linux-gnu/opensmtpd/). If you have the opensmtpd-extras-experimental package installed, I would advise that you rename the binary and not to overwrite the Debian provided filter-pause binary.
You can then configure the pause filter by adding the following line to your /etc/smtpd.conf configuration file:
filter pause pause
By default, the filter will cause a delay of five seconds between accepting a connection and displaying the initial banner, but this can be configured by passing an argument to the filter. So, to configure the filter to only wait for three seconds, use the following:
filter pause pause "-s" "3"
Then, tell OpenSMTPD which connections it should apply the filter to:
listen on eth0 port 25 tls pki mail.example.com filter pause
Now, if you run tail -f on you mail logs while an incoming mail is being received, you should see that there is a delay in smtpd printing the line where it gets the connection and where it starts handling the connection.
filter-postgreyThis filter doesn't do the greylisting itself, instead it talks to a greylisting daemon designed for use with postfix: postgrey. I wrote this filter based on filter-greylistd found here and my reasons for rolling my own solution instead of using the existing one can be found on the project page. For the impatient, postgrey is a little smarter than greylistd, especially when it comes to configuring whitelists.
Clone the repository for filter-postgrey and follow the build instructions. Copy the resulting binary to /usr/lib/$ARCH-linux-gnu/opensmtpd. Next, install postgrey. Note that if apt is configured to install recommended packages as well as hard dependencies (which is the default) then postfix may be pulled in when you install postgrey if you don't instruct it otherwise. You must also edit the postgrey configuration in /etc/default/postgrey and make postgrey listen on a Unix domain socket and not a network socket. It may also be desirable to configure postgrey with different timeout values e.g. adding --delay=600 to the postgrey options to greylist hosts for ten minutes instead of five.
The following configuration lines can now be added to /etc/smtpd.conf, as above:
filter postgrey postgrey "-s" "/path/to/postgrey/socket"
<snip>
listen on eth0 port 25 tls pki mail.example.com filter postgrey
Once again, send some test mails. You should see that they are rejected with temporary rejection messages and that once the greylisting period has expired (five minutes by default) then they are accepted. Note that if there are misconfigurations present then postgrey, filter-postgrey and smtpd will all complain loudly about it in your mail logs.
OpenSMTPD can combine individual filters together so that they can be tested in sequence. In this case we want to apply the pause filter and then the greylisting filter. The corresponding configuration file lines are:
filter pause pause
filter postgrey postgrey "-s" "/path/to/postgrey/socket"
filter antispam chain pause postgrey
<snip>
listen on eth0 port 25 tls pki mail.example.com filter antispam
And that's very close to what I use in my production configuration.