SURBL filtering configuration

May 24, 2019 Roberto Puzzanghera 16 comments

SURBLs are lists of web sites that have appeared in unsolicited messages. Unlike most lists, SURBLs are not lists of message senders.

Web sites seen in unsolicited messages tend to be more stable than the rapidly changing botnet IP addresses used to send the vast majority of them. Sender lists like zen.spamhaus.org can be used in a first stage filter to help identify 80% to 90% of unsolicited messages. SURBLs can help find about 75% of the otherwise difficult, remaining unsolicited messages in a second stage filter. Used together with sender lists, SURBLs have proven to be a highly-effective way to detect 95% of unsolicited messages.

The SURBL filter is part of the DKIM patch by Manvendra Bhangui and it's embedded in my combined patch.

To enable this filter you must export the variable SURBL with any value in your run file and pass the filter program to the QMAILQUEUE variable so that it can be executed before the delivery:

export SURBL=1
export QMAILQUEUE=/var/qmail/bin/surblqueue
export SURBLQUEUE=/var/qmail/bin/simscan

Actually the program is wrapped by surblqueue, as you can see. SURBLQUEUE will make the program to execute simscan when finished. If you don't define SURBLQUEUE the program executes qmail-queue to do the delivery.

NB: Remember to remove QMAILQUEUE from your tcp.smtp, otherwise it will overwrite your run file.

Be aware that the directory /var/qmail/control/cache must have the write priviledges for the user who runs qmail-smtpd, vpopmail in our case. I have adjusted my combined patch accordingly.

surblfilter requires two control files level2-tlds and level3-tlds in /var/qmail/control. The same can be obtained from surbl.org website http://www.surbl.org/tld/three-level-tlds http://www.surbl.org/tld/two-level-tlds. These files sholud not be confused with the SURBL lists themselves but it is worth to update them monthly or so on building a cronjob like this:

cat > /usr/local/bin/update_tlds.sh << __EOF__
#!/bin/bash
#

cd /var/qmail/control
/usr/bin/wget http://www.surbl.org/tld/three-level-tlds http://www.surbl.org/tld/two-level-tlds
mv two-level-tlds level2-tlds
mv three-level-tlds level3-tlds
__EOF__
chmod +x /usr/local/bin/update_tlds.sh

Now setup the cronjobs (the second one cleans the cache folder once a day):

# surbl tlds update
2 2 23 * * /usr/local/bin/update_tlds.sh 1> /dev/null
# surbl cache purge
2 9 * * * find /var/qmail/control/cache/* -cmin +5 -exec /bin/rm -f {} \;

Executing surblfilter, qmail-dkim and simscan in cascade

Using a combination of QMAILQUEUE, SURBLQUEUE and DKIMQUEUE will make you run both filters and finally pass the message to simscan, which in turn calls clamd, spamd and finally executes qmail-queue:

export SURBL=1  # Comment to disable SURBL filtering
export QMAILQUEUE=/var/qmail/bin/surblqueue # executes surblfilter
export SURBLQUEUE=/var/qmail/bin/qmail-dkim # executes qmail-dkim afer surblfilter
export DKIMQUEUE=/var/qmail/bin/simscan # simscan is executed after qmail-dkim
export DKIMKEY=/usr/local/etc/domainkeys/%/default
# DKIM verification. Use carefully
export DKIMVERIFY="FGHKLMNOQRTVWp"
# This is to allow msg without "subject" in the h= list
export UNSIGNED_SUBJECT=1
# This is to avoid verification of outgoing messages
export RELAYCLIENT_NODKIMVERIFY=1

Testing

Send yourself an email with an URL such as http://surbl-org-permanent-test-point.com/ in the body. You should see the filter in action in your qmail-smtpd log:

qmail-smtpd: message rejected (message contains an URL listed in SURBL blocklist): user@domain.xy from 123.45.67.89 to yourself@yourdomain.xy helo yourmailserver.xy

Comments

Error compiling surblfilter

Hi

I'm trying to apply the patch on FreeBsd, but after solving many errors I was stuck with this

./load surblfilter envread.o strerr_die.o strerr_sys.o control.o alloc.o alloc_re.o error.o error_str.o auto_qmail.o case_startb.o byte_diff.o str_cspn.o byte_copy.o byte_chr.o byte_rchr.o byte_cr.o getln.o getln2.o open_read.o str_len.o str_diffn.o str_cpy.o str_chr.o scan_xlong.o now.o scan_ulong.o mess822_ok.o constmap.o ip.o dns.o ipalloc.o fmt_str.o fmt_ulong.o socket_v6any.o socket_v4mappedprefix.o sgetopt.o subgetopt.o base64sub.o case_diffb.o stralloc.a substdio.a -lresolv

/usr/bin/ld: cannot find -lresolv

cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1

After reading & reading I finally understund that FreeBSD doesn't have "resolv", so, the solution was to remove "-lresolv" from patch

I'm not using "all" the patch that you made, because I'm using ports on freebsd, but I use your guide to configure a lot of things

Thanks for your work

Reply | Permalink

Error compiling surblfilter

Thanks for the contribution. As many others here reported compilation breaks with freeBSD, it would be nice if you share more informations about the fixes you made...

Reply | Permalink

SURBL not working..

Hi Roberto!

i using your howto to build a new mailserver and SURBL seemed very cool.

I do all , but, not happend.. I send messages with urls from list but all have passed.

I not implement DKIM, this is problem?

my qmail-smtpd/run is:

QMAILDUID=`id -u vpopmail`
NOFILESGID=`id -g vpopmail`
MAXSMTPD=`cat /var/qmail/control/concurrencyincoming`
SOFTLIMIT=`cat /var/qmail/control/softlimit`

export SMTPD_GREETDELAY=15
export DROP_PRE_GREET=1

export CHKUSER_START=ALWAYS

export SURBL=1 # Comment out to enable SURBL filtering
export QMAILQUEUE=/var/qmail/bin/surblqueue # executes surblfilter
#export SURBLQUEUE=/var/qmail/bin/qmail-dkim # executes qmail-dkim after sublfilter <== i try enable this, but not happen
export DKIMQUEUE=/var/qmail/bin/simscan     # simscan is executed after qmail-dkim

exec /usr/local/bin/softlimit -m "$SOFTLIMIT" \
    /usr/local/bin/tcpserver -v -H -R -l 0 \
    -x /home/vpopmail/etc/tcp.smtp.cdb -c "$MAXSMTPD" \
    -u "$QMAILDUID" -g "$NOFILESGID" 0 25 \
    /var/qmail/bin/qmail-smtpd 2>&1

Please, send one light! :-)

[] s

Tiago Oliveira de Jesus

Reply | Permalink

did you try to include this

did you try to include this http://surbl-org-permanent-test-point.com/ in your test msg? did you check if the logs have that line mentioned above?

Reply | Permalink

Hi... again..

Hi... again..

forget my last message, after i remove QMAILQUEUE from tcp.smtp, simscan is gone...

The SURBL ok, but simscan not work :(

TFA

Reply | Permalink

now simscan is not enabled

now simscan is not enabled anymore because it is executed after qmail-dkim (take a moment to read the comments in the code as well :-)

DKIMQUEUE=/var/qmail/bin/simscan

but you are not running qmail-dkim.

Comment out that

#export SURBLQUEUE=/var/qmail/bin/qmail-dkim

line and you'll have it working

Reply | Permalink

Roberto, hi!

Roberto, hi!

Yes, i read... but, my file is exact you say... see:

export SURBL=1 # Comment out to enable SURBL filtering
export QMAILQUEUE=/var/qmail/bin/surblqueue # executes surblfilter
#export SURBLQUEUE=/var/qmail/bin/qmail-dkim # executes qmail-dkim after sublfilter
export DKIMQUEUE=/var/qmail/bin/simscan     # simscan is executed after qmail-dkim

In my first message i write a comment, say i try enable and disabled dkim, for tests only..

Reply | Permalink

you have the third line

you have the third line commented. Remove that comment and it will work :-)

Reply | Permalink

Sorry for the insistence, had

Sorry for the insistence, had already tested with this active line, but I commented, because I'm not using dkim.
Even so, it did not work.
I only had success when I removed QMAILQUEUE from /home/vpopmail/tcp.smtp, but then it did not pass the other tests, (virus and spam)

[] s

Reply | Permalink

Forget what you did before,

Forget what you did before, because you were overwriting QMAILQUEUE in your tcp.smtp. This is the reason why the directives in your run file were ignored. In particular you got the same behaviour both when that line was commented or not.

But now you have the run file dictating its rules. Unfortunately, according to what you have now in your run file, simscan can't be executed.

Your run file has no way to run simscan unless you don't let SURBLQUEUE or DKIMQUEUE point to it (simscan). If you don't want DKIM then use the first example on the top of this page

export QMAILQUEUE=/var/qmail/bin/surblqueue
export SURBLQUEUE=/var/qmail/bin/simscan

Please take the time to read carefully the docs, so that I don't have to repeat those explanations again inside the comments, something that is not useful for this blog

Reply | Permalink

Sorry for abuse... and, i

Sorry for abuse... and, i catch my wrong... after remove QMAILQUEUE from /home/vpopmail/etc/tcp.smtp, i forget run qmailctl cdb to compile those......

Thank you so much.

Reply | Permalink

This is the last.. i

This is the last.. i promissed..

I re-read all passes from your tutorial, and, i download your scripts from supervise, and all.

In config phase at simscan, at  "Turning on scanning" exists this:

echo ':allow,CHKUSER_WRONGRCPTLIMIT="3",QMAILQUEUE="/var/qmail/bin/simscan"' >> ~vpopmail/etc/tcp.smtp

How you note, my english is bad.. not find if this config need to disable.

How i do step by step, this stay at file. Sorry for my fault.

[] s

Reply | Permalink

You are right, I should have

You are right, I should have pointed out that QMAILQUEUE should be removed from tcp.smtp. I'll do that as soon as possibile.

Summarizing, now that you have removed QMAILQUEUE from tcp.smtp, SURBL is working and to execute simscan after SURBL you should define SURBLQUEUE as follows:

export SURBL=1
export QMAILQUEUE=/var/qmail/bin/surblqueue
export SURBLQUEUE=/var/qmail/bin/simscan

Reply | Permalink

Yes, i try.

Yes, i try.

I remove QMAILQUEUE from tcp.smtp and work:

@40000000583eef9018a2f7fc qmail-smtpd: message rejected (message contains an URL listed in SURBL blocklist): tiago@xxx.com.br from xxx.xxx.xx.xx to j

What you think, is done?

Reply | Permalink

I think that in this way you

I think that in this way you can't execute simscan. You must comment out that line.

I would look the details with strace

Reply | Permalink

Hi Roberto!

Hi Roberto!

The simscan is executing.. i enable simscan debug, see:

[...]

One question, at  my /home/vpopmail/etc/tcp.smtp o have:

127.0.0.1:allow,RELAYCLIENT=""
:allow,CHKUSER_WRONGRCPTLIMIT="3",QMAILQUEUE="/var/qmail/bin/simscan"

This environment read after or before QMAILQUEUE from smtpd/run ?

Reply | Permalink