SURBL filtering configuration

September 25, 2023 by Roberto Puzzanghera 32 comments

SURBLs are lists of web sites that have appeared in unsolicited messages. Unlike most lists, SURBLs are not lists of message senders.

Web sites seen in unsolicited messages tend to be more stable than the rapidly changing botnet IP addresses used to send the vast majority of them. Sender lists like zen.spamhaus.org can be used in a first stage filter to help identify 80% to 90% of unsolicited messages. SURBLs can help find about 75% of the otherwise difficult, remaining unsolicited messages in a second stage filter. Used together with sender lists, SURBLs have proven to be a highly-effective way to detect 95% of unsolicited messages.

The SURBL filter is part of the DKIM patch by Manvendra Bhangui and it's embedded in my combined patch.

Changelog

  • Sep 26, 2023
    -surblfilter logs the rejected URL in the qmail-smtpd log. It can now inspect both http and https URLs.
    -Improvements in man dkim.9, qmail-dkim.9 and surblfilter.9
  • May 17, 2023
    -Top level domains URL is changed. The update_tlds.sh script has been adjusted accordingly

 


To enable this filter you must export the variable SURBL with any value in your run file and pass the filter program to the QMAILQUEUE variable so that it can be executed before the delivery:

export SURBL=1
export QMAILQUEUE=/var/qmail/bin/surblqueue
export SURBLQUEUE=/var/qmail/bin/simscan

Actually the program is wrapped by surblqueue, as you can see. SURBLQUEUE will make the program to execute simscan when finished. If you don't define SURBLQUEUE the program executes qmail-queue to do the delivery.

NB: Remember to remove QMAILQUEUE from your tcp.smtp, otherwise it will overwrite your run file.

Be aware that the directory /var/qmail/control/cache must have the write priviledges for the user who runs qmail-smtpd, vpopmail in our case. I have adjusted my combined patch accordingly.

surblfilter requires two control files level2-tlds and level3-tlds in /var/qmail/control. The same can be obtained from the surbl.org website http://www.surbl.org/tld/three-level-tlds http://www.surbl.org/tld/two-level-tlds. These files sholud not be confused with the SURBL lists themselves but it is worth to update them monthly or so by building a cronjob like this:

cat > /usr/local/bin/update_tlds.sh << __EOF__
#!/bin/bash
#

cd /var/qmail/control
/usr/bin/wget https://www.surbl.org/static/three-level-tlds https://www.surbl.org/static/two-level-tlds
mv two-level-tlds level2-tlds
mv three-level-tlds level3-tlds
__EOF__
chmod +x /usr/local/bin/update_tlds.sh

Now setup the cronjobs (the second one cleans the cache folder once a day):

# surbl tlds update
2 2 23 * * /usr/local/bin/update_tlds.sh 1> /dev/null
# surbl cache purge
2 9 * * * find /var/qmail/control/cache/* -cmin +5 -exec /bin/rm -f {} \;

Executing surblfilter, qmail-dkim and simscan in cascade

Using a combination of QMAILQUEUE, SURBLQUEUE and DKIMQUEUE will make you run both filters and finally pass the message to simscan, which in turn calls clamd, spamd and finally executes qmail-queue:

export SURBL=1  # Comment to disable SURBL filtering
export QMAILQUEUE=/var/qmail/bin/surblqueue # executes surblfilter
export SURBLQUEUE=/var/qmail/bin/qmail-dkim # executes qmail-dkim afer surblfilter
export DKIMQUEUE=/var/qmail/bin/simscan # simscan is executed after qmail-dkim
export DKIMKEY=/var/qmail/control/domainkeys/%/default
# DKIM verification. Use carefully
export DKIMVERIFY="FGHKLMNOQRTVWp"
# This is to allow msg without "subject" in the h= list
export UNSIGNED_SUBJECT=1
# This is to avoid verification of outgoing messages
export RELAYCLIENT_NODKIMVERIFY=1

Testing

Send yourself an email with an URL such as http://surbl-org-permanent-test-point.com/ in the body. You should see the filter in action in your qmail-smtpd log:

qmail-smtpd: message rejected (message contains an URL listed in SURBL blocklist): user@domain.xy from 123.45.67.89 to yourself@yourdomain.xy helo yourmailserver.xy

Comments

SURBL working, but wrong result when filter

Hi

I'm testing how works SURBL , and I found that when a URL is found (I'm using the test url), qmail return an error

451 qq temporary problem (#4.3.0)

The mail is rejected, but !, something is not working :(

Any idea what could be happening ?

Txs

Reply |

SURBL working, but wrong result when filter

Pablo, it came to mind that I faced that error myself when upgrading the dkim/surbl patch to 1.42, but I think that the trouble was fixed before the release. Anyway the problem was that the script qmail/bin/surbqueue missed the first line #!/bin/sh and was not executed correctly

So I suggest to check that file. In case it misses the sh command download, patch and install again

Reply |

SURBL working, but wrong result when filter

Are you using my latest patch?

If yes use strace against qmail-smtpd, do the suggested surbl test and send me the output via mail

Reply |

Troubleshooting message_contains_an_URL_listed_in_SURBL_blocklist issue

Hello Roberto,

Thank you for the strace pointer as that assisted me with getting the qmail system up and running.

Now that the system is running I keep getting errors from random users ( Possibly who use outlook 365) that the message_contains_an_URL_listed_in_SURBL_blocklist and the message is rejected.

Any hint on troubleshooting?

Log details

2023-08-25 09:39:25.221051071 qlogenvelope: result=accepted code=250 reason=rcptto detail=chkuser helo=mail-pl1-f181.google.com mailfrom=user@remotedomain.tld rcptto=user@domain.tld relay=no rcpthosts=yes size= authuser= authtype= encrypted=tls sslverified=no localip=184.70.70.142 localport=25 remoteip=209.85.214.181 remoteport=59825 remotehost=mail-pl1-f181.google.com qp= pid=3465
2023-08-25 09:39:25.300011414 qlogreceived: result=rejected code=554 reason=queuereject detail=message_contains_an_URL_listed_in_SURBL_blocklist helo=mail-pl1-f181.google.com mailfrom=user@remotedomain.tld rcptto=user@domain.tld relay=no rcpthosts= size=10791 authuser= authtype= encrypted=tls sslverified=no localip=184.70.70.142 localport=25 remoteip=209.85.214.181 remoteport=59825 remotehost=mail-pl1-f181.google.com qp=3709 pid=3465

Also was working for this email address now get this

2023-08-22 10:32:22.107576646 qlogenvelope: result=accepted code=250 reason=rcptto detail=chkuser helo=CAN01-YQB-obe.outbound.protection.outlook.com mailfrom=user@remotedomain.tld rcptto=user@domain.tld relay=no rcpthosts=yes size= authuser= authtype= encrypted=tls sslverified=no localip=184.70.70.142 localport=25 remoteip=40.107.116.121 remoteport=1760 remotehost=mail-yqbcan01on2121.outbound.protection.outlook.com qp= pid=28069
2023-08-22 10:32:22.499573002 qlogreceived: result=rejected code=554 reason=queuereject detail=message_contains_an_URL_listed_in_SURBL_blocklist helo=CAN01-YQB-obe.outbound.protection.outlook.com mailfrom=user@remotedomain.tld rcptto=user@domain.tld relay=no rcpthosts= size=51374 authuser= authtype= encrypted=tls sslverified=no localip=184.70.70.142 localport=25 remoteip=40.107.116.121 remoteport=1760 remotehost=mail-yqbcan01on2121.outbound.protection.outlook.com qp=28378 pid=28069

Reply |

Troubleshooting message_contains_an_URL_listed_in_SURBL_blocklist issue

I don't think it's a trouble to shoot, as you have been hit by the SURBL filter, i.e. you had an URL listed in spam blocklist in the message body.

You can disable the SURBL filter, but you'll get those spam messages

PS: remember to avoid to post email addresses in the comments

Reply |

Troubleshooting message_contains_an_URL_listed_in_SURBL_blocklist issue

I understand that but these are not spam messages these are message that are sent via Outlook and because Microsoft insists on adding just to messages this happens from multiple clients.

Should they use webmail the messages goes through, it is only happening with Outlook Clients but not all. Some of my clients were having the issue and now they don't.

Is there a way to allow domains to bypass the surbl trieded adding domains to surbldomainwhite but I am not sure that it does anyting as that did not correct the issue.

Also this happes via Google hosted domains with clients using Outlook as the client.Again get random reception of emails from those addresses.

Is there a way to monitor the surbl to see what it is upset with?

Reply |

Troubleshooting message_contains_an_URL_listed_in_SURBL_blocklist issue

Now I understand better your case. Have a look at man surblfilter:

surblfilter  uses  QMAILRCPTS environment variable to get the recipient list. You can whitelist recipients by having the email addresses in surblrcpt control file. You can change the name of  this  control file by setting SURBLRCPT environment variable.

surblfilter uses the control file surbldomainwhite to whitelist a domain.

I've never tested these features, but I guess that this is what you are looking for, expecially as far as surbldomainwhite is concerned

Let me know if it solves.

Reply |

Troubleshooting message_contains_an_URL_listed_in_SURBL_blocklist issue

So the surbrcpt file does not change anything Had not switched my sending client to HTML.

is there any place that can get the detail of what the URL that is failing on this surbl filter?

Reply |

Troubleshooting message_contains_an_URL_listed_in_SURBL_blocklist issue

One thing you can do is to search a file in the control/cache dir with the exact time of the reject logline. Files with rejected domains are not empty, so it shouldn't be so hard to identify the domain

Reply |

Troubleshooting message_contains_an_URL_listed_in_SURBL_blocklist issue

unfortunately no. The log line doesn't report that, but I want to have a look at the code to see if we can extract the banned URL and write it in the log line

Reply |

Troubleshooting message_contains_an_URL_listed_in_SURBL_blocklist issue

Thanks for that I am now able to receive from those domain names.

Does that mean I will have to add all the domain names that use outlook clients because that is where the issue is. If I send from Outlook with RTF format it gets through but if I send with HTML format it gets blocked.

The surblrcpt has been built and updated with *@domainname.com and this appears to work for the entire domain.

Is there anything else that may allow this crap from outlook as it is the way that Microsoft Formats it's HTML email that is failing and not plain text or rtf formats.

Thanks in advance as you have done a great job with this documention.

Reply |

Troubleshooting message_contains_an_URL_listed_in_SURBL_blocklist issue

I think yes, I don't see any other way

Reply |

Typo in the domainkey directory

Howdy,

After looking through the script I found a minor typo, the location for the domainkeys are located here:

/var/qmail/control/domainkeys/%/default

So the part to change should be:

export SURBL=1  # Comment to disable SURBL filtering
export QMAILQUEUE=/var/qmail/bin/surblqueue # executes surblfilter
export SURBLQUEUE=/var/qmail/bin/qmail-dkim # executes qmail-dkim afer surblfilter
export DKIMQUEUE=/var/qmail/bin/simscan # simscan is executed after qmail-dkim
export DKIMKEY=/var/qmail/control/domainkeys/%/default
# DKIM verification. Use carefully
export DKIMVERIFY="FGHKLMNOQRTVWp"
# This is to allow msg without "subject" in the h= list
export UNSIGNED_SUBJECT=1
# This is to avoid verification of outgoing messages
export RELAYCLIENT_NODKIMVERIFY=1

Reply |

Typo in the domainkey directory

Thanks. Corrected

Reply |

Minor typo in the update_tlds.sh - should be httpS

Hi Roberto,

in your script update_tlds.sh you use http. This throws an error, 404 not found, but the httpS is working so the script should be.

cat > /usr/local/bin/update_tlds.sh << __EOF__
#!/bin/bash
#

cd /var/qmail/control
/usr/bin/wget https://www.surbl.org/tld/three-level-tlds https://www.surbl.org/tld/two-level-tlds
mv two-level-tlds level2-tlds
mv three-level-tlds level3-tlds
__EOF__
chmod +x /usr/local/bin/update_tlds.sh

As always, thnx for all your hard work!

GoofY

Reply |

Minor typo in the update_tlds.sh - should be httpS

Thank you. Corrected.

Reply |

Error compiling surblfilter

Hi

I'm trying to apply the patch on FreeBsd, but after solving many errors I was stuck with this

./load surblfilter envread.o strerr_die.o strerr_sys.o control.o alloc.o alloc_re.o error.o error_str.o auto_qmail.o case_startb.o byte_diff.o str_cspn.o byte_copy.o byte_chr.o byte_rchr.o byte_cr.o getln.o getln2.o open_read.o str_len.o str_diffn.o str_cpy.o str_chr.o scan_xlong.o now.o scan_ulong.o mess822_ok.o constmap.o ip.o dns.o ipalloc.o fmt_str.o fmt_ulong.o socket_v6any.o socket_v4mappedprefix.o sgetopt.o subgetopt.o base64sub.o case_diffb.o stralloc.a substdio.a -lresolv

/usr/bin/ld: cannot find -lresolv

cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1

After reading & reading I finally understund that FreeBSD doesn't have "resolv", so, the solution was to remove "-lresolv" from patch

I'm not using "all" the patch that you made, because I'm using ports on freebsd, but I use your guide to configure a lot of things

Thanks for your work

Reply |

Error compiling surblfilter

Thanks for the contribution. As many others here reported compilation breaks with freeBSD, it would be nice if you share more informations about the fixes you made...

Reply |

SURBL not working..

Hi Roberto!

i using your howto to build a new mailserver and SURBL seemed very cool.

I do all , but, not happend.. I send messages with urls from list but all have passed.

I not implement DKIM, this is problem?

my qmail-smtpd/run is:

QMAILDUID=`id -u vpopmail`
NOFILESGID=`id -g vpopmail`
MAXSMTPD=`cat /var/qmail/control/concurrencyincoming`
SOFTLIMIT=`cat /var/qmail/control/softlimit`

export SMTPD_GREETDELAY=15
export DROP_PRE_GREET=1

export CHKUSER_START=ALWAYS

export SURBL=1 # Comment out to enable SURBL filtering
export QMAILQUEUE=/var/qmail/bin/surblqueue # executes surblfilter
#export SURBLQUEUE=/var/qmail/bin/qmail-dkim # executes qmail-dkim after sublfilter <== i try enable this, but not happen
export DKIMQUEUE=/var/qmail/bin/simscan     # simscan is executed after qmail-dkim

exec /usr/local/bin/softlimit -m "$SOFTLIMIT" \
    /usr/local/bin/tcpserver -v -H -R -l 0 \
    -x /home/vpopmail/etc/tcp.smtp.cdb -c "$MAXSMTPD" \
    -u "$QMAILDUID" -g "$NOFILESGID" 0 25 \
    /var/qmail/bin/qmail-smtpd 2>&1

Please, send one light! :-)

[] s

Tiago Oliveira de Jesus

Reply |

did you try to include this

did you try to include this http://surbl-org-permanent-test-point.com/ in your test msg? did you check if the logs have that line mentioned above?

Reply |

Hi... again..

Hi... again..

forget my last message, after i remove QMAILQUEUE from tcp.smtp, simscan is gone...

The SURBL ok, but simscan not work :(

TFA

Reply |

now simscan is not enabled

now simscan is not enabled anymore because it is executed after qmail-dkim (take a moment to read the comments in the code as well :-)

DKIMQUEUE=/var/qmail/bin/simscan

but you are not running qmail-dkim.

Comment out that

#export SURBLQUEUE=/var/qmail/bin/qmail-dkim

line and you'll have it working

Reply |

Roberto, hi!

Roberto, hi!

Yes, i read... but, my file is exact you say... see:

export SURBL=1 # Comment out to enable SURBL filtering
export QMAILQUEUE=/var/qmail/bin/surblqueue # executes surblfilter
#export SURBLQUEUE=/var/qmail/bin/qmail-dkim # executes qmail-dkim after sublfilter
export DKIMQUEUE=/var/qmail/bin/simscan     # simscan is executed after qmail-dkim

In my first message i write a comment, say i try enable and disabled dkim, for tests only..

Reply |

you have the third line

you have the third line commented. Remove that comment and it will work :-)

Reply |

Sorry for the insistence, had

Sorry for the insistence, had already tested with this active line, but I commented, because I'm not using dkim.
Even so, it did not work.
I only had success when I removed QMAILQUEUE from /home/vpopmail/tcp.smtp, but then it did not pass the other tests, (virus and spam)

[] s

Reply |

Forget what you did before,

Forget what you did before, because you were overwriting QMAILQUEUE in your tcp.smtp. This is the reason why the directives in your run file were ignored. In particular you got the same behaviour both when that line was commented or not.

But now you have the run file dictating its rules. Unfortunately, according to what you have now in your run file, simscan can't be executed.

Your run file has no way to run simscan unless you don't let SURBLQUEUE or DKIMQUEUE point to it (simscan). If you don't want DKIM then use the first example on the top of this page

export QMAILQUEUE=/var/qmail/bin/surblqueue
export SURBLQUEUE=/var/qmail/bin/simscan

Please take the time to read carefully the docs, so that I don't have to repeat those explanations again inside the comments, something that is not useful for this blog

Reply |

Sorry for abuse... and, i

Sorry for abuse... and, i catch my wrong... after remove QMAILQUEUE from /home/vpopmail/etc/tcp.smtp, i forget run qmailctl cdb to compile those......

Thank you so much.

Reply |

This is the last.. i

This is the last.. i promissed..

I re-read all passes from your tutorial, and, i download your scripts from supervise, and all.

In config phase at simscan, at  "Turning on scanning" exists this:

echo ':allow,CHKUSER_WRONGRCPTLIMIT="3",QMAILQUEUE="/var/qmail/bin/simscan"' >> ~vpopmail/etc/tcp.smtp

How you note, my english is bad.. not find if this config need to disable.

How i do step by step, this stay at file. Sorry for my fault.

[] s

Reply |

You are right, I should have

You are right, I should have pointed out that QMAILQUEUE should be removed from tcp.smtp. I'll do that as soon as possibile.

Summarizing, now that you have removed QMAILQUEUE from tcp.smtp, SURBL is working and to execute simscan after SURBL you should define SURBLQUEUE as follows:

export SURBL=1
export QMAILQUEUE=/var/qmail/bin/surblqueue
export SURBLQUEUE=/var/qmail/bin/simscan

Reply |

Yes, i try.

Yes, i try.

I remove QMAILQUEUE from tcp.smtp and work:

@40000000583eef9018a2f7fc qmail-smtpd: message rejected (message contains an URL listed in SURBL blocklist): tiago@xxx.com.br from xxx.xxx.xx.xx to j

What you think, is done?

Reply |

I think that in this way you

I think that in this way you can't execute simscan. You must comment out that line.

I would look the details with strace

Reply |

Hi Roberto!

Hi Roberto!

The simscan is executing.. i enable simscan debug, see:

[...]

One question, at  my /home/vpopmail/etc/tcp.smtp o have:

127.0.0.1:allow,RELAYCLIENT=""
:allow,CHKUSER_WRONGRCPTLIMIT="3",QMAILQUEUE="/var/qmail/bin/simscan"

This environment read after or before QMAILQUEUE from smtpd/run ?

Reply |