SURBL filtering configuration

Hello Roberto,

Thank you for the strace pointer as that assisted me with getting the qmail system up and running.

Now that the system is running I keep getting errors from random users ( Possibly who use outlook 365) that the message_contains_an_URL_listed_in_SURBL_blocklist and the message is rejected.

Any hint on troubleshooting?

Log details

2023-08-25 09:39:25.221051071 qlogenvelope: result=accepted code=250 reason=rcptto detail=chkuser helo=mail-pl1-f181.google.com mailfrom=user@remotedomain.tld rcptto=user@domain.tld relay=no rcpthosts=yes size= authuser= authtype= encrypted=tls sslverified=no localip=184.70.70.142 localport=25 remoteip=209.85.214.181 remoteport=59825 remotehost=mail-pl1-f181.google.com qp= pid=3465
2023-08-25 09:39:25.300011414 qlogreceived: result=rejected code=554 reason=queuereject detail=message_contains_an_URL_listed_in_SURBL_blocklist helo=mail-pl1-f181.google.com mailfrom=user@remotedomain.tld rcptto=user@domain.tld relay=no rcpthosts= size=10791 authuser= authtype= encrypted=tls sslverified=no localip=184.70.70.142 localport=25 remoteip=209.85.214.181 remoteport=59825 remotehost=mail-pl1-f181.google.com qp=3709 pid=3465

Also was working for this email address now get this

2023-08-22 10:32:22.107576646 qlogenvelope: result=accepted code=250 reason=rcptto detail=chkuser helo=CAN01-YQB-obe.outbound.protection.outlook.com mailfrom=user@remotedomain.tld rcptto=user@domain.tld relay=no rcpthosts=yes size= authuser= authtype= encrypted=tls sslverified=no localip=184.70.70.142 localport=25 remoteip=40.107.116.121 remoteport=1760 remotehost=mail-yqbcan01on2121.outbound.protection.outlook.com qp= pid=28069
2023-08-22 10:32:22.499573002 qlogreceived: result=rejected code=554 reason=queuereject detail=message_contains_an_URL_listed_in_SURBL_blocklist helo=CAN01-YQB-obe.outbound.protection.outlook.com mailfrom=user@remotedomain.tld rcptto=user@domain.tld relay=no rcpthosts= size=51374 authuser= authtype= encrypted=tls sslverified=no localip=184.70.70.142 localport=25 remoteip=40.107.116.121 remoteport=1760 remotehost=mail-yqbcan01on2121.outbound.protection.outlook.com qp=28378 pid=28069

I don't think it's a trouble to shoot, as you have been hit by the SURBL filter, i.e. you had an URL listed in spam blocklist in the message body.

You can disable the SURBL filter, but you'll get those spam messages

PS: remember to avoid to post email addresses in the comments

I understand that but these are not spam messages these are message that are sent via Outlook and because Microsoft insists on adding just to messages this happens from multiple clients.

Should they use webmail the messages goes through, it is only happening with Outlook Clients but not all. Some of my clients were having the issue and now they don't.

Is there a way to allow domains to bypass the surbl trieded adding domains to surbldomainwhite but I am not sure that it does anyting as that did not correct the issue.

Also this happes via Google hosted domains with clients using Outlook as the client.Again get random reception of emails from those addresses.

Is there a way to monitor the surbl to see what it is upset with?

Now I understand better your case. Have a look at man surblfilter:

surblfilter  uses  QMAILRCPTS environment variable to get the recipient list. You can whitelist recipients by having the email addresses in surblrcpt control file. You can change the name of  this  control file by setting SURBLRCPT environment variable.

surblfilter uses the control file surbldomainwhite to whitelist a domain.

I've never tested these features, but I guess that this is what you are looking for, expecially as far as surbldomainwhite is concerned

Let me know if it solves.

So the surbrcpt file does not change anything Had not switched my sending client to HTML.

is there any place that can get the detail of what the URL that is failing on this surbl filter?

One thing you can do is to search a file in the control/cache dir with the exact time of the reject logline. Files with rejected domains are not empty, so it shouldn't be so hard to identify the domain

unfortunately no. The log line doesn't report that, but I want to have a look at the code to see if we can extract the banned URL and write it in the log line

Thanks for that I am now able to receive from those domain names.

Does that mean I will have to add all the domain names that use outlook clients because that is where the issue is. If I send from Outlook with RTF format it gets through but if I send with HTML format it gets blocked.

The surblrcpt has been built and updated with *@domainname.com and this appears to work for the entire domain.

Is there anything else that may allow this crap from outlook as it is the way that Microsoft Formats it's HTML email that is failing and not plain text or rtf formats.

Thanks in advance as you have done a great job with this documention.

I think yes, I don't see any other way

Howdy,

After looking through the script I found a minor typo, the location for the domainkeys are located here:

/var/qmail/control/domainkeys/%/default

So the part to change should be:

export SURBL=1  # Comment to disable SURBL filtering
export QMAILQUEUE=/var/qmail/bin/surblqueue # executes surblfilter
export SURBLQUEUE=/var/qmail/bin/qmail-dkim # executes qmail-dkim afer surblfilter
export DKIMQUEUE=/var/qmail/bin/simscan # simscan is executed after qmail-dkim
export DKIMKEY=/var/qmail/control/domainkeys/%/default
# DKIM verification. Use carefully
export DKIMVERIFY="FGHKLMNOQRTVWp"
# This is to allow msg without "subject" in the h= list
export UNSIGNED_SUBJECT=1
# This is to avoid verification of outgoing messages
export RELAYCLIENT_NODKIMVERIFY=1

Thanks. Corrected

Hi Roberto,

in your script update_tlds.sh you use http. This throws an error, 404 not found, but the httpS is working so the script should be.

cat > /usr/local/bin/update_tlds.sh << __EOF__
#!/bin/bash
#

cd /var/qmail/control
/usr/bin/wget https://www.surbl.org/tld/three-level-tlds https://www.surbl.org/tld/two-level-tlds
mv two-level-tlds level2-tlds
mv three-level-tlds level3-tlds
__EOF__
chmod +x /usr/local/bin/update_tlds.sh

As always, thnx for all your hard work!

GoofY

Thank you. Corrected.

Hi

I'm trying to apply the patch on FreeBsd, but after solving many errors I was stuck with this

./load surblfilter envread.o strerr_die.o strerr_sys.o control.o alloc.o alloc_re.o error.o error_str.o auto_qmail.o case_startb.o byte_diff.o str_cspn.o byte_copy.o byte_chr.o byte_rchr.o byte_cr.o getln.o getln2.o open_read.o str_len.o str_diffn.o str_cpy.o str_chr.o scan_xlong.o now.o scan_ulong.o mess822_ok.o constmap.o ip.o dns.o ipalloc.o fmt_str.o fmt_ulong.o socket_v6any.o socket_v4mappedprefix.o sgetopt.o subgetopt.o base64sub.o case_diffb.o stralloc.a substdio.a -lresolv

/usr/bin/ld: cannot find -lresolv

cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1

After reading & reading I finally understund that FreeBSD doesn't have "resolv", so, the solution was to remove "-lresolv" from patch

I'm not using "all" the patch that you made, because I'm using ports on freebsd, but I use your guide to configure a lot of things

Thanks for your work

Thanks for the contribution. As many others here reported compilation breaks with freeBSD, it would be nice if you share more informations about the fixes you made...

Hi Roberto!

i using your howto to build a new mailserver and SURBL seemed very cool.

I do all , but, not happend.. I send messages with urls from list but all have passed.

I not implement DKIM, this is problem?

my qmail-smtpd/run is:

QMAILDUID=`id -u vpopmail`
NOFILESGID=`id -g vpopmail`
MAXSMTPD=`cat /var/qmail/control/concurrencyincoming`
SOFTLIMIT=`cat /var/qmail/control/softlimit`

export SMTPD_GREETDELAY=15
export DROP_PRE_GREET=1

export CHKUSER_START=ALWAYS

export SURBL=1 # Comment out to enable SURBL filtering
export QMAILQUEUE=/var/qmail/bin/surblqueue # executes surblfilter
#export SURBLQUEUE=/var/qmail/bin/qmail-dkim # executes qmail-dkim after sublfilter <== i try enable this, but not happen
export DKIMQUEUE=/var/qmail/bin/simscan     # simscan is executed after qmail-dkim

exec /usr/local/bin/softlimit -m "$SOFTLIMIT" \
    /usr/local/bin/tcpserver -v -H -R -l 0 \
    -x /home/vpopmail/etc/tcp.smtp.cdb -c "$MAXSMTPD" \
    -u "$QMAILDUID" -g "$NOFILESGID" 0 25 \
    /var/qmail/bin/qmail-smtpd 2>&1

Please, send one light! :-)

[] s

Tiago Oliveira de Jesus

did you try to include this http://surbl-org-permanent-test-point.com/ in your test msg? did you check if the logs have that line mentioned above?

Hi... again..

forget my last message, after i remove QMAILQUEUE from tcp.smtp, simscan is gone...

The SURBL ok, but simscan not work :(

TFA

now simscan is not enabled anymore because it is executed after qmail-dkim (take a moment to read the comments in the code as well :-)

DKIMQUEUE=/var/qmail/bin/simscan

but you are not running qmail-dkim.

Comment out that

#export SURBLQUEUE=/var/qmail/bin/qmail-dkim

line and you'll have it working

Roberto, hi!

Yes, i read... but, my file is exact you say... see:

export SURBL=1 # Comment out to enable SURBL filtering
export QMAILQUEUE=/var/qmail/bin/surblqueue # executes surblfilter
#export SURBLQUEUE=/var/qmail/bin/qmail-dkim # executes qmail-dkim after sublfilter
export DKIMQUEUE=/var/qmail/bin/simscan     # simscan is executed after qmail-dkim

In my first message i write a comment, say i try enable and disabled dkim, for tests only..

you have the third line commented. Remove that comment and it will work :-)

Sorry for the insistence, had already tested with this active line, but I commented, because I'm not using dkim.
Even so, it did not work.
I only had success when I removed QMAILQUEUE from /home/vpopmail/tcp.smtp, but then it did not pass the other tests, (virus and spam)

[] s

Forget what you did before, because you were overwriting QMAILQUEUE in your tcp.smtp. This is the reason why the directives in your run file were ignored. In particular you got the same behaviour both when that line was commented or not.

But now you have the run file dictating its rules. Unfortunately, according to what you have now in your run file, simscan can't be executed.

Your run file has no way to run simscan unless you don't let SURBLQUEUE or DKIMQUEUE point to it (simscan). If you don't want DKIM then use the first example on the top of this page

export QMAILQUEUE=/var/qmail/bin/surblqueue
export SURBLQUEUE=/var/qmail/bin/simscan

Please take the time to read carefully the docs, so that I don't have to repeat those explanations again inside the comments, something that is not useful for this blog

Sorry for abuse... and, i catch my wrong... after remove QMAILQUEUE from /home/vpopmail/etc/tcp.smtp, i forget run qmailctl cdb to compile those......

Thank you so much.

This is the last.. i promissed..

I re-read all passes from your tutorial, and, i download your scripts from supervise, and all.

In config phase at simscan, at  "Turning on scanning" exists this:

echo ':allow,CHKUSER_WRONGRCPTLIMIT="3",QMAILQUEUE="/var/qmail/bin/simscan"' >> ~vpopmail/etc/tcp.smtp

How you note, my english is bad.. not find if this config need to disable.

How i do step by step, this stay at file. Sorry for my fault.

[] s

You are right, I should have pointed out that QMAILQUEUE should be removed from tcp.smtp. I'll do that as soon as possibile.

Summarizing, now that you have removed QMAILQUEUE from tcp.smtp, SURBL is working and to execute simscan after SURBL you should define SURBLQUEUE as follows:

export SURBL=1
export QMAILQUEUE=/var/qmail/bin/surblqueue
export SURBLQUEUE=/var/qmail/bin/simscan

Yes, i try.

I remove QMAILQUEUE from tcp.smtp and work:

@40000000583eef9018a2f7fc qmail-smtpd: message rejected (message contains an URL listed in SURBL blocklist): tiago@xxx.com.br from xxx.xxx.xx.xx to j

What you think, is done?

I think that in this way you can't execute simscan. You must comment out that line.

I would look the details with strace

Hi Roberto!

The simscan is executing.. i enable simscan debug, see:

[...]

One question, at  my /home/vpopmail/etc/tcp.smtp o have:

127.0.0.1:allow,RELAYCLIENT=""
:allow,CHKUSER_WRONGRCPTLIMIT="3",QMAILQUEUE="/var/qmail/bin/simscan"

This environment read after or before QMAILQUEUE from smtpd/run ?