Merry Xmas and happy new... patch!

Massive Christmas present by my italian friend Luca Franceschini of digitalmind. He merged his combo with my combined patch (2016.12.02 version) adding several (heavily customized) patches and functionalities. Luca is an expert system administrator and a C programmer who manages big servers.

The most powerful addition is his "qlog patch", which records detailed logs of qmail-smtpd. Once you will apply the new patch the qmail-smtpd log will be shown as follows:

@400000005855db3020335304 CHKUSER accepted sender: from <sender@senderdomain.com|remoteinfo/auth:|chkuser-identify:> remote <helo:smtp.senderdomain.com|remotehostname:unknown|remotehostip:1.2.3.4> rcpt <> : sender accepted
@400000005855db302064cefc CHKUSER accepted rcpt: from <sender@senderdomain.com|remoteinfo/auth:|chkuser-identify:> remote <helo:smtp.senderdomain.com|remotehostname:unknown|remotehostip:1.2.3.4> rcpt <user@rcptdomain.com> : found existing recipient
@400000005855db3021a8d434 qmail-smtpd[11928]: rbl: ip=1.2.3.4 query=4.3.2.1.zen.spamhaus.org result=ignore message=''
@400000005855db3028802424 qmail-smtpd[11928]: rbl: ip=1.2.3.4 query=4.3.2.1.b.barracudacentral.org result=ignore message=''
@400000005855db3028806a74 policy_check: remote sender@senderdomain.com -> local user@rcptdomain.com (UNAUTHENTICATED SENDER)
@400000005855db302880ef44 policy_check: policy allows transmission
@400000005855db3028811e24 qlogenvelope: result=accepted code=250 reason=rcptto detail=chkuser helo=smtp.senderdomain.com mailfrom=sender@senderdomain.com rcptto=user@rcptdomain.com relay=no rcpthosts=yes size= authuser= authtype= encrypted= sslverified=no localip=10.0.0.4 localport=25 remoteip=1.2.3.4 remoteport=43618 remotehost= qp= pid=11928
@400000005855db3226010eac simscan:[11928]:CLEAN (1.30/9.50):1.8847s:####### Subject ############## 1.2.3.4:sender@DEM.senderdomain.com:user@rcptdomain.com
@400000005855db322a890f9c mail recv: pid 11928 from <sender@DEM.senderdomain.com> qp 11934
@400000005855db322a891b54 qmail-smtpd: message accepted: sender@DEM.senderdomain.com from 1.2.3.4 to user@rcptdomain.com helo smtp.senderdomain.com
@400000005855db322a892324 qlogreceived: result=accepted code=250 reason=queueaccept detail= helo=smtp.senderdomain.com mailfrom=sender@senderdomain.com rcptto=user@rcptdomain.com relay=no rcpthosts= size=2689 authuser= authtype= encrypted= sslverified=no localip=10.0.0.4 localport=25 remoteip=1.2.3.4 remoteport=43618 remotehost= qp=11934 pid=11928

You can store the "qlogenvelope" lines in separate files organized by date for backup purposes, if you like, by means of a scriptlet (look at the new qmail-smtpd/log/run file in the Configuring page).

You should have noticed that the "qmail-smtpd:" log line (due to the logging Kyle Wheeler's patch) is still present despite some redundancy, for the fact that it is more explicit than the new "qlog" in some cases.

The qmail-dnsbl logic from the Fabio Busatto patch was slightly changed (see below), and moved after chkuser in order to decrease the number of calls to spamhaus and the other block lists.

Another big addition is the qregex patch, that I've never added before because of collisions with the dkim patch in qmail-smtpd.c. Many of the functions of the qregex were improved by Manvendra Banghui, while Luca Franceschini added the new control file 'badhelonorelay' and renamed "badmailto" to "badrcptto". The original logic of the qregex patch remains intact.

For those who have my previous patch installed there are some minor adjustments to consider:

  • the libdomainkey.a library is now linked from /usr/lib, so you have to copy that library there
  • the old qmail-tap was replaced with an improved version by Mihai Secasiu. Look below to understand how to modify your control/taps file

Have fun!

Changelog

This is the detailed changelog with respect to the patch I published on december 2 2016:

an entry 'qlogenvelope' is generated after accepting or rejecting every recipients in the envelope phase, example:

​qlogenvelope: result=rejected code=553 reason=rblreject detail=b.barracudacentral.org helo=test.machine.it mailfrom=test@domain.com rcptto=test@pippo.com relay=no rcpthosts=yes size= authuser= authtype= encrypted= sslverified=no localip=15.15.15.15 localport=25 remoteip=14.143.30.83 remoteport=57502 remotehost= qp= id=39156
​qlogenvelope: result=accepted code=250 reason=rcptto detail=chkuser helo=test mailfrom=test@test.com rcptto=test@pippo.com relay=no rcpthosts=yes size= authuser= authtype= encrypted= sslverified=no localip=15.15.15.15 localport=25 remoteip=12.181.218.154 remoteport=57742 remotehost= qp= pid=37357

an entry 'qlogreceived' is generated after DATA (message accepted o rejected by qmail-queue)

qlogreceived: result=accepted code=250 reason=queueaccept detail= helo=test.machine.it mailfrom=test@domain.com rcptto=test@gmail.com relay=yes rcpthosts= size= authuser=pippo@pippo.com,pluto@pippo.com authtype=login encrypted=tls sslverified=no localip=192.168.200.162 localport=25 remoteip=192.168.200.162 remoteport=52602 remotehost= qp=30982 pid=30980

Comments

Hi Roberto, thank you for the good news about big patch improvement :-)

Two quick things:

1) regarding the "reject null senders patch", while I do not have the control/rejectnullsenders file, I cannot receive null sender email. Is this intended to be the default behaviour ? BTW, I'm not getting nullsenders mails to the mailing lists, so I suspect it might me caused my ezmlm and not qmail...

In the "anonymized" log used as example, you are leaving a public IP address...of a postfix server I'm managing ! Did you experience any problem with that or it happened just by chance ?

Ciao !

1) regarding the "reject null senders patch", while I do not have the control/rejectnullsenders file, I cannot receive null sender email. Is this intended to be the default behaviour ? BTW, I'm not getting nullsenders mails to the mailing lists, so I suspect it might me caused my ezmlm and not qmail...

No, this is not the default behaviour, even because it would break RFC compatibility

In the "anonymized" log used as example, you are leaving a public IP address...of a postfix server I'm managing ! Did you experience any problem with that or it happened just by chance ?

uh, what an unbelievable coincidence! That was really your IP in my logs... do you want that I hide it?

Thank you Roberto. Ok, then ezmlm has some issue with the null sender mail, since it doesn't get through. I was completing a registration process for a ezmlm managed address and cannot get the mail. The site explicitly says "NOTE: If you are using a mail server that is not RFC 2821/2822 compliant in that it rejects or discards emails with "NULL" addresses, you will NOT be able to complete the registration process. Many mail appliances have this option available and some set it as a default. You should not use this feature except in extreme circumstances as it often breaks normal mail server operations."

Regarding the IP that's probably better if you can anonymize it, but just if it's quick for you, didn't want to cause hassle :-)

I double checked the code, and the default behaviour is

unsigned int rejnsmf = 0;

so it is disabled by default. And I can confirm that I receive nullsenders msg here on my server

Thank you Roberto. Can you confirm you are getting the message also to a mailing list ?

I've never had any issue with mailing lists as well

Thank you, I will have a deeper look at that, might be some mailing list setting.

Just in case it might be helpful for someone else, the null sender email is rejected from ezmlm as it appears as a bounce:

@400000005890b1cd22b50c44 starting delivery 23226: msg 137479 to local domain.xxx-mailinglist@domain.xxx
@400000005890b1cd22b5102c status: local 1/10 remote 0/204
@400000005890b1cd2329944c delivery 23226: failure: ezmlm-reject:_fatal:_I_do_not_reply_to_bounce_messages_(#5.7.2)/