Greylisting for qmail

April 18, 2023 by Roberto Puzzanghera 13 comments

Greylisting is a method of defending e-mail users against spam. A mail transfer agent (MTA) using greylisting will "temporarily reject" any email from a sender it does not recognize. If the mail is legitimate, the originating server will try again after a delay, and if sufficient time has elapsed, the email will be accepted.

While greylisting is not effective as in the past, it still cut a certain fraction of the total spam.

Changelog

  • Apr 18, 2023
    Bai Borko showed a different approach in purging the database of jgreylist and posted his script here. Basically, instead of deleting the old records his script leaves in the database the records with trusted IPs of the last 30 days based on the qmail-smtpd logs. Have a look!

qmail-spp greylisting plugin

 

I introduce here how greylisting can be implemented on qmail by means of another qmail-spp plugin, which saves the data in MySQL. Having the data in MySQL is useful to measure how much spam is blocked by greylisting.

  • More info here
  • Author: Manuel Mausz

Download

cd /usr/local/src
wget https://manuel.mausz.at/coding/qmail-spp/greylisting/greylisting-0.5.tgz
tar xzf greylisting-0.5.tgz
cd greylisting-0.5

Compile and install

gcc -std=c99 -o /var/qmail/plugins/greylisting greylisting.c -I/usr/include -I/usr/include/mysql -I/usr/local/include/mysql -L/usr/lib/mysql -L/usr/lib64/mysql -L/usr/local/lib/mysql -L/usr/local/lib64/mysql -lmysqlclient
strip greylisting

Install the configuration files

cp mysql.cnf /var/qmail/control
chown vpopmail:vchkpw /var/qmail/control/mysql.cnf
chmod 600 /var/qmail/control/mysql.cnf
cp greylisting.config /var/qmail/control/greylisting
chown root:root /var/qmail/control/greylisting
chmod 644 /var/qmail/control/greylisting

Install the mysql database and create the mysql user:

> mysql -u root -p

CREATE USER 'greylisting'@'localhost' IDENTIFIED BY '***';
GRANT USAGE ON *.* TO 'greylisting'@'localhost' REQUIRE NONE WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0;
CREATE DATABASE IF NOT EXISTS greylisting;
GRANT ALL PRIVILEGES ON greylisting.* TO 'greylisting'@'localhost';

Now install the database schema from the greylisting.sql file (note that in the same file you have examples for whitelist and blacklist queries):

mysql < greylisting.sql -D greylisting -u greylisting -p

Copy your mysql access parameters into your mysql.cnf file. For example:

cat > /var/qmail/control/mysql.cnf << __EOF__ 
[client] 
#socket=/run/mysqld/mysqld.sock 
#host=10.1.2.3   
user=greylisting 
password=secret
database=greylisting 
__EOF__

If the connection has to be done against a local mysql host, it will be via the mysql socket. If, as in my case, you have mysql in a different host, uncomment the "host" line.

Now define the "greylist" in the smtpplugin file:

# smtpplugins sample file 

# other lines here

[rcpt]
plugins/ifauthskip
plugins/greylisting

To enable greylisting you have to export this variable in your tcpserver environment. So put this in your qmail-smtpd run file

export GREYLISTING=""

or enable a tcprule in your tcp.smtp, for example:

:allow,GREYLISTING=""

Be aware that greylisting is never enabled for RELAYCLIENTS.

Finally, adjust the greylisting parameters in the /var/qmail/control/greylisting file

cat > /var/qmail/control/greylisting << __EOF__ 
mysql_default_file=control/mysql.cnf 
block_expire=2 
record_expire=2000 
record_expire_good=36 
loglevel=4 
__EOF__

block_expire=2 means that the IP will be blocked for 2 minutes, while record_expire=2000 means that after 2000 minutes it will be greylisted again.

Now install the program which will purge the database in your cronjob

cp greylisting_cleanup.sh /usr/local/sbin/greylisting_cleanup.sh
chmod +x /usr/local/sbin/greylisting_cleanup.sh

Here is a crontab example

# greylisting 
0 2 * * * /usr/local/sbin/greylisting_cleanup.sh >> /var/log/cron

If you want to use greylisting also in your submission port, it would be a good idea to switch off it in case the client have passed the authentication. In order to do this install the "skipifauth" plugin of the same author

wget https://notes.sagredo.eu/files/qmail/patches/qmail-spp/plugins/ifauthskip.c
cc -o /var/qmail/plugins/ifauthskip ifauthskip.c

and install it before greylisting in control/smtpplugin

[rcpt]
plugins/ifauthskip
plugins/greylisting

jgreylist

This is a greylist program by John Simpson. It is very good and honestly I don't know if I have to suggest this one or the previous one. It stores the data on files (have a look to the page of the author for details). He ships both a Perl and a C version of the program.

Here is how to install and configure the C program:

cd /usr/local/src
wget https://notes.sagredo.eu/files/qmail/patches/greylisting/jms/jgreylist.c
wget https://notes.sagredo.eu/files/qmail/patches/greylisting/jms/jgreylist-clean

Compile and install

cc -o /var/qmail/bin/jgreylist jgreylist.c
chmod 0750 /var/qmail/bin/jgreylist
cp jgreylist-clean /usr/local/sbin/jgreylist-clean
chmod +x /usr/local/sbin/jgreylist-clean
chown root:root /usr/local/sbin/jgreylist-clean

Define your jgreylist directory (where the data will be saved) and let qmail-smtpd to write into it

mkdir -m 0700 /var/qmail/jgreylist
chown vpopmail:vchkpw /var/qmail/jgreylist

add these variables to your qmail-smtpd run file

export JGREYLIST_DIR="/var/qmail/jgreylist" 
export JGREYLIST_LOG_SMTP=1

finally execute jgrelist before qmail-smtpd, for example

exec /usr/local/bin/softlimit -m "$SOFTLIMIT" \
/usr/local/bin/tcpserver -4 -v -R -l "$LOCAL" \
-x /home/vpopmail/etc/tcp.smtp.cdb -c "$MAXSMTPD" \
-u "$QMAILDUID" -g "$NOFILESGID" 0 25 \
/var/qmail/bin/jgreylist \
/var/qmail/bin/qmail-smtpd 2>&1

Install the "clean" program in your cronjob

# jgreylist
0 2 * * * /usr/local/sbin/jgreylist-clean >> /var/log/cron

Now restart qmail.


Bai Borko has a different approach in purging the database and posted his script here. Basically, instead of deleting the old records his script leaves in the database the records with trusted IPs of the last 30 days based on the qmail-smtpd logs. Have a look!

Comments

correction

[rcpt]
plugins/skipifauth <-- must be /plugins/ifauthskip
plugins/greylisting

Reply |

correction

thank you. Corrected

Reply |

small mistake

# jgreylist
0 2 * * * /usr/local/sbin/jgreylist_clean >> /var/log/cron

Hi Roberto,

pls correct name of the file to jgreylist-clean

Thx.

Reply |

small mistake

Thanks, corrected

Reply |

jgreylist clean DB

Hi Roberto,

i am using jgreylist for a long time. I was decided to change the approach for clean DB for old records.
Instead to clean DB for not used for a long time records , i create completely new DB.
To create the new DB i am using the backup files created in last 30 days in /var/log/qmail/backup/qmail-smtpd.* and /var/log/qmail/smtpd/current.
I parse these files for "result=accepted code=250" and create completely new DB only with IP's of mail servers which my mail server successful received mail in last 30 days.
I am using this script for a few weeks without any issues. The result is very good.
The DB is twice less and there is only trusted IP's of mail servers, also you can see lot of records with spammers IP's like this "greylist[586630]: 113.93.57.136: GREY first time" in /var/log/qmail/smtpd/current.
If somebody decides to use this script must disable in crontab original script /usr/local/bin/jgreylist-clean and to create new job like this:

0 3 * * * /usr/local/bin/jgreylist-create-db.sh

Feel free to test and share.

#!/bin/bash

SMTPDLOG='/var/log/qmail/smtpd/current'
BACKUPDIR='/var/log/qmail/backup'
JGRAYDIR='/var/qmail/jgreylist'
FIELD=`echo ${JGRAYDIR}|tr -dc '/'|wc -c`
FIELD=$((FIELD+3))
TMPLIST="$(mktemp)"
TMPLIST_TMP="$(mktemp)"
DAYS=30
INTNET='192.168.0'

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/qmail/bin

grlist() {
grep "result=accepted code=250" $1|tai64nlocal| cut -d ' ' -f1-2,20|tr '=' ' '|grep -v ${INTNET}|awk -F '.' '{print $1" "$2"."$3"."$4}'|cut -d ' ' -f1,2,5 >>$TMPLIST
}

for i in `find $BACKUPDIR -name "qmail-smtpd.*" -mtime -${DAYS}`; do
grlist $i
done

grlist $SMTPDLOG

cat $TMPLIST |sort -k 3 -u > $TMPLIST_TMP
mv $TMPLIST_TMP $TMPLIST
umask 077

while read line; do
MTIME=`echo "$line"|tr -d '-'|awk -F: '{print $1$2"."$3}'|awk '{print $1$2}'`
TMPSTR=''
for NUM in `echo "$line"|cut -d ' ' -f3|tr '.' ' '`;do
if [[ ${#NUM} -eq 1 ]] ; then
NUM="00${NUM}"
elif [[ ${#NUM} -eq 2 ]] ; then
NUM="0${NUM}"
fi
TMPSTR=${TMPSTR}${NUM}/
done

FILE=`echo ${JGRAYDIR}.tmp/${TMPSTR} |rev |cut -c2- |rev`
DIR=`echo ${FILE} |cut -d / -f -${FIELD} -`
#Make dir if not exist
if [ ! -d "$DIR" ]; then
mkdir -p "$DIR"
fi

touch -a -m -t $MTIME $FILE
done < $TMPLIST
chown -R vpopmail:vchkpw "${JGRAYDIR}.tmp"
mv ${JGRAYDIR} "${JGRAYDIR}.4del"
mv "${JGRAYDIR}.tmp" ${JGRAYDIR}
rm -fr "${JGRAYDIR}.4del" $TMPLIST

Reply |

jgreylist clean DB

Thanks a lot. Appreciated. I added a link to your post in the jgreylist section and at the top of this page.

Reply |

One more thing

Hi Roberto,

Please update your guide for jgreylist.

This row "/var/qmail/bin/jgreylist \" needs to be added after tcpserver and before qmail-smtpd in /var/qmail/supervise/qmail-smtpd/run script.
Like example below:

exec /usr/local/bin/softlimit -m "$SOFTLIMIT" \
/usr/local/bin/tcpserver -4 -v -R -l "$LOCAL" \
-x /home/vpopmail/etc/tcp.smtp.cdb -c "$MAXSMTPD" \
-u "$QMAILDUID" -g "$NOFILESGID" 0 25 \
/var/qmail/bin/jgreylist \
/var/qmail/bin/qmail-smtpd 2>&1

Reply |

One more thing

Thank you. Corrected

Reply |

Greylisting recommendations

Hello,

Congratulations for greate job!

I missed the note on the gray list. What is your recommendation?

Paulo.

Reply |

Greylisting recommendations

I installed the patch by Jan Mojzis and didn't liked it because it seems to me that the postgrey server needs posftix installed, as it looks for its socket.

Then I played with jms' jgreylist program and it works great here, despite of its age. I'm testing it right now. It's easy to install, with no need to patch, and it's well documented.

Reply |

Greylisting recommendations

Hi Roberto,

jgreylist program works great. But in some cases it's not useful agains the spammers.
Оnce spammers IP is already in the jgrey list DB jgreylist is not helping.
Like this example in following log lines:

@400000006202eb491916bcb4 jgreylist[2379920]: 85.202.169.20: OK known
@400000006202eb581830f47c GREETDELAY from 85.202.169.20: client disconnected

OR

@400000006203220e21823964 jgreylist[2406984]: 193.56.29.119: OK known
@400000006203221d24897e24 qmail-smtpd: read failed (hang up before quit cmd): (null) from 193.56.29.119 to (null) helo unknown

I made small script to clean jgreylist DB from such spammers IPs. The script works with both example above.
You can run the script depends of your needs . I run it hourly :

10 * * * * /usr/local/bin/jgreylist-clean-spammers.sh > /dev/null 2>&1


#!/bin/bash

SMTPDLOG='/var/log/qmail/smtpd/current'
JGRAYDIR='/var/qmail/jgreylist'
FIELD=`echo ${JGRAYDIR}|tr -dc '/'|wc -c`
FIELD=$((FIELD+3))
for i in ` grep -E "GREETDELAY .* client disconnected$|qmail-smtpd: read failed .* to \(null\) helo unknown$" $SMTPDLOG|grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}"|cut -d . -f1-3|uniq`; do
TMPSTR=''
for NUM in `echo $i|tr '.' ' '`;do
if [[ ${#NUM} -eq 1 ]] ; then
NUM="00${NUM}"
elif [[ ${#NUM} -eq 2 ]] ; then
NUM="0${NUM}"
fi
TMPSTR=${TMPSTR}${NUM}/
done
FILE=`echo ${JGRAYDIR}/${TMPSTR} |rev |cut -c2- |rev`
DIR=`echo ${FILE} |cut -d / -f -${FIELD} -`
if [ -f "$FILE" ]; then
rm -f "$FILE"
fi

if [ -d "$DIR" ] && [ -z "$(ls -A $DIR)" ]; then
rmdir $DIR
fi
done

Reply |

Greylisting recommendations

Thanks, appreciated. I'll test it in the following days.

I'm not sure that 100% of the example 2 is spam.  Look at the latest patch which is more verbose there.

As greylist program, I also tested this plugin, which is great http://qmail-spp.sourceforge.net/plugins/details/?id=30

It saves data to mysql. This can give you a measure of the mail percentage which is definitely blocked as spam and compare it with the spam perc blocked in other ways

Reply |

Greylisting recommendations

I'm not familiar at all with greylisting for qmail. I only played with Jan Mojzis' qmail-postgrey but it was ages ago and I don't remember much. But I know that Erwin Hoffmann grabbed this code to make his own greylisting in s/qmail, so if I wanted to investigate greylisting for qmail, I would look at what e.h. has done there and the differences with Jan Mojzi's patch.

If you find out that this is doable or you find another nice solution please let me know...

Reply |