Installing and configuring Spamassassin

Hi Roberto,

The purpose of the ExtractText plugin is to, when enabled, convert attachments (including imagens, byt the use of an OCR) into plain text in order to SpamAssassin to apply its rules to this text. So if we receive doc/pdf/images with spammy text in them, SpamAssassin will now be able to safely mark the email as spam.

In order to do that, we need to have installed some external programs in our server. The configuration lines added to local.cf have to load these programs to scan each message attachment. Also, the rules provided in the example at the SpamAssassin Apache page are just examples. Here are the correct instructions to make the ExtractText plugin to work. I hope you can test them yourself and update this page (and also the "migrating to SpamAssassin 4.0.0" page.

1. Install the required external programs

Here we run Debian, so we need:

apt-get install antiword
apt-get install docx2txt
apt-get install unrtf
apt-get install odt2txt
apt-get install tesseract-ocr
apt-get install poppler-utils

2. Add the following lines to the local.cf file. NOTE: your tutorial is missing the required extracttext_external lines:

loadplugin Mail::SpamAssassin::Plugin::ExtractText
ifplugin Mail::SpamAssassin::Plugin::ExtractText

extracttext_external  pdftotext  /usr/bin/pdftotext -nopgbrk -layout -enc UTF-8 {} -
extracttext_use       pdftotext  .pdf application/pdf

# http://docx2txt.sourceforge.net
extracttext_external  docx2txt   /usr/bin/docx2txt {} -
extracttext_use       docx2txt   .docx application/docx

extracttext_external  antiword   /usr/bin/antiword -t -w 0 -m UTF-8.txt {}
extracttext_use       antiword   .doc application/(?:vnd\.?)?ms-?word.*

extracttext_external  unrtf      /usr/bin/unrtf --nopict {}
extracttext_use       unrtf      .doc .rtf application/rtf text/rtf

extracttext_external  odt2txt    /usr/bin/odt2txt --encoding=UTF-8 {}
extracttext_use       odt2txt    .odt .ott application/.*?opendocument.*text
extracttext_use       odt2txt    .sdw .stw application/(?:x-)?soffice application/(?:x-)?starwriter

extracttext_external  tesseract  {OMP_THREAD_LIMIT=1} /usr/bin/tesseract -c page_separator= {} -
extracttext_use       tesseract  .jpg .png .bmp .tif .tiff image/(?:jpeg|png|x-ms-bmp|tiff)

add_header   all          ExtractText-Flags _EXTRACTTEXTFLAGS_
header       PDF_NO_TEXT  X-ExtractText-Flags =~ /\bpdftotext_NoText\b/
describe     PDF_NO_TEXT  PDF without text
score        PDF_NO_TEXT  0.001

header       DOC_NO_TEXT  X-ExtractText-Flags =~ /\b(?:antiword|openxml|unrtf|odt2txt)_NoText\b/
describe     DOC_NO_TEXT  Document without text
score        DOC_NO_TEXT  0.001

header       EXTRACTTEXT  exists:X-ExtractText-Flags
describe     EXTRACTTEXT  Email processed by extracttext plugin
score        EXTRACTTEXT  0.001

endif

3. Instructions on how to configure the plugin

The official website suggests the configuration lines above. But this is just an example and there is a small adjustment to be made.

a. Here with me, the string added to X-ExtractText-Flags was only 'NoText' when the attached word/pdf file didn't have any text. So the first two rules wouldn't be triggered. For this reason, I replaced the PDF_NO_TEXT and DOC_NO_TEXT with:

header DOC_NO_TEXT X-ExtractText-Flags =~ /\bNoText\b/
describe DOC_NO_TEXT Document without text
score DOC_NO_TEXT 0.2

b. Note how I adjusted the score from 0.001 to 0.2. With a score of 0.001, the original rule won't do anything. But, it is up to you to assign a score for emails containing doc/pdf files with no text. I ajusted the rule to a score of 0.2.

Personal note: honestly, both rules PDF_NO_TEXT and DOC_NO_TEXT could be safely removed. They are present in the documentation only so you know they are available.

The final rule, EXTRACTTEXT should be left alone. This rule only exists so we have EXTRACTTEXT in the email header in the list of all rules that the message triggered, so we know that attachments were present in the email and they were converted/scanned by the plugin. It is a great debugging tool.

Cheers,

Gabriel.

Thanks a lot Gabriel. I'll read and test carefully and I'll use it to improve this page

Hi Roberto,

After finally making the move to SA 4.0.0, I noticed that a new header was included, but its value is weird, IMHO:

X-Spam-ExtractText-Flags: _EXTRACTTEXTFLAGS_

Isn't this _EXTRACTTEXTFLAGS_  a variable that should be replaced with something else?

Also:

add_header   all          ExtractText-Flags _EXTRACTTEXTFLAGS_ 

Shouldn't this be:

add_header   all          X-ExtractText-Flags _EXTRACTTEXTFLAGS_ 

Please advise!

Cheers!

I did a test changing like this with no luck

X-Spam-ExtractText-Flags: _EXTRACTTEXTFLAGS_

Hi Gabriel,

Actually I would expect it to fill that _EXTRACTTEXTFLAGS_ tag with the notes of the plugin, something like

X-ExtractText-Flags: openxml_NoText

but I have it as is.

Concerning the other question, I copied the example from the man page here

I hope that someone else can be more specific than me...

I'll investigate this further and let you know. Thanks.

Hi Roberto,

After upgrading our OS here to the latest version, spamd stopped working. Upon further inspection, I found this in the log file:

Tue Aug 24 18:54:43 2021 [9957] warn: spamd: unauthorized connection from ::1 [::1]:59928 to port 783, fd 5 at /usr/local/bin/spamd line 1627.

This meant that spamd was listening on 127.0.0.1 but not at ::1 (IPv6 version of 127.0.0.1). Therefore, I replaced IP= with:

IP=::1

and the problem was gone.

Just a small note for you to upgrade the script to support both 127.0.0.1 and ::1.

Cheers.

Thank you.

Probably you have to adjust for ipv6 dovecot as well...

For anyone still having issues with GEOIP or GEOIP2, I had to manually patch the file with the help of some freenode people.

Here's the link to the patch: https://github.com/neilgierman/awstats/compare/GeoIP2...Grinnz:patch-1

Here I had to manually create /etc/mail/spamassassin/.spamassassin/ and chown spamd:spamd otherwise I would see error messages in the log files saying that the tx-reputation file couldn't be created.

Hi,

Spamassassin is installed correctly and is working fine for years and still is. While spamassassin scans incoming email from external sources, it does not scan emails that is being send locally, (from local to local or local to external domains). Is there a way to make it scan local emails as well? I tried modifying tcp rules to include simscan for local IP addresses but it still does not work.

thanks

Try to recompile simscan adding 

--enable-spam-auth-user=y 

to your configure. This should "turn on spam scanning for authenticated users (default no)."

installed everithing, but got this error (slackware current)

Util.c: loadable library and perl binaries are mismatched (got handshake key 0xdb00080, needed 0xde00080),

anybody knows where to grab..

Let me understand... you upgraded perl and then your installed modules stopped to work?

Hello,

Being a long time qmail user  I recently wanted to switch to full SSL. Unfortunately I cannot succeed having both SSL and AUTH working at same time.

When using stunnel, it just creates an openrelay : any login/pass is considered as OK (all vars are the same as for normal smtp)

/usr/local/bin/tcpserver -v -R -l mail.watchmusic.com -x /etc/tcp.smtp.cdb -c 50 -u 1008 -g 1003 0 465 /usr/local/bin/stunnel /var/qmail/control/stunnel_smtpd.conf

and stunnel_smtpd.conf

foreground = yes
cert = /var/qmail/control/servercert.pem
exec = /var/qmail/bin/qmail-smtpd
execargs = /var/www/vpopmail/bin/vchkpw /bin/true

When using sslserver, I've no openrelay, but I don't get AUTH on prompt (I've well exported the SMTPAUTH var)

sslserver -e -vR -l myserverfqdn -c 30 -u 508 -g 503 -x /etc/tcp.ssmtp.cdb 0.0.0.0 465 qmail-smtpd myserverfqdn /var/www/vpopmail/bin/vchkpw /bin/true

Has anybody succeeded having SSL and AUTH at same time ?

I think I did manage to have SSL on 465 with AUTH. I had to play with the run file of the specific service (on my system, /var/qmail/service/smtpd-465/run). Can you post yours ?

I don't use SSL 465

Spamassassin 3.4.1 was not working.  I was getting spamd.log hits

warn: spamd: unauthorized connection from ::1 [::1]:52080 to port 783, fd 5 at /usr/local/bin/spamd line 1600.

and spamd was not working at all.

I added ::1  in the spamdctl file like so:

DOPTIONS="-x -u spamd -A 127.0.0.1,::1 -s /var/log/spamd.log -H /home/spamd -d -c --pidfile=$PIDFILE"

I think you have to check that spamd is listening on IPv6 as well and eventually add a "-i [::1]:783" option to your spamdctl. Look here for details http://spamassassin.apache.org/full/3.4.x/doc/spamd.html

Roberto,

We have a new email server for 300 people thanks to your guide.  Being able to use Roundcube with PGP was the key piece.   I had to keep the old qmail / sqmail  with the PGP plugin up until today.

IPv6 is working great and spamassassin is doing the job.

I sent you a couple of coffee's and we thank you for keeping the awesome qmail alive.

John D. Trolinger

Hi,

I have this error on load :

error: Can't locate loadable object for module Geo::IP in @INC (@INC contains: /usr/local/share/perl5 /usr/local/lib64/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at /usr/local/share/perl5/Geo/IP.pm line 42.

Thanks

the problem is missing geoip-perl-api - solve like this:

cd /usr/local/src
wget https://github.com/maxmind/geoip-api-perl/archive/v1.45.tar.gz
tar zxvf https://github.com/maxmind/geoip-api-perl/archive/v1.45.tar.gz
cd geo-api-perl-1.45
perl Makefile.PL
make
make install
ldconfig

GEO:IP is already mentioned above among the packages that must be installed via CPAN...

I would try to see which module is missing looking inside that file/line

did you installed GEO:IP?

I have this in cpan

install Geo::IP

Geo::IP is up to date (1.45).

I had to disable loadplugin Mail::SpamAssassin::Plugin::URILocalBL  in v341.pre   to get rid of this error in the spamd.log

error: Can't locate loadable object for module Geo::IP in @INC

I had the same issue. It was resolved by installing GeoIP2 and uninstall IP::GeoIP. Spamd may complain about missing Class::XSAccessor, after installing it there was no more error.

For update Spamassassin 3.4.1 is necesary update the db (in my case):

ALTER TABLE `awl` ADD signedby VARCHAR(255) NOT NULL DEFAULT '' AFTER totscore;

Regards

How can I disable the use of spamassassin? Just uninstalling it?

Thanks!

no, just edit /var/qmail/control/simcontrol and put spam=no. update the simcontrol after that:

$ /var/qmail/bin/simscanmk

Wow, just found this post yesterday when I wanted to install Spamassassin on my Slackware64-current box. I can't thank you enough for this superb post. Grazie mille!

Thank you for a great tutorial.

I was wondering did you manage to make the new Geo::IP plug in working. I have installed from cpan and updated (geoiplookup works) but spamassassin doesn't seems to take that in consideration when I block a country. I have enable it from init.pre. Any ideea what can be?

Thanks

Sorry, I don't use that plugin