[FrontPage] [TitleIndex] [WordIndex]

SpamAssassin

Leveraging SpamAssassin analysis from TMDA provides a nice way to reduce TMDA's challenge-response workload, in particular towards messages that are clearly spam, and/or those that are clearly ham according to SpamAssassin. There are countless ways to combine SA with TMDA, so this page is here to collect descriptions of users different configurations. Feel free to add your own page below describing how you use SA with TMDA, or add comments to an existing page.

Serverwide Setup

I use Spamassassin globaly with 'qmailqueue' Patch. My Configuration:

In my tcpserver rules file i use: 

127.0.0.1:allow,RELAYCLIENT=""
server_ip:allow,RELAYCLIENT=""
:allow,QMAILQUEUE="/var/qmail/bin/qmail-queue.spamd"

qmail-queue.spamd: 

#!/bin/sh
/usr/bin/spamc | /var/qmail/bin/qmail-queue.orig

So only incoming mails are checked, not mails from localhost or ofmipd.

my 'user/.qmail' files: 

|preline /usr/local/src/tmda/bin/tmda-filter -S ~vpopmail/bin/vpopmail-vdir.sh
./Maildir/

my 'user/.tmda/config' files: 

macro SPAM_DELIVER deliver=/usr/local/vpopmail/domains/xxx.xx/enrico/Maildir/.spam/
# spam to the spam folder
headers 'X-Spam-Level: \*\*\*\*\*\*\*\*' SPAM_DELIVER
headers 'Subject: \*\*\*\*\*SPAM\*\*\*\*\*' SPAM_DELIVER

Quick and dirty... any comments, questions?

Slight variation (spamd/spamc)

By using spamd/spamc, you can gain some efficiency by not having to launch two processes for every arriving message. This setup is on a mailserver inside a firewall, which explains why I'm using a non-routable IP address in the second line of tcp.smtp.

tcp.smtp: 

127.:allow,RELAYCLIENT=""
192.168.0.:allow,RELAYCLIENT="",QMAILQUEUE="/usr/local/bin/qmail-spamc"

rc.local: 

/usr/local/bin/spamd &

~/.tmda/filters/incoming: 

# Whitelist/blacklist/filter rules here...
# Drop spam above a certain threshold
headers "^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*\*" drop 
# Save questionable spam for later review
headers "^X-Spam-Level: \*\*\*\*\*" deliver=~/MailFolders/spam/
EOF

Messages that aren't marked as spam are handled by the TMDA confirmation process.

Shell Account Setup (non-serverwide)

A simple setup for a qmail server where you only have access through .qmail files (like on a shell account with an ISP)

First set up a file ".default-delivery" that contains: 

| /var/qmail/bin/preline /path/to/spamassassin | /path/to/tmda/bin/tmda-filter
/path/to/Maildir/

Now soft link .qmail-default and .qmail to this file. In my case I have a set of domains pointed at my shell account, so I set up specific users on domains and dont use a catch all .qmail file. To do this I soft link .qmail-mydomain:com-marcus-default and .qmail-mydomain:com-marcus to this file - your setup may well be different.

My incoming filter deals with mail in this order: virus tagged mail gets put on hold, blacklisted mail gets bounced, whitelisted mail gets allowed through, mail tagged as spam is put on hold (I trust spamassassin enough that I rarely look at the pending directory and it gets cleaned out every 30 days).

So in my incoming filter I have something like: 

# Put virus tagged mail on hold
pipe "/path/to/virusscan" hold

# bounce blacklisted email
from-file ${DATADIR}/lists/blacklist bounce

# allow whitelisted email
from-file ${DATADIR}/lists/whitelist ok
from-file ${DATADIR}/lists/whitelist-confirmed ok

# Put all spam on hold
headers "X-Spam-Status: YES.*" hold

# confirm everything else
from * confirm

Where ${DATADIR} is my .tmda directory. I set spamassassins spam level score to 5 points - this generates a few incorrect results but they end up going though the confirm process mainly anyway. I used to be clever and drop spam with scores over 10 points but I dont get much spam on my ISP account (I'm _very_ protective about my home email address). If you want to drop spams over a certain score and hold spams under that score you can use: 

# drop all spam with a level 10+

headers 'X-Spam-Level: \*\*\*\*\*\*\*\*\*\*.*' drop

# hold all spam above level 5, below 10 (header rule above deals
# with level 10 and above remember)

headers 'X-Spam-Level: \*\*\*\*\*.*' hold

# confirm all spam above level 4, below 5 (header rules above deal
# with levels 5 and above remember)

headers 'X-Spam-Level: \*\*\*\*.*' confirm

If you prefer you can use the regexp 

# 10 *'s or more
headers 'X-Spam-Level: \*\{10\}' drop
# 5 *'s or more
headers 'X-Spam-Level: \*\{5\}' hold
# 4 *'s or more
headers 'X-Spam-Level: \*\{4\}' confirm

You may need to add ".*" to the end of the regexp (after the curly brace).

Via procmail

.procmailrc

I have this in my .procmailrc: 

# If it's not going to a TMDA tagged address, 
# and it's not already been to see TMDA (because it's a confirmed message),
# then run it through spamassassin

:0 fw
* !^To: .*kyle-(exp|expires|dated|d|key|keyword|kw|cnf|confirm|c|snd|src|sender|
s)-.*@
* !^X-TMDA-
| spamassassin

# If it's to a dated address, I want to give it a chance to deliver even if
# TMDA considers the address expired.  If it's still valid, don't spam check
# it.  TMDA may act on the SA headers before it realizes it's a valid dated
# address.  That is, I want valid dated addresses to deliver even if SA thinks
# they're spam.

:0 fw
* !^X-TMDA-
* ^To: .*\/kyle-(exp|expires|dated|d)-.*@
* $!? /usr/bin/tmda-check-address '$MATCH' | grep -q '^STATUS: VALID'
| spamassassin

# There's no point in letting TMDA see these now.
:0
* ^X-Spam-Status: No
$DEFAULT

# Run the message through tmda-filter.

Before pasting that into your own config, be sure to change the login name (kyle) to your own, and use the tags from your own TMDA setup.

It's important to avoid running SpamAssassin on anything that TMDA will consider a valid tagged address. We want those messages to be delivered by TMDA regardless of what SpamAssassin thinks of them. If we add the SpamAssassin analysis, TMDA could act on that when we don't want it to.

There are a couple of "gotchas" here. One is that delivering email that SpamAssassin considers ham directly to your mailbox bypasses blacklist entries that TMDA knows about. It's possible for a blacklisted spammer to send a low scoring message and have it get to your mailbox. The other thing is that the patterns above match the To: header for tagged addresses, but email can be sent to you at those addresses without it showing up in the To: part of the header. If your MTA inserts Delivered-To: headers, it might be better to match those.

.spamassassin/user_prefs

Here are parts of my SpamAssassin user_prefs: 

# Don't fold the message into an attachment. 
report_safe 0
use_terse_report 1

# Messages -1.5 and higher are "suspicious"
required_hits -1.5

bayes_ignore_header X-TMDA-Recipient
bayes_ignore_header X-TMDA-Action
bayes_ignore_header X-TMDA-Fingerprint
bayes_ignore_header X-TMDA-Confirmed
bayes_ignore_header X-TMDA-Confirm-Done
bayes_ignore_header X-TMDA-Originally-To

The report_safe and use_terse_report settings are to accommodate confirmed messages. Normally SpamAssassin will take the original message and fold it into an attachment and create a new message that encompasses it. The problem I encountered was that the wrapper did not have a Return-Path: header, and TMDA didn't want to release it from pending when it was confirmed. I use_terse_report just so that a released message doesn't have a big ugly SpamAssassin report at the top of it after it's confirmed. I see now from the documentation that you can avoid the Return-Path: problem also using the report_safe_copy_headers option to include the Return-Path. I haven't tried this myself.

The required_hits parameter is the maximum score at which SpamAssassin considers a message ham. When I first implemented this, the SpamAssassin default (5) allowed all of SpamAssassin's false positives to get into my mailbox. I set required_hits very low (-1.5) to avoid that problem. You could also avoid the problem by eliminating the rules in .procmailrc and .tmda/filters/incoming which allow SpamAssassin-approved mail to be delivered.

The bayes_ignore_header options are used to make sure that SpamAssassin's bayesian classifier doesn't try to use those to identify messages.

.tmda/filters/incoming

Finally, the TMDA part of this setup is pretty simple. This is at the end of my FILTER_INCOMING: 

# 
# At this point, we know the message is NOT on my whitelist,
# and it's also not on any blacklist.  Procmail is set up to
# filter the message through SpamAssassin only when it's not
# to a tagged address.  Therefore, the following rules will not
# match a tagged addressed mail, and those will be evaluated
# based on their tags.
#

#                       1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
headers 'X-Spam-Level: \*\*\*\*\*\*\*\*\*\*\*' hold
headers 'X-Spam-Status: No' ok

The first rule says that a message with a spam level of 11 or more will be held without challenge. Note that the second rule is redundant since those messages were already delivered by procmail using a similar rule. It's important that this come at the end of the config so that whitelists and blacklists have a chance to deliver/drop the message first. The rule based on SpamAssassin's headers is the last thing TMDA sees before it takes its ACTION_INCOMING.

Summary

Ultimately, TMDA gets the mail meant for it (tagged addresses) and treats it on its terms. Email that SpamAssassin loves (scores under required_hits) bypasses TMDA. Everything else is subject to TMDA's white/black lists and then either held or challenged based on how much SpamAssassin hated it (X-Spam-Level).

Another procmail approach

This is much more simplistic than the previous approach.

Here's the relevant portion of my .procmailrc. It runs spamc rather than the full SA, then runs tmda, which delivers my mail: 

#spam assassin
#spamc requires spamd (/etc/init.d/spamassassin) to be running
#f = consider this a filter; w = wait for completion
#:0 fw: spamassassin.lock
:0 fw
| /usr/bin/spamc


# Run the message through tmda-filter.
:0 w
| /usr/bin/tmda-filter

# Take the exit code from TMDA.
# If the exit code indicates failure,
# exim will keep the message in its queue, even if
# it's already been delivered
EXITCODE=$?

# TMDA takes care of final delivery 
# NOTE THIS WILL LOSE ALL YOU EMAIL IF TMDA HAS NOT BEEN
# SET UP TO DELIVER YOUR EMAIL.....
DEFAULT=/dev/null

Then, in my .tmda/filters/incoming, I have the following -- after all of my whitelists and various acceptance rules : 

#from SpamAssassin
headers 'X-Spam-Flag: YES' deliver=~/mail/spam

SpamAssassin on my system sets the above flag when a message scores at least a five. Here's the relevant portion of my /etc/spamassassin/local.cf : 

required_hits 5.0

And that's it!

qmail-smtpd with tmda-inject for vpopmail users

This setup allows for only running one daemon for incoming smtp connection. It uses tmda-inject for users that authenicated and have tmda installed on there account. I find it works very well, but on the the big downsides is that qmail-smtpd must be running as the UID and GID of the vpopmail accounts. In the future I think a small C wrapper program could be have the suid set and it will then run tmda-inject. I just have not gotten around to this, as it works now, and I am lazy.

Somethings used to get this working:

qmail-smtpd run script

#!/command/execlineb
multisubstitute
{
  define limit "421 per host concurrency limit reached\\r\\n"
  backtick -n me { head -1 /var/qmail/control/me }
  backtick -n concurrencysmtp { head -1 /var/qmail/control/concurrencysmtp }
  backtick -n QMAILQUEUE { head -1 /var/qmail/control/qmailqueue }
}
export QMAILQUEUE ${QMAILQUEUE}
fdmove -c 2 1
chpst -m 8000000
tcpsvd -vv -uvpopmail -x./peers.cdb -c100 -C${concurrencysmtp} 0.0.0.0 25 /var/qmail/bin/qmail-smtpd ${me} /usr/local/vpopmail/bin/vchkpw /usr/bin/true

/var/qmail/control/qmailqueue File in the run script allows me to keep my default settings in /var/qmail/control. This File just contains: 

/var/qmail/bin/qfilter-tmda

qmail-queue processing

The File Below /var/qmail/bin/qfilter-tmda is called for with the QMAILQUEUE Patch on all incoming smtp connection. This little script will first check to see if SMTP-AUTH was use and if so it will check for that a .tmda/config file in the users vpopmail homedir. If any test fails the mail will be handed off to qmail-scanner.pl to be scanned and processed as normal. 

#!/command/execlineb
unexport QMAILQUEUE
importas WHO TCPREMOTEINFO
ifthenelse { test "X${WHO}" != "X" }
{
  backtick -n HOMEDIR { /usr/local/vpopmail/bin/vuserinfo -d ${WHO} }
  ifthenelse { test -r ${HOMEDIR}/.tmda/config }
  {
    /var/qmail/bin/qmail-qfilter /usr/local/share/tmda/bin/tmda-inject -q -c ${HOMEDIR}/.tmda/config
  }
  {
    /var/qmail/bin/qmail-scanner-queue.pl
  }
}
{
  /var/qmail/bin/qmail-scanner-queue.pl
}

Have fun........

Qmailrocks installation + TMDA

I did a complete qmailrocks installation (http://www.qmailrocks.org/) with Vpopmail, Clam Antivirus and SpamAssassin and then I installed TMDA.

Contents of my /home/vpopmail/domains/xxx.yyy/user/.tmda/filters/incoming: 

from-file /home/vpopmail/domains/xxx.yyy/user/.tmda/lists/whitelist ok
from-file /home/vpopmail/domains/xxx.yyy/user/.tmda/lists/blacklist drop
#
# Option: Deliver spamassassin spamtagged mail directly to spam folder,
# do not challenge it:
#macro SPAM_DELIVER deliver=/home/vpopmail/domains/xxx.yyy/user/Maildir/.spam/
#headers 'Subject: :SPAM:' SPAM_DELIVER
#
# Option: Deliver spamassassin spamtagged mail as usual,
# do not challenge it (MUA spam sorting rule perhaps?):
headers 'Subject: :SPAM:' ok
# Option: Deliver sa spam level <= 1.0 as usual:
headers 'X-Spam-Status:.*hits=0\..*' ok
headers 'X-Spam-Status:.*hits=1\.0.*' ok

Order of spam detection checks:

1. MTA level: RBL lookups with rblsmtpd, permanent error 553 if sender blacklisted.
2. qmail-scanner + Clam Antivirus: quarantine if virus found
3. SpamAssassin: Only tag Subject as spam if score treshold reached (never delete, reject or quarantine)
4. TMDA: check lists and follow rules, possible c/r
5. MUA: Filter mail tagged as spam to special folder
6. Wetware: Have a quick look on mails in spam folder, perhaps a look on TMDA Pending with tmda-cgi also.

Some scenarios:

1. A virus infected mail arrives: clamav takes care of it and quarantines it. No notification is sent from clamav to the sender: this address is probably spoofed anyway. This mail will not be processed by spamassassin or TMDA.

2. A clean mail arrives (no virus found) but spamassassin tags it as spam (Subject: :SPAM:): I prefer to not challenge it with TMDA, the sender address is likely spoofed anyway, and instead deliver it as usual and let the recipient deal with it in the MUA: a rule which matches the ":SPAM:" in the Subject: and moves it to a dedicated spam folder might be a good idea.

3. A clean mail arrives (no virus found) and spamassassin does not tag it as spam and the sender is TMDA whitelisted: Deliver it as usual. No other action taken.

4. A clean mail arrives (no virus found) and spamassassin does not tag it as spam and the sender is not TMDA whitelisted but spamassassin gives it a score <= 1.0: Deliver it as usual. No other action taken.

5. A clean mail arrives (no virus found) and spamassassin does not tag it as spam and the sender is not TMDA whitelisted but spamassassin gives it a score > 1.0: Finally, this is the right time for TMDA to do it's job and send a challenge to the sender :-)

I use one or two relatively safe RBL lists with rblsmtpd (few false positives). One improvement to this setup would be to check if the sender domain is valid at the MTA level (check domain for MX/A records).

Summary:

I think this is a good configuration which ensures that TMDA is pretty quiet and avoids sending challenges to spoofed sender addresses, which anti C/R people usually complain about.

Timo

2007-02-24 17:18