TmdaTipsAndTricks

TMDA Tips & Tricks

Contents

Rejecting mail with non-ASCII headers (i.e, Asian spam)
List of spammer domains who "auto-confirm"
Supporting the Habeas SWE headers
Rotating TMDA logs
Blocking SPAM where sender equals recipient (Exim 4.x)
Using SMTP server instead of tmda-ofmipd for non-Unix MUAs mail sending (Exim 4.x)
How to improve performance (for high volume sites)
X-TMDA: and Mozilla Thunderbird

Rejecting mail with non-ASCII headers (i.e, Asian spam)

You'll find that quite a bit of spam originating from Asian countries like Korea and Taiwan is sent with raw 8-bit data in the headers.

This is a clear violation of RFC 2821, section 2.3.1. 8-bit data in headers must be properly ASCII encoded according to RFC 2047. Thus, it's a good way to weed out some obvious spam, as properly encoded mail containing Asian character sets should not be affected by this.

You can reject mail with non-ASCII data in the headers using the following FILTER_INCOMING entry:

# Reject mail with non-ASCII text (byte values > 127) in the Subject:
# or From: headers.
headers '^(subject|from)\s*:.*[^\x00-\x7f]{5,}' bounce

I've found it catches most of my .kr, .tw, and .ru spam, with 0 false positives.

This regex can also be used at the MTA level; in fact I got it from a Postfix user who using it this way.

List of spammer domains who "auto-confirm"

Currently, there are no known spammers who actively try and "auto-confirm" TMDA confirmation requests, but in a roundabout way, this sometimes happens.

Sometimes non-RFC-compliant mail software returns undeliverable messages to the Reply-To address instead of the envelope sender address. When spammers forge messages from such servers, TMDA will send a confirmation request back, and the server will respond to the confirmation address in Reply-To, and the spam will be released to you. See FAQ 1.13 for some workarounds.

We are no longer providing an actual list of domains, as there are too many, and it's unfair to keep a domain listed that might have corrected their practices.

One interesting idea would be to maintain an RBL (real-time blackhole list) service listing domains who auto-confirm. This would provide a more reliable and dynamic way for TMDA users to reduce the likelihood of automated confirmations. At this point, it's probably overkill, but something to keep in mind if challenge/response starts to become more popular.

Supporting the Habeas SWE headers

Habeas provides a creative way to "prove" that an e-mail message is NOT spam by including a copyrighted haiku poem in the headers of an outbound message. You must agree to a license agreement and acceptable use policy before you can use the Habeas SWE headers in a message, so the idea is that unauthorized users (i.e, spammers) will not use them due to fear of legal repercussions.

It's not clear yet whether this strategy will be able to effectively resist abuse. Spamming is an inherently illegal activity, so abiding by U.S. copyright/trademark law may not be a concern for many spammers. Issues such as international litigation against overseas spammers and difficulty in tracking the true origin of spammers are additional impediments.

Theoretically, it would provide a nice way for the TMDA user to reduce the number of confirmation requests he sends out to unknown correspondents. Of course, this assumes the system won't be abused, and that a large number of users adopt Habeas.

If you'd like to unconditionally accept messages that contain the Habeas SWE headers, you can can use the following entry in your FILTER_INCOMING file:

# Accept mail containing the Habeas SWE headers
headers '^X-Habeas-SWE-3.*like Habeas SWE \(tm\)' ok

If you wish to add the Habeas SWE headers to your own outgoing messages, you can do so with ADDED_HEADERS_CLIENT and ADDED_HEADERS_SERVER entries. Make sure to fill out the Habeas license agreement first though. For example,

ADDED_HEADERS_CLIENT = {
    # The Habeas SWE headers  
    'X-Habeas-SWE-1' : 'winter into spring',
    'X-Habeas-SWE-2' : 'brightly anticipated',
    'X-Habeas-SWE-3' : 'like Habeas SWE (tm)',
    'X-Habeas-SWE-4' : 'Copyright 2002 Habeas (tm)',
    'X-Habeas-SWE-5' : 'Sender Warranted Email (SWE) (tm). The sender of this',
    'X-Habeas-SWE-6' : 'email in exchange for a license for this Habeas',
    'X-Habeas-SWE-7' : 'warrant mark warrants that this is a Habeas Compliant',
    'X-Habeas-SWE-8' : 'Message (HCM) and not spam. Please report use of this',
    'X-Habeas-SWE-9' : 'mark in spam to <http://www.habeas.com/report/>.'
    }
ADDED_HEADERS_SERVER = ADDED_HEADERS_CLIENT

Rotating TMDA logs

Here are some tips for rotating your TMDA logs.

Here's a solution that should work everywhere TMDA does, but grows indefinitely in size because old logs are never deleted. Add the following to your .tmda/conf file:

import os, time
homedir = os.path.expanduser("~")
YYYYmm = time.strftime("%Y-%m")         # 2001-04

LOGFILE_DEBUG = os.path.join(homedir, "log/tmda", "debug." + YYYYmm)
LOGFILE_INCOMING = os.path.join(homedir, "log/tmda", "incoming." + YYYYmm)

The end result looks like this:

-rw-------  1 jason  wheel  1101449 Dec 31 23:58 incoming.2002-12
-rw-------  1 jason  wheel  1394177 Jan 31 22:51 incoming.2003-01
-rw-------  1 jason  wheel  1172741 Feb 24 13:47 incoming.2003-02

Here's a more powerful log rotation scheme that is specific to FreeBSD. Something similar should work on Linux using the "logrotate" program.

1. Create your own "newsyslog.conf" configuration file. In my case, I made it:

/home/mark/etc/newsyslog.conf

The contents look like this:

# logfilename              mode count size when [ZB]
/home/mark/.tmda/log-in    600  3     100  *     Z
/home/mark/.tmda/log-out   600  3     100  *     Z

2. Then I created a cron entry to run every hour, using "crontab -e" to edit my personal crontab file:

# Rotate TMDA logs
13 * * * * newsyslog -r -f /home/mark/etc/newsyslog.conf

If you have other personal log files you want to rotate, you can simply add entries to you personal "newsyslog.conf" file for them.

Here's one way to use logrotate on linux:

1. Set up your configuration file.

I put mine in

 /home/monique/.logrotate.conf

Here's an example configuration. Use the same format for the debug and outgoing logs.

# global: create new (empty) log files after rotating old ones create

# rotates log into a gzipped file every week.
# logs from the most recent five weeks are kept;
# older ones are purged.
/home/monique/log/tmda/incoming {
   weekly
   rotate 5
   compress
}

2. Create a logrotate state file specific to your user. The logrotate program will populate it for you. This will contain a list of the log files you're rotating and the date they were last checked.

touch /home/monique/log/logrotate.state

3. Use crontab -e to schedule logrotate

@daily /usr/sbin/logrotate --state /home/monique/log/logrotate.state   /home/monique/.logrotate.conf

Once the cron job has run, you should see something similar to this:

monique-home:~$ ls log/tmda/
debug  debug.1.gz  incoming  incoming.1.gz  outgoing  outgoing.1.gz

Blocking SPAM where sender equals recipient (Exim 4.x)

The Exim SMTP server introduced ACL's (Access Control Lists) in version 4.x. These provide a very flexible way to control, and conditionally reject, unwanted mail and/or SMTP communications. Exim ACL's can check all stages of the SMTP session, for example - connect, auth, data and the one we will use here - rcpt (recipient).

This simple ACL allows the blocking of emails where the sender address is the same as the recipient address, a trick used by spammers to get past filters. The logic checks if the sending address is the same as the recipient address and the sending host IP address is not in the list of allowed relaying hosts. In a simple MTA installation the "relay_from_hosts" will usually just include the internal non-Internet routable address block and localhost.

The "relay_from_hosts" entry is defined in the main section of the Exim configuration file.

# Main Section:

hostlist relay_from_hosts = 192.168.1.0/24 : 127.0.0.1

The ACL is placed in the "acl_check_rcpt" part of the ACL section of the config file. The condition uses regular expressions to return true if the sender address = recipient address then checks if the sender's host IP address is not in the list of allowed relay hosts.

# ACL Section:

acl_check_rcpt:

# Block spoofed addresses

  deny  condition       = ${if eq {$sender_address}{$local_part@$domain}{yes}{no}}
        hosts           = !+relay_from_hosts
        message         = Spammers not welcome - go away.
        log_message     = Spam from sender $sender_address at $sender_fullhost

The condition can also be changed to check addresses on the domain level if you know that addresses coming into the mail domain from the outside should never have the internal mail domain as their sending domain.

${if eq {$sender_address_domain}{$domain}{yes}{no}}

If the condition is true the ACL will reject the RCPT with a 550 permanent error. You can define different messages to be returned to the sender and written to the Exim log files.

Using SMTP server instead of tmda-ofmipd for non-Unix MUAs mail sending (Exim 4.x)

Introduction

If you have Windows OS client MUA's and you want to take advantage of TMDA's outgoing tagging features without having the extra setup of tmda-ofmipd -

need to have extra tofmipd daemon running on Unix/Linux server
need to setup authenticated accounts for tofmipd on Unix/Linux server
changing the SMTP port and authentication configuration on each client machine

- this can be achieved if your MTA is Exim 4.x by having Exim work directly with the tmda-inject module.

Setup

The tmda-sendmail and tmda-ofmipd modules both call tmda-inject which does the work of checking user's tmda settings, including whitelist files, before rewriting the sending email address if necessary. The method described below has Exim cycle outgoing mail directly through tmda-inject which then sends it back to Exim for final delivery.

Exim uses "routers" to decide on delivery path for email and "transports" to physically deliver mail to the next stage of the mail handling process. The order in which routers are processed is important, as the first router to match an email address will process the email. As we want to 'catch' any outgoing emails and send them to tmda-inject before they are processed by the usual outgoing router (dnslookup) we place the new router called 'outgoing_tmda' before the 'dnslookup' router. The router is:

# Outgoing tmda router - sends all first run of outgoing mail for tmda users to tmda
outgoing_tmda:
    driver = accept
    senders = name@domainname.com : name2@domainname.com : etc....
    domains = !+local_domains
    condition = "${if !def:header_X-Delivery-Agent:{1}{0}}"
    transport = outgoing_tmda_pipe

The details are:

driver - basically says accept the email if the conditions are met. Other driver types for Exim transports can be - redirect, dnslookup etc.
senders - the plain email address of the sender i.e. before tagging by tmda. The senders lookup can be anything supported by Exim - a colon separated list (adequate for small sites) all the way up to an SQL database lookup. This is necessary for situations where you may have a site with tmda and non-tmda users.
domains - "!+local_domains" means where destination domains is not inside domain
condition - looks to see if the tmda header "X-Delivery-Agent:" is not present in the mail headers. This will be the case on the first pass of the mail message through Exim. Note the double quotes around the condition.
transport = says which physical transport will handle the email

Then the mail will go to the defined transport "outgoing_tmda_pipe". The key consideration with the transport is that it must be setup to function in the context of the user sending out the email. If the context does not match the user it will not work. Now this may differ in various sites so I will list my environment -

Redhat 7.x;
Python 2.x installed on top of default Python 1.5 installed by Redhat 7.x;
user directories under /home;
user's tmda settings and crypt key in /home/user/.tmda;
all users using tmda added to OS group 'tmda'.

To ensure that the message is not expanded resulting in multiple deliveries where there are multiple recipients you must use Exim's batching facility (see "Address batching in local transports" in Exim spec document) as is shown in the transport below.

# Outgoing tmda transport - pipes email in batch to tmda-inject
outgoing_tmda_pipe:
    driver = pipe
    batch_max = 1000
    home_directory = /home/$sender_address_local_part
    command = /usr/bin/python2 /usr/bin/tmda-inject $pipe_addresses
    user = $sender_address_local_part
    group = tmda

The details are:

driver - pipe command obviously!
batch_max = 1000 - as we piping the email to tmda we don't want Exim to run multiple instances of the transport for each address otherwise the message will be sent x n for n recipients.
home_directory - /home/ + Exim expansion variable which is the local part (before the @ sign) of the sending email address.
command - the python2 executable is specified, the tmda command is tmda-inject, the Exim variable $pipe_addresses tells Exim that this email is being piped, not actually finally delivered, so batch all recipient addresses.
user - which will be the OS user, taken again from the local part of the user address. Obviously this must match, so if the sending address is - fred@bloggs.com then the OS user name must be 'fred'.
group - OS group for all tmda users.

Testing

You should test the functioning of this before going live, I recommend doing a debug test first to follow through the processing of the email then do a live test. Exim provides many command line testing options (see Exim spec.txt), you need the following command:

[root@server bin]# ./exim -d -bs

type in the SMTP commands manually - internal sender, external recipient and a simple data line then quit. Things to check:

The router accepts the email and sends to correct transport
The uid/gid is changed to sending user before calling pipe command
The tmda-inject program accepts the email successfully - "outgoing_tmda_pipe transport yielded 0"

In the debug output you should see something like this:

    outgoing_tmda_pipe transport returned OK for recipient@domain.com
    post-process recipient@domain.com (0)
    recipient@domain.com delivered
    LOG: MAIN
     => recipient <recipient@domain.com> R=outgoing_tmda T=outgoing_tmda_pipe

Then you can run a full test, if you are using Eximon you can watch the results live. One thing which may confuse you - you will see 'incoming' messages first then the various 'outgoing' messages. Just refer to the message ID's to see the flow. If the recipient is not on your whitelist you will (depending on your tmda setup) see your email address be transformed from your bare address to your tmda tagged address - in my case a14d dated address.

How to improve performance (for high volume sites)

TMDA generally performs adequately, but high volume sites might need to tune their configurations according to the suggestions below. This particular list was written for an ISP which processes ~50,000 messages/day with TMDA. Even if you don't run a high volume installation, you may find one or more of these suggestions helpful.

This list assumes that the reader is already familiar with TMDA and how it works.

Avoid doing content filtering via 'headers' and 'body' rules. If you need to do content filtering, use another tool which is more optimized for this task such as maildrop.
Upgrade your Python to the latest available release (2.5 at current writing). Besides being more stable, Python 2.5 can be up to 30% faster than previous versions of Python.
Upgrade your TMDA to the latest available release (1.1.x at current writing). TMDA's performance is constantly improving.
Set CONFIRM_ACCEPT_NOTIFY = False in /etc/tmdarc to greatly reduce the number of messages TMDA sends out.
Don't use any wildcards (e.g, *@foo.com) in your list files, but only plain e-mail addresses (e.g, jason@foo.com) or domains (e.g, foo.com, mail.foo.com). The wildcard code is very expensive and isn't compatible with -autocdb or -autodbm. In most cases you can accomplish what you want using plain domains anyway.
Use -autocdb or -autodbm with all flat file lists. -autocdb is probably better, but requires you install the python-cdb extension module. -autodbm works out of the box.
Disable all logging if you have it enabled (TMDA does no logging by default).
If you are really looking to optimize, consider setting values in /etc/tmdarc for TMDA variables whose defaults are "expensive" to calculate. The best way to do this is to page through the TMDA/Defaults.py file and look at how the various default values are calculated. Some have negligable expense, others not. For example, `SENDMAIL_PROGRAM' is a good one to hardcode as it saves at least one stat() for each incoming message.

X-TMDA: and Mozilla Thunderbird

Now you too can add an X-TMDA to Mozilla Thunderbird on a per-message basis, just like the pros:

Find and edit your user.js file in your profile, and add the following line:
```
user_pref("mail.compose.other.header", "X-TMDA");
```
Shutdown Thunderbird and restart it. The next time you compose an email and want to add an X-TMDA header, click on a new line in the recipient area, and click on the down-arrow beside "To". Choose "X-TMDA" and put in the setting you need.