FilterSpecification

TMDA Filter Specification

TMDA filter files are used to control mail both coming in to and going out of TMDA. For incoming, the filter controls how the message is disposed of. For outgoing, it controls how the message is tagged. The incoming filter file default is ~/.tmda/filters/incoming, which can be changed by setting FILTER_INCOMING in your ~/.tmda/config. The outgoing filter file default is ~/.tmda/filters/outgoing, which can be changed by setting FILTER_OUTGOING.

Format of Filter Files:

A filter file is composed of filters, blank lines and comments. For ease of organization, macros, variable interpolation and the ability to include other filter files are supported. These features are documented below in the Additional Features section.

Each filter in a filter file is expected to be a string containing three fields separated by whitespace. Each filter may be placed on a single line or may be spread over several lines, to increase readability. You must follow a few rules when formatting your filters.

• Each filter must start at the beginning of a line.

• Subsequent lines that are part of the same filter must be indented by one or more spaces or tabs.

• Blank lines (either empty or containing only whitespace) or a line beginning a new filter signify the end of the previous filter.

• A line containing only a comment does not end a filter. You may intersperse comment lines within your filters (see the examples).

• Everything after '''#''' on a line is considered a comment and ignored.

• Filters with invalid syntax are logged with a specific error message into the debug log and the mail is deferred. This is to allow you the chance to fix your filter before processing the mail.

The filter file is read sequentially from top to bottom. The first filter that matches is used. The three fields in a filter are:

•       source match action

* source: specifices the source of the match.

Some sources take optional arguments. An argument begins with a dash (-). Some arguments take options. If an argument takes an option, the argument is followed immediately by an equals sign (=) and the option. No whitespace is allowed on either side of the equals sign.

Single arguments look like this:

•       -argument

  • and arguments with options look like this:

        -argument=option

Sources are documented here along with any optional arguments and the expected match field.

* match: should be an expression, or the full path to a textfile, CDB database, or DBM database containing more expressions if source was suffixed with -file', -cdb', or `-dbm'.

The match expression and the contents of the textfile vary based on the source. For an explanation of the match field expected for a given source, see FilterSources.

The second field within a textfile, CDB or DBM is optional, but overrides action if present. e.g,

•   foo@mastaler.com
    bar@mastaler.com bounce

DBM support comes with your Python interpreter, but CDB support currently requires that you first install the python-cdb extension module.

The printcdb/printdbm scripts in the contrib directory can be used to print the contents of CDB/DBM files in TMDA-list format.

* action: action specifies what action to take on the message. An optional '''=''' separates the action from the action's option. Possible values differ based on whether the message is incoming or outgoing.

(for incoming, action can be one of):

•   bounce,reject (bounce the message)
    bounce,reject=<template> (bounce the message using a specific template)
    drop,exit,stop (silently drop the message)
    ok,accept,deliver (deliver the message)
    ok,accept,deliver=<instruction> (deliver to a Maildir, mbox, program, or address)
    confirm (request confirmation for the message)
    confirm=<template> (request confirmation for the message using a specific template)
    hold (store the message in the pending queue but do not send a confirmation request)

The template option to bounce,reject and confirm can be used to bounce or confirm the message using a specific template file rather than the default (bounce.txt or confirm_request.txt). The option can be an absolute pathname to the template file (e.g, /path/to/template.txt). Pathnames starting with a tilde will be expanded. It can also be a relative pathname, in which case the template will be searched f or under TEMPLATE_DIR as described in TemplateHowto.

The instruction option to ok,accept,deliver can be used to short-circuit the default delivery method by delivering the message to a specific location. Delivery to qmail-style Maildirs, mboxrd-format mboxes,programs (pipe), and different e-mail addresses are supported.

Please note the following restrictions to mmdf, mbox and Maildir delivery:

• TMDA will not create Maildirs, mmdf or mbox files if they do not exist. You must create them prior to having TMDA deliver mail to them with commands like maildirmake and touch.

• TMDA requires write access to any Maildir, mmdf or mbox file you wish to deliver mail to.
• TMDA will not deliver to an mmdf or mbox symlink. Specify the path to the actual mbox file instead.

• Do not deliver to mmdf or mbox files located on an NFS filesystem. This is unsafe and can corrupt your mbox file -- this applies to all MDAs, not just TMDA. Use Maildir instead as it is immune to such problems.
  
  

(for outgoing, action can be one of):

•   bare (don't tag)
    bare=append (don't tag, and also add recipient to your BARE_APPEND file)
    sender (tag with a sender address based on recipient)
    sender=address (tag with a sender address based on address instead)
    domain (tag with a sender address based on recipient's domain)
    domain=address (tag with a sender address based on the domain of address instead)
    dated (tag with a dated address)
    dated=timeout_interval
    exp,explicit,as=full_address (use an explicit address)
    ext,extension=address_extension (add an extension to the address)
    kw,keyword=keyword (tag with a keyword address)
    default (take the default action specified by ACTION_OUTGOING)
    tag (tag one or more specific headers with the above actions)

For all of the outgoing actions except tag, the tag applies to both the From: (or Resent-From:) header and the envelope sender. Both will be set to the same address.

The tag action is a little more flexible, in that it can be used to tag more than one header and each header can be tagged with a different address. The tag syntax is as follows:

•   <to* | from*> <match> tag <header action> [header action] ...

Any of the to* or from* sources may be used. The <match> is explained in FilterSources. The <match> is followed by the keyword tag which is followed by a list of header/action pairs.

The header is the name of the RFC822 header field you want to tag, usually From: or Reply-To:. When you specify the header name, do not include the colon (''':'''). Use envelope to tag the envelope sender.

The action is any one of the outgoing actions from the list above (except tag, of course). That action will be used to tag the specified header only. If the specified action is not one from the list above, the action is inserted verbatim as a string. This can be used to add arbitrary headers to your outgoing messages based on destination. Quotes are required for text that includes whitespace and are unnecessary for single word strings.

The tag action is especially useful for mailing lists. Here, the envelope sender is set to your subscription address but the From: header is set to a dated address, making it easy for list members to respond to you but greatly reducing the likelihood that spammers will be able to harvest and use the dated address before it expires.

•   to tmda-users@tmda.net tag
       envelope  exp=foo@catseye.net
       from      dated=1w

In simple cases where you wish to tag the From: header and the envelope sender with the same address but tag Reply-To: differently, you may specify the From: header but not the envelope sender. TMDA will use address in the From: header as the envelope sender.

•   to closed-list@lists.com tag
       from      sender=closed-list-admin@lists.com
       reply-to  dated

This will set both the From: header and the envelope sender to the same address (a sender address) based on the envelope sender of the list. Reply-To: will be dated to allow list members to reply to you.

Additional Features:

* Macros

A macro is, in its simplest form, a piece of text that is to be substituted for another piece of text, rather like a #define in C. Here's how to define a simple macro in a filter file.

•   macro INBOX ~/Mail/Inbox

Here's how to use that macro in a filter rule.

•   to foo@catseye.net deliver=INBOX

Before the filter parser attempts to parse a rule it performs a macro substitution pass on the text. The result of that substitution on the a bove example filter is this.

•   to foo@catseye.net deliver=~/Mail/Inbox

After the substitution is performed, the rule is parsed normally and any syntax errors are identified and logged as usual.

Macros may also take parameters. For example, assume that you always deliver list mail to a unique maildir per list. Here's a macro to help with that.

•   macro LIST_DELIVER(m) deliver=~/Mail/m/

Here's how you might use it for the two TMDA mailing lists.

•   from tmda-users-bounces@tmda.net   LIST_DELIVER(TMDA-Users)
    from tmda-workers-bounces@tmda.net LIST_DELIVER(TMDA-Workers)

Expanding the above macros results in these rules.

•   from tmda-users-bounces@tmda.net   deliver=~/Mail/TMDA-Users/
    from tmda-workers-bounces@tmda.net deliver=~/Mail/TMDA-Workers/

Macros may refer to other macros but may not refer to themselves, even indirectly. In other words, if macro A refers to macro B, B may not refer to A. This will result in a parser syntax error.

Macros are not case-sensitive. Use case as you prefer to help clarify your filter files.

* Included Filters

Filter files may include the contents of other filter files. The included filter rules are placed inline right where they were included. This positioning is important to remember, since the first matching rule encountered is the one that will be applied to an incoming email.

A filter include statement looks like this.

•   include [ -optional ] /path/to/included/filter_file

If the -optional flag is specified, the parser will not raise an error if the file is missing. If you expect that the file will always be present and its absence would be an error, do not specify this flag! That way, the parser will log an error and defer the mail, so that you can fix the problem.

Filter includes may be nested; that is, the included filter file may include other filter files itself, and those files may also include other files. If a filter file tries to include one of the files that included it, however, an error will be logged and the mail will be deferred.

* Variable Interpolation

Filter files support variable interpolation or substitution, like the shell. A variable is recognized if it begins with a dollar sign and a left curly brace ('''${''') and ends with a right curly brace ('''}'''), like this: ${TMDADIR}.

A variable name is looked up first in the Defaults namespace (all of the configuration variables set in Defaults.py or your ~/.tmda/config) and then in the environment if it was not found in the Defaults namespace. If it is still not found, the parser generates an error and the mail is deferred. Variable names are case-sensitive.

Using variables may allow you to create a system-wide filter that users can modify by providing their own additions. A Python variable, such as 'username', would need to be set in the user's configuration file, like this.

•   username = "foo"

Here's an extremely simple system-wide filter that uses the 'username' variable to include a user's filter.

•   from-file /etc/company_blacklist drop
    from-file /etc/company_whitelist accept
    include -optional /home/vdomain/tmda/lists/whitelist.${username} 

If the 'foo' user chooses to create a whitelist and names it "whitelist.foo", it will automatically be included. Because of the use of the -o ptional flag, the file is not required.

Example Incoming Filter:

•   ### ~/.tmda/filters/incoming (first match wins) ###
    # Accept all bounces (messages with an empty envelope sender)
    from <> ok
    # Accept all messages to postmistress
    to postmistress@* accept
    # Bounce all messages from badboy.dom
    from *@=badboy.dom bounce
    # Accept all messages from mycorp.dom
    from *@=mycorp.dom ok
    # Include my blacklist and whitelist
    from-dbm ~/.tmda/lists/blacklist.db drop
    from-cdb ~/.tmda/lists/whitelist.cdb accept
    from-file -autodbm ~/.tmda/lists/nastygrams bounce
    from-file -autocdb ~/.tmda/lists/confirmed ok
    from-file ~/.tmda/lists/whitelist_wildcards accept
    # Mailman mailing list subscribers and digest subscribers
    from-mailman -attr=members ~mailman/lists/viewnet-news ok
    from-mailman -attr=digest_members ~mailman/lists/viewnet-news ok
    # ezmlm mailing list subscribers and digest subscribers
    from-ezmlm ~alias/all-acl-users ok
    from-ezmlm ~alias/all-acl-users/digest ok
    # Revoked addresses
    to jason-stupid_promo.289076@mastaler.com bounce
    to jason-jcrew.832234@mastaler.com confirm
    # Examine the message content
    body "viagra|ginseng" confirm
    headers 'Precedence:.*junk' reject
    headers -case 'MAKE MONEY FAST' drop
    # Accept all messages smaller than 10KB, but drop messages larger than 1MB
    size < 10000 deliver
    size > 1000000 exit
    # Throw the message away if Razor says it's spam
    pipe /usr/bin/razor-check drop
    # Save the message to a special mbox if Razor says it's spam
    pipe "/usr/bin/razor-check -rs=razor.server.com" deliver=~/mbox.spam

Example Outgoing Filter:

•   #### ~/.tmda/filters/outgoing (first match wins) ###
    # All mail from postmaster is sent bare
    from postmaster@* bare
    # And so is all mail from the tmda-users mailing list owner
    to-mailman -attr=owner ~mailman/lists/tmda-users bare
    # All whitelisted contacts receive untagged messages
    to-cdb ~/.tmda/lists/whitelist.cdb bare
    to-file ~/.tmda/lists/whitelist_wildcards bare
    # Keyword Addresses
    to *@myisp.net kw=myisp
    to king@grassland.com keyword=elvis_parsley
    # Dated addresses (some with a non-default timeout interval)
    to bobby@peru.com dated
    to-dbm /var/dbm/slowpokes.db dated=6M
    # Allow anyone at whitehouse.gov to reply
    to president@whitehouse.gov domain
    # Majordomo and Mailman check the From: header for membership
    to mutt-users@mutt.org tag
       from      sender=owner-mutt-users@mutt.org
       reply-to  dated
    # ezmlm checks the envelope sender address for membership
    to tmda-admin@samstech.net tag
       # Set envelope sender to the extension address I subscribed with
       # and From: to 'default'.  TMDA comes with the default action
       # (ACTION_OUTGOING) set to 'dated'.  Reply-To: isn't necessary in
       # this case.
       envelope  extension=mlists-tmda-admin
       from      default
    # Add some arbitrary headers.
    to foo@bar.com tag organization "Whatwerks.com, Inc."
    to foo@bar.com tag
       from             sender
       reply-to         dated
       organization     "Disney Land"
       x-favorite-dwarf Hungry
    # Use a different username and/or domain
    to *@gnus.org exp=jason@gnus.org
    to xemacs-binary-kits* explicit=binkit-manager@XEmacs.ORG
    to *mail*@=xemacs.org as=postmaster@XEmacs.ORG
    to *@=xemacs.org as=jasonrm@xemacs.org

A set of starter filter files are included in the TMDA distribution (tarball) within the contrib/dot-tmda subdirectory.