Spam filtering in the CS Department

Spam filtering in the CS Department

The department has implemented a spam e-mail solution that enables individuals to decide if they want spam filtering turned on or off, and to specify what level of filtering they would like. This seems like a reasonable course of action: no one is filtering your e-mail against your will; you have the opportunity to easily filter your own e-mail. This addresses the issues of academic freedom, freedom of speech, and choice.

Proofpoint: Quarantined messages

All of Computer Science's mail is first routed through OIT's mail gateways before being routed to the CS mail servers. If you suspect an email might not have been delivered, you can log into https://quarantine.oit.duke.edu/euweb/login to see if the message was quarantined.

General information on spam filtering

In the interest of giving users more control over filtering unwanted e-mail messages, the Lab Staff has installed a program called SpamAssassin This program augments the headers of incoming e-mail by adding several additional lines. It does not change the way that e-mail is delivered in any way, but the addition of these new headers gives users the opportunity to filter their incoming mail according to their own standards. These additional headers provide a score for each message, which estimates the likelihood that a particular e-mail message is spam. Below is an example of what the augmented headers look like:

X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on duke.cs.duke.edu
X-Spam-Level: **********
X-Spam-Status: Yes, score=10.0 required=8.0 tests=CMAE_1 shortcircuit=spam
    autolearn=disabled version=3.2.4

SpamAssassin uses several heuristics to determine if a message is spam. A detailed report is included in the email headers and is quite thorough. The system-wide default score, as shown in the X-Spam-Status line, is set to 8.0; this can be adjusted by the user if need be.

The Lab currently purchases spam-filtering services from Cloudmark and Trend Micro, as plug-ins for the Spamassassin system. These services use real-time black lists and other techniques to provide rapid and generally reliable scoring of e-mail messages. Generally most spam is excluded after checking with these services and no further testing is required.

The Lab Staff recommends using this system, as opposed to spam scoring systems built into mail clients. If mail is being incorrectly marked as spam, the Lab Staff can use the detailed report, as well as other logs, to diagnose the problem; if other spam filters are in use, the Lab Staff may not be able to track the problems down as easily. Instructions for setting up the most common e-mail clients to use this filtering is given below.

Default spam filtering for user accounts

As of 11-Aug-2009, the CS department is enabling spam filtering by default as part of the setup of all new user accounts. Please see the default spam filtering page for details.

Common problems

Users will occasionally find that e-mail messages to them have been erroneously marked as spam; this problem is common to all spam scoring systems. For this reason, the Lab Staff recommends that spam be moved to a separate folder, which can be deleted at regular intervals, but provides some safety in the case that an e-mail is improperly tagged as spam.

If you are experiencing repeated instances where e-mails are being improperly tagged, please contact the Lab staff for assistance. There are some commonly encountered problems to which we can offer solutions, though occasionally more complex problems occur. By following the guidelines in this document, the user can make the process of fixing the problem quicker and easier.

Additionally, users who forward their e-mail to sites such as Gmail occasionally have issues with delivery. Please see our Forwarding E-mail & Filtering Spam with Procmail page for additional suggestions regarding forwarding to such sites.

Setting up mail clients to filter

Instructions for setting up filtering for the most common e-mail clients, as well as for using procmail are listed below:

If you have any questions please contact the Lab Staff.