Forums & Blog

I am writing a VBS script that checks the message body URIs against URIBL lists. It offers the following checks/functions:

- URIBL (Hosts + Domains + NS) with advanced URL decoding
- Body Charset
- Body Spamwords, supporting Regex
- ServerIP RBL
- ServerIP RDNS
- Country check for ServerIP + URI Hosts
- Bad format checks (No From/Subject, Extremely Short message or Body, etc.)
- Inline image check
- Run a Command Line virus scanner (which should Clean or delete the message)
- Backup/Archive all email in a folder yyyymmdd. Includes both EML and HDR files for easy resending: just copy them in the SM Spool folder.
- Cache "clean" domains (score=0) to a file to save resources (skip subsequent URI checks for these domains)
- Log "clean" mailserver IP's to a file: handy to automatically update Smartermail's greylisting bypass file greyListBypass.xml with clean IP's

Actions:
- Add a X-header (e.g. X-SpamFilter-Score: HIGH)
- Add a string to the subject (e.g. [SPAM-HIGH])
- Delete if Score>n (=move to Deleted folder)

Specific requirements:
- CDO Installed (used to decode the body)
- Free TCPIP control for (R)DNS lookup is included

All the checks can be enabled/disabled, and many limits can be set to save resources (MaxMsgSize, Maxlookups, ExitOnHighScore, IgnoreDomains). I know that some of the additional functions are duplicating SM and  SA, but I prefer them to be in one (global) place and even URIBL alone is worth it! It works from the Command line (Spool settings) and works fine on my small mail server. I would not recommend it for high-load servers, but it seems to be pretty efficient so far.

I use the following RBL blacklists to block connections at smtp level (also see http://www.dnsbl.com for some useful info):
- zen.spamhaus.org
- bl.spamcop.net
- psbl.surriel.com

The latest build can always be found here (updated 20/11/08):
http://www.iisworks.com/spamfilter/spamfilter.zip

Installation
- Requirements: Microsoft CDO (present on every Windows machine), ActiveX DNS control (Run: regsvr32 <FullPath>fathdns.dll)

- Unzip the files, regsvr32 fathdns.dll, check all settings in the script and wordlist. Then add the vbs command line to the Spool command line option in SM: "D:\Spamfilter\spamfilter.vbs %filepath".
Make sure to set the script timeout long enough, because multiple dns lookups may take a while (recommended: 30s)!

- To hook it in SM you need to setup a "Custom headers" spam filter rule:
Check for header "X-SpamFilter-Level", which can have the values LOW, MED or HIGH. Assign the appropriate weight to each value (e.g. 10, 20, 30).

DISCLAIMER Test the script well before using it in a production environment (there is a potential to delete valid email, use the backup function)!
The script is provided for free without any installation support. Feedback, bug reports and feature suggestions are welcome at jh@iisworks.com

########################################
To get an idea, here are the settings of the script.
########################################

'### User specific settings
SpamCharsetList="windows-1254, iso-8859-3, gb2312, iso-8859-9, BIG-5, iso-2022-jp" '### Score Charset occurrences in the email body and header
CountryList="CN, KR, JP, TW, BR, TR, CL, PL" '### Format: "CountryCode=Weight". Use a negative weight if you want to downweigh certain countries"
ForbiddenAttachmentsList= ".cmd, .bat, .scr, .pif" '### Check if any of the attachments matches these strings. Example: ".cmd, .bat, .pdf, spam.gif"

'### Blacklist Checks
EnableURIDomainCheck=True '### Check if a URI domain is listed in URIBL
EnableURI_IPCheck=True '### Check if URI IP is listed in a RBL blacklist
EnableURICountryCheck=True '### Also check the country for the IP of URI hosts. Note: Only performed if EnableURI_IPCheck is enabled.
EnableMailserverRDNSCheck=True '### Check if the Mailserver IP can be resolved to a name.
'EnableURI_NSCheck=True '### Check if the Nameservers of the URI Domain are listed in a RBL blacklist. ATTENTION! This check can be relative slow!
'EnableMailserverRBLCheck=True '### Check the Mailserver IP in RBL list. ATTENTION! This is usually done by the receiving mailserver at SMTP connection level!
EnableOriginatingIPRBLCheck=True '### Check if the Originating address (Originating IP or First Received-Hop) is blacklisted in URI_IP_BLList. ATTENTION! This may cause false positives (e.g. when sending mail from a blacklisted internet cafe). These headers may also These headers may be forged and they be forged.

'### Content checks
EnableWordsCheck=True '### Enable spam keyword matching
EnableCharsetCheck=True '### Detect Characterset definitions in the email body and header
EnableCountryCheck=True '### Check if a host IP is located in a specific country (uses zz.countries.nerd.dk)
EnableBadFormatCheck=True '### Check if message looks invalid (No TO and FROM, Message or Body too small)
EnableInlineImageCheck=True '### Check if the message contains an inline image
'EnableAttachmentsCheck=True '### Enable attachment scoring based on type/name

'### Additional functions
'EnableRBLTest=True '### Test MailserverIP on a list of RBL's and log results. Note: This requires additional DNS lookups, but it will not affect the spamscore. handy to see which RBL's are effective in your situation.
EnableBackup=True '### Create a backup/archive of all messages to a subfolder /Backup/yyyymmdd
BackupHDRFiles=True '### Also backup Smartermail HDR files if they exist
BackupTreshold=4 '### Only backup a message if the score is lower than this setting. Comment out this value to backup all messages.
'EnableAntivirus=True '### Run command line for Antivirus program. Make sure this command line deletes or cleans the email file.
AntiVirusCommandLine="C:\Progra~1\ESET\Nod32.exe /log- /selfcheck- /sound- /list- /quit+ /scanmem- /scanboot- /scanmbr- /heur+ /ah /pack+ /sfx+ /arch+ /adware /unsafe /quarantine /delete"
'EnableDelete=True '### Move spam message to subfolder /Deleted/yyyymmdd. This wil prevent it from being delivered.
DeleteTreshold=20 '### Only move/delete messages if the score is higher than this value

'### Tresholds
ExitOnHighTreshold=True '### Stop all further processing when the High Treshold has been reached
LowTreshold=3 '### Minimal score for a header LOW to be added
MediumTreshold=10 '### Minimal score for a header MEDIUM to be added
HighTreshold=20 '### Minimal score for a header HIGH to be added

'### Weights added for each check
URIWeight=10 '### Score added for each URIBL related match.
MailserverRBLWeight=10 '### Score added for each Mailserver IP RBL match.
OriginatingRBLWeight=2 '### Score added for each originating/Sending IP RBL match.
DefaultWordWeight=2 '### Default score added for each spam word found
DefaultCountryWeight=2 '### Default score added for each IP that is found in the countrylist
CharsetWeight=3 '### Score added for each Characterset match.
BadFormatWeight=3 '### Score added for each Badly-Formatted message check
NoRDNSWeight=2 '#### Score added if the Sending server's IP does not resolve
InlineImageWeight=2 '### Score added if an inline image is found
ForbiddenAttachmentsWeigth=2 '### Score added for each attachment match

'### Block Lists to use
URIBLList="multi.surbl.org, black.uribl.com" '### Comma separated list of URIBL blacklists to check
URI_IP_BLList="sbl-xbl.spamhaus.org" '### Comma separated list of RBL blacklists to check the URI IP's
RBLList="zen.spamhaus.org" '### Comma separated list of URIBL blacklists to check for Sending IP
RBLTestList="zen.spamhaus.org, bl.spamcop.net, dnsbl-1.uceprotect.net, psbl.surriel.com" '### Test MailserverIP on these RBL lists

'### Advanced Settings
DNSServer="" '### DNS server to use (Empty=System default)
Loglevel=3 '### Log level: 1=Normal, 2=More, 3=Max, 4=Debug
RewriteMessage=True '### If enabled, the message will be rewritten with the added spam score header. If disabled, the original email will be left alone (=log only).
StripHTMLFromBody=True '### Remove small HTML tags before matching keywords. This will cleanup strings like "Via<span>gra".
MaxURILookups=5 '### Limit the number of lookups of URI domains found in the body. Found URI's will be randomized before lookups are performed.
'LogCleanIP=True '### Log mailserver IP to CleanIPList if TotalScore=0. This list can be used for whitelisting or for Greylisting bypass (see CleanIPList parameter below).
CacheCleanDomains=True '### Automatically add clean URI domains to CleanDomainCache file if TotalScore=0. Note: This helps performance, but has a potential to allow domains that should be blocked.
MaxCacheItems=250 '### Maximum number of clean domain entries to cache
'Resolve_IP_URI=True '### Resolve IP URI's to host and test the parent domain (Note: takes a lot of lookups and may take a long time).
IncludeFromEmailDomain=True '### Include the domain in the FROM or Reply-To email address in URI checks. Note: From-addresses are forged in most spam, but there is not a big risk of false positives, since normal users will never use a spam domain.
MaxWordCounts=2 '### Limit number of word counts for each spamword found (e.g. if 10 occurences are present, only 3 are counted)
MaxMessageSizeKB=250 '### Only process message files smaller than this value
MaxProcessSizeKB=16 '### Process only the first x kB of the email body if it's bigger.
MaxDNSLookups=25 '### Stop further processing after x DNS lookups
ScriptTimeout=20 '### Exit script after approximately this amount of time (s).
SpamSubjectPrefix="[SPAM-%Level%]" '### Add this string to the subject if message is not clean.
SpamHeaderPrefix="X-SpamFilter"
NonzeroSpamSubjectPrefix="[+]" 'Add this prefix to the subject if TotalScore>0
BadFormatMinMessageSize=400 '### Score if the raw message is smaller than this value
BadFormatMinTotalBodySize=50 '### Score if the TXT + HTML body size is smaller than this value

'### Files and paths
LogFolder="Logs" '### Folder where logfiles are placed.
BackupFolder="Backup"
DeleteFolder="Deleted"
CleanDomainCache="CleanDom.tmp" '### Skip URI checking of domains in this file.
CleanIPList="cleanip.dat" '### Log clean IP's to this list. Enter the full path to Smartermail's greyListBypass.xml file if you want to update the greylisting bypass list automatically.
WordList="spamwords.dat" '### File containing spam keywords
WhiteList="whitelist.dat" '### File containing whitelisted terms (all spam checking is skipped if any entry is found in the email)
RBLTestLog="RBLTest.log" '### Log used for RBL test results

I have updated the script to include URI NS checking as well.
It also includes a new ActiveX DNS control that does efficient name resolving.

 

Added:

- Improved URL decoding for Obfuscated links
- Check for inline images
- Check for bad message format
- Improved Domain handling
- Optional caching of "Clean" domain names to save performance
- Fix for Unicode headers

Update: small fixes.

I have now been running the script for a few months without SA or Greylisting, and the results are still fine (only RBL + this script).

I have had some requests for the script from others, and if more people are interested I will put it on-line for free download.

I am currently using invuribl. How do you compare it.

InvURIBL is limited to URI checking and it does that very well.

However, the script has some additional tricks I was also missing in SM (see first posting), and it does them globally. For example I wanted an easy and quick way to do keyword and charset matching.

I am testing Jan-Hein's script and it works great! It can be easily finetuned to yor needs and it catches SPAM much better than the standard SA.
there were no false positives in the testing period, which is great.
All the checks in the script should be implemented in SM so we can use a gui config with it.

It's tested on SM4 and SM5 running on Win2003 and Win2008 with Plesk 8.3.

Big thank you to Jan-Hein who also has some other interesting scripts at his site: http://www.iisworks.com

 Bastiaan

I decided to put the script up for download for everyone who wants to give it a try. 

The latest build can be found here:
http://www.iisworks.com/spamfilter/spamfilter.zip

Installation
- Requirements: Microsoft CDO (present on every Windows machine), ActiveX DNS control (Run: regsvr32 <FullPath>fathdns.dll)

- Unzip the files, regsvr32 fathdns.dll, check all settings in the script and wordlist. Then add the vbs command line to the Spool command line option in SM: D:\Server\Scripts\Spamfilter\spamfilter.vbs %filepath

- To hook it in SM you need to setup a "Custom headers" spam filter rule:
Check for header "X-SpamFilter-Level", which can have the values LOW, MED or HIGH. Assign the appropriate weight to each value (e.g. 10, 20, 30).

DISCLAIMER Test it well before using it in a production environment (there is a potential to delete valid email, use the backup function)!
The script is provided for free without any installation support. Feedback, bug reports and feature suggestions are welcome though: jh@iisworks.com

 I am trying to use your script, for the past couple of weeks. I didn't even realise it's not working, but today I saw a permission denied error box popup while I was remoted into my server.

How can I make sure the script it being run by the proper user?

 

Thanks

Steve

Always test it manually first (Run something like SpamFilter.vbs c:\temp\test.eml) and see if it finishes without errors (make sure to regsvr32 the Fathdns.DLL). It should create a logfile and add headers to teh original EML file as configured.

The script will need NTFS write permissions to the logfile folder and the place where the spool resides. This is usually the case, since it will run under the context of the SM service (SYSTEM).

Update 12/04/08:

Added a Spamreason header that gives a summary of the reasons why an email was marked as spam for a quick analysis. Example:
X-SpamFilter-Reason:  WORD CNTRY RDNS URI

Very nice work! I´m interested to know if the capabilities on the RegEx component you have used are good to use frank´s excellent keyword filter (http://1wit.com/ORF) we have used with ORF software.

It looks like the expressions used there are much more sophisticated. My intention with this script was to not rely on extensive sets of definitions, but provide an easy way to add a short list of obvious spam terms that can easily be edited and weighed on a global level. It's just one of the script's functions, but not the most important...

I like RBL's and URIBL's because they are maintained by specialized and reliable services and require no tweaking. :o)

I have just implemented Jan-Hein's script and it works really well cutting spam down across all domains significantly.  Can recommend to anyone.  Many thanks Jan-Hein.

Thx

Mark

You're welcome. It's nice to hear it's useful to others as well!

BTW: If someone gets spam that in their opinion should have been detected (please make sure to check the logfile first to see what happened): just zip the raw messages and send them to me for analysis.