Spam Filtering

Junk mail – it’s irritating enough when it comes through your door, but it’s just as frustrating when it clutters up your e-mail inbox too. Sadly, there’s no one-stop solution for dealing with spam – it’s a constant war between the spammers and the spammed. As we deal with one wave of spam, the next wave is already being set up.

As we’ve mentioned in a previous post, whilst spam coming in to the University is irritating, it’s a disaster if we send it out. That’s what we’re focussing on in this post, as we’ve recently made some changes to help us detect and prevent us sending spam out. To the uninitiated, this begs a very valid question: how and why are we sending out spam?

There are many sources of spam: virus/malware infected workstations (often operating as part of a botnet), spammer-bought VPSs, compromised servers, compromised web sites, and compromised user accounts, to name just a few. Some of these result in spam coming in to the University, which we attempt to block (with a relatively high degree of success). Compromised workstations, websites, servers and user accounts are all on the list of how we could send spam out. If a workstation is infected with malware, it can be used to try and send out spam. This one is fairly trivial to fix: we block port 25 (the port used for sending mail to other organisations) outbound almost everywhere. As such, an infected workstation can try and send spam to mail servers, but it won’t be able to connect. The same is true of the significant majority of our servers.

The source which has been the big problem for us in recent years has been compromised user accounts. User accounts are typically taken control of in one of two ways: a brute-force attempt on the password – the spammer literally guesses the password; and the user falling for some kind of phishing scam. We’re pretty confident that a big portion of our compromised accounts fall in to the latter reason.

How does a compromised account cause us problems though? Once the details of a particular account are discovered, any malicious party has access to everything that that user has. This is very concerning for many reasons: they potentially can access the users filestore, their e-mails, the University VPN, off-site journal articles, and most University services. With some more work they could potentially gain access to their payroll details too. One of the services that they definitely have access to is our SMTP service (provided by Microsoft Exchange), which is what users can use to send e-mail. Herein lies the spam problem.

Once an account is compromised, the spammer connects to our SMTP service, authenticates using the username and password they’ve obtained and starts using (or rather, abusing) the service to send spam or phishing messages – this may be to University addresses (typically to trick people in to giving out more usernames and passwords) or to other organisations.

With Microsoft Exchange as the SMTP service provider, we’re limited on what we can do here. We can limit:

• The number of connections from an IP address per minute
• The number of e-mails from an account per minute
• The number of recipients an e-mail contains

We’d limited all these things, but none of these options prevent the spam from going out – merely limit the rate at which it does. We have monitoring in place so we can easily see when an individual is sending a lot of mail – but spammers aren’t stupid. They’ll typically time their attacks overnight and at weekends when IT staff are unlikely to be watching. We’d set up systems to alert us by text message when we saw a large amount of mail coming from a single address, which resulted in our admins being woken up multiple times per night to deal with spamming accounts. This went on for a while, with us scrambling to reset the passwords on compromised accounts at all hours of the night – this naturally was not a feasible long-term option. At certain points this year, we were resetting up to forty passwords a week (often more than ten a day!), which was naturally irritating to the users (especially those that had their account compromised more than once…), and was taking up significant amounts of time for our systems teams.

As such, after a few months of work, we’ve partially replaced the SMTP service provided by Microsoft Exchange with something we have much more control over. Having spent much time dealing with the spam issues, we became adept at spotting patterns within them, so we built something to deal with these patterns. The result is a custom-built solution utilising a wide variety of open-source packages running on Linux, with some in-house written filters and rules to block common things we’d seen in a wide variety of attacks. For security reasons, I’m not going to go in to a great deal of detail on exactly how all this works, but as a summary:

• Connections to our SMTP service no longer necessarily go to Exchange, they may go to our new service depending on a set of rules.
• The connections to our new service go to one of four Postfix servers which handle the e-mail transaction and mail delivery.
• These servers are set up to allow us to easily block by username, IP address, sender e-mail address, and a variety of other properties
• We’ve implemented the use of blacklists that inform us of IPs that are known to have compromised machines on the end of them – we reject all mail from these addresses
• We’ve put in SpamAssassin. This allows us to detect common spam patterns using a pre-defined set of rules. We’ve also put a lot of custom rules in here on things that we see try and go via our mail servers.
• We detect IPs that are trying to guess passwords. This was something we didn’t expect to trigger quite as often as it does… the result of this is us now outright-blocking large blocks of IP addresses from talking to our SMTP servers at all.
• We have some custom-written mail filters (or “milters”) which detect certain other spam characteristics and perform logging

All of this was quietly introduced several weeks back. Since then, the amount of compromised accounts has dropped massively. In the week previous to go-live of this service, we reset 25 passwords (which was about average at the time). In the three weeks since then, we’ve averaged about 5 per week – an 80% reduction. The much larger benefit though is that of all the compromised accounts we’ve detected, only one of them has succeeded in getting any spam out. This number is also somewhat artificially higher as we’ve detected compromised accounts that we previously wouldn’t have been able to. It’s also highlighted some recipient addresses which could possibly belong to the spammers themselves.

It’s not all good news though.

As I said at the start of this post, we deal with one wave of spam and then next wave comes along. We’ve oddly seen an increase of inbound spam since go-live (we have once this month blocked over 1.3 million messages in a single day!) – this may well be a coincidence, but there’s no guarantee. It could be that given we’ve stopped the spam going directly between our users, that they’re falling back to delivering spam the more traditional route. Additionally, having blocked spammers from their original route of getting spam out from our network, they’ve already started another method – using Outlook Web App. This route is trickier for them, but still a feasible method.

And so the war continues.