The University revolves around e-mail. It’s one of our most important and largest services, both in terms of number of users, and amount of hardware required to run it. It ties in to many systems, both within the University and externally, so it also one of our more complex systems. As mentioned in a previous blog post, we deliver an estimated 40 million e-mails a year. This number might seem high, but it averages out to only 3-4 messages per person per day, which doesn’t seem so bad – this makes it look like not much at all. However, this number only includes mails which leave or enter the University (so are processed by our “edge” mail servers) – mails between members of staff or third-year students aren’t counted here. As such, the total number of messages delivered in total is considerably higher.
As a brief overview, our mail system consists of a few high-level logical components:
- Microsoft Exchange: This hosts all staff, researcher, and post-graduate e-mail, along with e-mail for all third-year students (currently), many SUSU societies, and a selection of other mail accounts.
- Microsoft Live@Edu: This hosts mail for all first-year and second-year students. This service is provided by Microsoft, and we have systems in place that tie this in to the rest of the University e-mail system.
- Microsoft Office365: This new service will host mail for all new students starting in 2013, and existing students currently using Live@Edu will be migrated to it over the Summer. Again, this service is provided by Microsoft, and will tie in (albeit differently) to the University e-mail system.
- Edge Mail Servers: These servers handle all mail that comes in to or goes out of the University, including mails between our local Microsoft Exchange and the external Live@Edu/Office365 e-mail services. These systems perform mail routing, address re-writing and redirection, along with a plethora of virus and spam scanning checks.
- School Mail Servers: Some schools still have their own mail systems (most notably, ECS, although parts of that have now been migrated to iSolutions). Our mail systems have to integrate in to these.
- Mailing Lists: We have a couple of mailing list systems used for departmental and SUSU society mail distribution.
All of these components are linked together in a many ways and tie in to other services. Exchange ties into the University Subscribe system for provisioning new mail accounts for new staff and determining which of their e-mail address to use; Live@Edu ties into Subscribe via another service in order to provision new mail accounts and synchronise passwords; Office365 will tie in to Subscribe for mail account provisioning and into ADFS for authentication. The Edge Mail Servers tie into all of these systems and more – they are the crucial piece that holds everything together. If all of the inbound edge mail servers are down, mail flow in to the University stops – including Live@Edu and Office365. If all of the outbound edge mail servers are down, Staff/PGRs can’t mail out of the University, or to schools with their own mail systems; and again users on Live@Edu would receive no mail at all. Thankfully, it’s quite rare for any of the edge mail system to be entirely down.
Pretty much every part of the e-mail system is being upgraded this year. Exchange is having it’s underlying storage upgraded, and new servers are being added, all with the aim to increase performance and reduce the impact (and frequency) of outages. Microsoft is replacing Live@Edu with Office365 – this is due to happen for us during Summer 2013. The ageing Majordomo mailing list system is scheduled for a replacement (thought that might not happen this year!). The big change here though is to the edge mail servers – they are getting a complete overhaul. The existing, ageing, physical hardware is being removed and replaced by virtual machines; the software is being upgraded or completely replaced – we’re essentially rebuilding the entire service from scratch. We’ll also be adding in some new features so that we’re able to do things we didn’t do before. Below is a list of some of the most notable changes:
- Quarantining: we see lots of ServiceLine tickets regarding attachments not being delivered due to file types or file name restrictions. We’re planning on adding a feature whereby these messages are not discarded – they are delivered, along with a link to a system to release the attachment from quarantine. Attachments that are detected as viruses will still not be delivered however. This should allow you to receive attachments that previously would have been blocked – including executables, certain Matlab files, and encrypted compressed archives.
- Improved inbound spam scanning: We already block over 600,000 spam messages every day – but occasionally it can still get through. Updated software, additional spam checks and spam rules will hopefully reduce the amount of spam that gets delivered. We’ll also be setting up some mailboxes so that you can send any spam you receive to us so that our automated checks can be improved.
- Outbound spam scanning: As seen earlier this year, spam is a massive problem. It’s annoying when it gets sent to the University… but it’s disastrous for us when we send it out. During the targeted phishing attacks against UK universities (we were certainly not alone in being targeted) that happened during January/February we had to take some drastic action to prevent us sending out spam, including the unprecedented reset of everyone’s password. We appeared on many e-mail blacklists meaning we were unable to send mail to many organisations – Hotmail and GMail also started blocking our mail or automatically marking our mail as junk. Working around and solving these issues; detecting compromised accounts; and getting passwords reset took weeks of effort – with some iSolutions staff working literally (and I mean that in it’s actual meaning) round the clock. As such, we’re enabling outbound spam scanning, so that when accounts are compromised and start trying to send spam, they’ll get blocked by our edge mail system.
- Improved performance: Presently, e-mail from external addresses going to Live@Edu mailboxes takes a slightly odd route through our mail system, going through our edge mail system twice as well as having to be handled by Exchange. This routing will be re-done so that the edge mail systems spam check it and then send it directly to Live@Edu without having to go through Exchange (or through the edge mail system again!). The improved software, upgraded OS, and faster virtual hardware will also increase the speed at which mail is processed, so mail won’t be queued for so long before being handled for delivery.
- Improved management: This should hopefully improve ServiceLine ticket resolution times. We’re writing some custom management systems for our mail system so that when a mail doesn’t get delivered properly, we’ll have more people be able to find out why and be able to find out much easier.
- Under-the-hood changes: Various configuration changes will be made and mail-verification technologies (DKIM, for example) implemented, which should help improve our mail deliverability and reduce the risk of us being blacklisted
Much of this is still several months away and there’s lots of work ahead to get this all in place. We’ll be keeping you updated with how all this progresses!
Recent Comments