As you’re probably aware, Blackboard was scheduled to be unavailable for four days recently for us to do some maintenance. There’s never a good time to take down Blackboard, but this date was chosen to minimise the impact to our users – most students will be off home over this weekend and looking forward to Christmas! But why was this maintenance period necessary? Earlier in the year, we wrote about how we’d made some changes to improve the performance of Blackboard and about how we fixed some issues we were seeing with stability. During this maintenance period, we did more. A lot more.
One of the main aims of this piece of work was to move the service from its current location to our brand new data centre. On the face of it, four days seemed like a lot of down time to move some servers from one place to another, but we weren’t simply lifting-and-shifting the kit. Also, moving the service to the data centre was only the tip of the iceberg – the service was completely rebuilt – new hardware, new OS and greatly upgraded specs.
We discussed in our last post on Blackboard about how we had a couple of Sun SPARC T5240s running the front end webservers, which are load-balanced by some F5 NLBs and some LDOMs (Solaris virtual machines) on an Oracle SPARC T4 series box running the database, file server and collaboration server. The improved Blackboard service does not include any of this hardware – it is all virtualised on VMware. Our VMware hosts are 32-core systems with 128gb of storage – and we have over 50 of them! As part of this move to VMware virtual machines, we also needed to change the OS. We could have opted for Solaris on x86… but the better solution was to move to Linux – Red Hat Enterprise Linux 6 to be precise.
The move to these VMs on the awesome host hardware is already a big boost to performance, but we’ve not stopped there. Previously we had two front-end webservers. We’re doubling this to four, along with the separate collaboration server. Each of the front-ends will be a 4-core, 16gb system, all individually much more powerful than the previous front ends. These front ends are then load balanced by a pair of brand new F5 network load balancers. Behind all of this is a much improved Oracle database server, with 48gb of RAM.
Naturally, we did some testing in a pre-production environment to see exactly how much better this new platform is, and the outcome was quite impressive. To give you some kind of idea:
- Installing a particular patchset (a selection of upgrades or bugfixes) for this version of Blackboard took around 25 minutes on the old platform. This new platform ran the same task in 7.5 seconds.
- Restarting Blackboard services on the webservers on the old platform took around 12-15 minutes – we’re down to about a minute now.
- An issue with Blackboard means that the first user to click the “Personalise Page” link on the existing system causes some high load on the front-ends. On the old environment, this could last for over 10 minutes, whilst on the new environment it recovers in about 20 seconds.
- With requests for static content (e.g. images), we can now manage over 13,000 requests a second, a more than six times increase
The speed of the new production environment (and the amount of preparation done by the upgrade team beforehand!) meant we actually able to complete all this work way ahead of schedule. Instead of Blackboard being down for the scheduled four days, it was only actually down for about 36 hours!
The proof of the new environment will come in the new year when everyone piles on to Blackboard for the semester one exams and then leading on in to semester two. We’ll be looking very closely at how the new environment handles it, and we’ll have more blog posts about it in the new year, as will our MLE team (whose excellent blog you can read at blog.soton.ac.uk/mle, as well as following them on Twitter, @uos_mle).
2 pings
Blackboard in the new data centre, what does it mean for me?
December 17, 2013 at 9:42 am (UTC 0) Link to this comment
[…] This wasn’t a simple case of unplugging it, putting it a white van, and driving it over, but a complex migration over the extended network into a brand new set of hardware running on a different operating system, using brand new storage and load balancers. You can find out more of the technical details in the tech team blog. […]
Blackboard Performance Update » iSolutions Technical Team Blog
February 17, 2014 at 2:29 pm (UTC 0) Link to this comment
[…] to the new data centre and performed the various hardware upgrades (see the post we made about that here), so it’s about time that we take a look at how the service has performed since it went live, […]