I’ve spoken to a number of people who seem frustrated that the information available from http://data.gov.uk/ and other providers of RDF and SPARQL isn’t really usable by the layman yet. There seems to be a misplaced belief that as tools get better, providing raw RDF will be useful to your mum.

I’m sorry, but that’s not going to happen.

You will always need skilled people to understand the subtleties and create the mash-ups for the general public to use. Almost everybody in the UK has access to a computer. Using a computer is far more effective with even a small amount of programming skills but I’m guessing that less than 1% of the population attempt to acquire any. Maybe some people learn a little in school. Understanding how to get an answer out of a big complicated dataset is similar. With linked data these datasets may have errors, be patchy, and you may be trying to overlap two not entirely identical sets of identifiers. These require special skills which must be learned.

Information Shamen

The shaman was, in popular belief (I am hardly an expert on these things), the member of the tribe tasked with going and getting knowledge and then passing it on to the tribe. OK, maybe their approach was to chew on some dodgy fungus, but bear with me on the analogy. In our culture there’s many things we have to trust an expert on. The expert often has no more access to information than us, but they are trained to navigate it and we are not. Medical doctors and archaeologists, financial advisers and sporting commentators.

With the advent of linked data we see the need for a new breed who can venture into the complexities of the web of data and bring back an answer. Better still, bring back a recipe, or nice PHP website, which can help us get answers in a certain scope. However, any person willing to become informed and skilled enough to do this for themselves will be rare, and in doing so becomes such a specialist. They do not need to be protective of their skills, as most people will be too lazy, busy or plain unable to follow where they explore.

For an example, one of the pioneers of this new exploration is Tony Hirst. You should really read his blog. He has written up a number of his forays into the data including how he did it. His blog was a key inspiration in us keeping this one.

Data Journalism

While I like the term InfoShamen, it is a wee bit pretentious. It harks back to my days of wanting to be a cyberpunk computer programmer. These days my friends glare at me if I refer to the laptop as my “deck”.

The more acceptable term is “Data Journalist”. We’re all journalists now, so long as we have a code of integraty for our blogs. But leading the field in this is the Guardian Data Blog. Check it out.

Caveat Datanaut

This morning, over breakfast I was chatting with my housemate and cleaner about cabbages and kings and which side of the road various places drive. This led to a discussion of this very cool bridge between China and Hong Kong (irrelevant to this article but so cool it’s worth including). I then wrote a quick bit of SPARQL, between mouthfuls of bacon sandwich, to see a list of places which drive on the left and right, according to wikipedia.

I did this by first looking up a country, Japan, in Wikipedia, to see if it had the data I wanted in the infobox.. it did [View Japan on Wikipedia] as it’s in the infobox there was a good chance it would be available from DBPedia. I then went to dbpedia to find out what the predicate was [View DBPedia Data]. This gave me the predicate, dbpprop:drivesOn

Armed with this I went to http://dbpedia.org/sparql and wrote out my query:

SELECT DISTINCT ?name ?side WHERE {
   ?place dbpprop:drivesOn ?side .
   ?place foaf:name ?name .
}

Which failed as I forgot the namespaces, so a quick trip to prefix.cc got me those. I use prefix.cc often enough that I can just write http://prefix.cc/foaf,dbpprop.sparql into a browser to get the cut and paste bit I need. So I added this to the top to make it work.

PREFIX foaf:    <http://xmlns.com/foaf/0.1/>
PREFIX dbpprop: <http://dbpedia.org/property/>

All of this took about 4 minutes, compared with the 30 or so it’s taken to document.

• See my table of country to side of the road driveyness

This data was interesting for a quick view that lots more places than we thought drive on the left. However it’s flawed in a number of ways. It lists countries once for each variation of their foaf:name. All I want is any name, not all of them. Maybe there’s a predicate in dbpedia for preferred name, but it’s not worth the time to find out over breakfast. Maybe I should have just listed the URI and driveysideness, but I wanted to present it to people not used to dbpedia URIs.

My housemate asked about South Africa and it’s not in the list. I just checked and the wikipedia page does have a “drives on” element in the info box, but even when I remove the requirement for a foaf:name, it still does not show up in the data.

I think that this illustrates the current level of skill required to work with such data, and also the risks of attempting to interpret it. We can improve on this but it will be like improvements to library science, it’ll still require a skilled operator to get meaningful results.

1966

I’m hardly the first to comment on this. The following quote has been on my homepage for years and it’s exciting seeing it beginning to come into it’s own 44 years after it was said.

“Today the forerunners of these synthesists are already at work in many places. Their titles may be anything; their degrees may be in anything – or they may have no degrees.

Today the are called `operations researchers’, or sometimes `systems development engineers’, or other interim tags. But they are all interdisciplinary people, generalists, not specialists – the new Renaissance Man. The very explosion of data which forced most scholars to specialise very narrowly created the necessity which evoked this new non-specialist. So far, this `unspeciality’ is in its infancy; its methodology is inchoate, the results are sometimes trivial, and no one knows how to train to become such a man. But the results are often spectacularly brilliant, too — this new man may yet save all of us.” – Robert A. Heinlein, 1966.

That last bit isn’t a bit of an exaggeration. Once a respectable number of science datasets become available online in usefully open and interoperable formats we will have to rethink science. Some people are skilled at generating data and some at interpreting it. Currently prestige is given for collecting and interpreting some data (ie. publishing in a high impact journal) but it seems likely that these will, should and must become decoupled. Tim Berners-Lee, speaking at the Royal Society this year, said something along the lines of “Do your research in the mashuposphere but give all due credit to the datasphere”.

The world has changed in some very exciting ways over the last few years, and few people can see it yet!…

* one caveat. We may get to the point where AI can take a human question and query the web of linked data for the results, and report back with an explanation of any gaps or uncertainties in the data. Once we get to that level of AI, it becomes hard to justify the purpose of the human in the process.

A brief post on how we’re keeping our RDF accurate and up to date:

We’ve had our infrastructure data available as RDF for a while now (thanks to Nick Gibbins et al.), and have recently started pushing our datasets into a triplestore.

As time goes on, we’re consuming and re-using more and more of our RDF, with a plan to make our SPARQL endpoint public. This means that it’s increasingly important that the RDF we make available is up to date and an accurate representation of the data it’s generated from.

Keeping our RDF updated

This is the easy part. Our underlying data is mostly in MSSQL/MySQL databases. Each RDF document (e.g. http://rdf.ecs.soton.ac.uk/person/60) is built dynamically each time the URL is requested (with some caching).

Keeping our Triplestore updated

All triples from each RDF document are stored in a named graph. We’re using the URL of the RDF document as a graph name (so triples read from http://rdf.ecs.soton.ac.uk/person/60 would be stored in a the named graph http://rdf.ecs.soton.ac.uk/person/60).

Every time an RDF document is requested (and once per night), a hash of the data is calculated, and stored in a database, along with the time the hash was calculated.

A set of scripts running on the same host as the triplestore query the hash database periodically, and re-import triples from any RDF documents which have changed since it last checked.

We’re also slowly moving towards live updates. Many of our systems have a single point up change (e.g. a person’s profile pages are edited from a single form), so we’re planning to insert hooks into these that trigger a refresh of the data into the triplestore.

Keeping our Data Safe

An ongoing worry is that private information could get into our triplestore, or exposed on our RDF pages. Once incorrect/private data gets out, it’d be difficult to remove should the data ever get into external datasets.

As well as having plenty of checks throughout code, we’ve also added monitoring for this to our Nagios IT monitoring system.

At regular intervals, we build a list of anyone in the School who shouldn’t have information about themselves visible. This is then used build a number of SPARQL queries, which are execute to ensure the data hasn’t made it into our triplestore.

If something is found, we log the date this was first noticed. If this isn’t corrected within a reasonable timeframe (currently 24 hours or so), our Nagios system sends warning emails out. (The 24h delay is to allow for the delay between the data changing at source, and the triplestore picking up on that change)

Further Checks

As we deal with more (and larger) datasets, I can see a use for setting up additional checks like this – monitoring our data is just as important as monitoring our systems infrastructure in a lot of cases.

SPARQL (along with multiple datasets kept in a triplestore) provides a convenient (and consistent) way of effectively querying across multiple data sources.

A few ideas for future checks include:

• Data integrity checks (e.g. check each module has 1+ teachers)
• Spelling checks (e.g. check any literals for common misspellings)
• Link checks (e.g. check any URL/URI used actually resolves)

Unit tests for data?

We announced on Tuesday (13th July 2010) that all the RDF made available about our school would be placed in the public domain.

Around five years ago we (The School of Electronics and Computer Science, University of Southampton) had a project, led by Dr Nick Gibbins, to create open data of our infrastructure data. This included staff, teaching modules, research groups, seminars and projects. This year we have been overhauling the site based on what we’ve learned in the interim. We made plenty of mistakes, but that’s fine and what being a university is all about. We’ll continue to blog about what we’ve learned.

We have formally added a “CC0” public domain license to all our infrastructure RDF data, such as staff contact details, research groups and publication lists. One reason few people took an interest in working with our data is that we didn’t explicitly say what was and wasn’t OK, and people are disinclined to build anything on top of data which they have no explicit permission to use. Most people want to instinctively preserve some rights over their data, but we can see no value in restricting what this data can be used for. Restricting commercial use is not helpful and restricting derivative works of data is non-sensical!

Here’s an Example; Someone is building a website to list academics by their research area and they use our data to add our staff to this. How does it benefit us to force them to attribute our data to us? They are already assisting us by making our staff and pages more discoverable, why would we want to provide a restriction?. If they want to build a service that compiles and republishes data they would need to track every license and that’s going to be a bother of a similar scale to the original BSD Clause 3.

Our attitude is that we’d like an attribution where convenient, but not if it’s a bother. must-attribute is a legal requirement, we say “please-attribute”. It’s our hope that this step will help other similar organisations take the same step with the confidence of not being the first to do so.

The CC0 license does not currently extend to our research publications documents (just the metadata) or to research data. It is my personal view that research funders should make it a requirement of funding that a project publishes all data produced, in open formats, along with any custom software used to produce it, or required to process it, along with the source and (ideally) the complete cvs/git/svn history. This is beyond the scope of what we’ve done recently in ECS, but the University is taking the management of research data very seriously and it is my hope that this will result in more openness.

Another mistake we have learned from is that we made a huge effort to correctly model and describe our data as semantically accurately as possible. Nobody cares enough about our data to explain to their tool what an “ECS Person” is. We’re in the process of adding in the more generic schemes like FOAF and SIOC etc. The awesome thing about the RDF format is that we can do this gently and incrementally. So now everybody is both (is rdf:type of) a ecs:Person and a foaf:Person. (example). The process of making this more generic will continue for a while, and we may eventually expire most of the extraneous ecs:xyz site-specific relationships except where no better ones exist.

The key turning point for us was when we started trying to us this data to solve our own problems. We frequently build websites for projects and research groups and these want views on staff, projects, publications etc. Currently this is done with an SQL connection to the database and we hope the postgrad running the site doesn’t make any

cock-ups which result in data being made public which should not have been. We’ve never had any (major) problems with this approach, but we think that loading all our RDF data into a SPARQL server (like an SQL server, but for RDF data and connects with HTTP) is a better approach. The SPARQL server only contains information we are making public so the risks of leaks (eg. staff who’ve not given formal permission to appear on our website) is minimised. We’ve taken our first faltering steps and discovered immediately that our data sucked (well, wasn’t as useful as we’d imagined). We’d modelled it with an eye to accuracy, not usefulness, believing if you build it they will come. The process of “eating our own dogfood” rapidly revealed many typos, and poor design decisions which had not come to light in the previous 4 or 5 years!

Currently we’re also thinking about what the best “boilerplate” data is to put in each document. Again, we’re now thinking about how to make it useful to other people rather than how to accurately model things.

There’s no definitive guidance on this. I’m interested to hear from people who wish to consume data like this to tell us what they *need* to be told, rather than what we want to tell them. Currently we’ve probably got an overkilll!

One field I believe should be standard which we don’t have is where to send corrections to. Some of the data.gov.uk is out of date and an instruction on how to correct it would be nice and benefit everyone.

At the same time we have started making our research publication metadata available as RDF, also CC0, via our EPrints server. It helps that I’m also lead developer for
the EPrints project! By default any site upgrading to EPrints 3.2.1 or later will get linked data being made available automatically (albeit, with an unspecified license).

Now let me tell you how open linked data can save a university time and money!

Scenario: The university cartography department provides open data in RDF form describing every building, it’s GPS coordinates and it’s ID number. (I was able to create such a file for 61 university buildings in less than an hours work. It is already freely published on maps on our website so no big deal making it available.

The university teaching support team maintain a database of learning spaces, and the features they contain (projectors, seating layout, capacity etc.) and what building each one is in. They use the same identifier (URI) for buildings as the cartography dept. but don’t even need to talk to them, as the scheme is very simple. Let’s say:
http://data.exampleuniversity.ac.uk/location/building/23

Each team undertakes to keep their bit up to date, which is basically work they were doing anyway. They source any of their systems from this data so there’s only one place to maintain it. They maintain it in whatever form works for them (SQL, raw RDF, textfile, Excel file in a shared directory!) and data.exampleuniversity.ac.uk knows how to get at this and provide it in well formed RDF.

The timetabling team wants to build a service to allow lecturers and students to search for empty rooms with certain features, near where they are now. (This is a genuine request made of our Timetable team at Southampton that they would like a solution for)

The coder tasked with this gets the list of empty rooms from the timetabling team, possibly this won’t be open data, but it still uses the same room IDs (URIs). eg. http://data.exampleuniversity.ac.uk/location/building/23/room/101

She can then mash this up with the learning-space data and the building location data to build a search to show empty rooms, filtered by required feature(s). She could even take the building you’re currently in and sort the results by distance away from you. The key thing is that she doesn’t have to recreate any existing data, and as the data is open she doesn’t need to jump through any hoops to get it. She may wish to register her use so that she’s informed of any planed outages or changes to the data she’s using but that’s about it. She has to do no additional maintenance as the data is being sourced directly from the owners. You could do all this with SQL, but this approach allows
people to use the data with confidence without having to get a bunch of senior managers to agree a business case. An academic from another university, running a conference at exampleuniversity can use the same information without having to navigate any of the politics and bureaucracy and improve their conference sites value to delegates by joining each session to it’s accurate location. If they make the conference programme into linked data (see http://programme.ecs.soton.ac.uk/ for my work in this area!) then a 3rd party could develop an iPhone app to mash up the programme & university building location datasets and help delegates navigate.

But the key thing is that making your information machine readable, discoverable and openly licensed is of most value to your own members in an organisation. It stops duplication of work and reduces time wasted trying to get a copy of data other staff maintain.

“If HP knew what HP knows, we’d be three times more profitable.” – Hewlett-Packard Chairman and CEO Lew Platt

I’ve been working on a mindmap to brainstorm every potential entity a university may eventually want to identify with a URI. Many of these would benefit from open data. Please contact me if you’ve got ones to add! It would be potentially useful to start recommending styles for URIs for things like rooms, courses and seminars as most of our data will be of a similar shape, and it makes things easier if we can avoid needless inconsistency!

Christopher Gutteridge, Web Projects Manager, Electronics and Computer Science (ECS), University of Southampton. July 2010. cjg@ecs.soton.ac.uk.