I’ve invented a new Batman villain. His name is “The Modeller” and his scheme is to model Gotham city entirely accurately in a way that is of no practical value to anybody. He has an OWL which sits on his shoulder which has the power to absorb huge amounts of time and energy.

The Modeller

In the first issue, “Batman vs the Modeller” the modeller gets away by confusing batman as to exactly which incarnation he currently is (Frank Miller, Golden age or Batman Begins) which forces Batman into an identity crisis where he registers different URIs and FOAF profiles for Batman and Bruce Wayne.

Over the 3 issues there’s a running subplot about the modelers master weapon, the FRBR, which everyone knows is very very powerful but when the citizens of Gotham talk about it none of them can quite agree on exactly what it does. All over the city citizens are becoming trapped in the modelers logical rabbit holes and rats nest traps, which The Batman does nothing but fiddle on the Bat-computer having now decided he must require a different URI for every possible version of himself that isn’t entirely identical. So far he’s at the level of one URI for every version in every media with a specific author, artist, director and/or actor. Any slight variation demands a new URI.

While unpopular with the fans, issue two, “Batman vs the Protégé“, will later be hailed as a Kafkaesque masterpiece. Batman descends further into madness as he realises that every moment he’s the Batman of that second in time, and each requires a URI, and every time he considers a plan of action, the theoretical Batmen in his imagination also require unique distinct identifiers which he must assign before continuing. Gotham Police are unable to do anything as they have not yet finished their OWL ontology of Gotham crime, which fails to map onto the normal crime ontology. Commissioner Gorden can’t work out what rdf:type the Penguin’s last caper should be modelled as. It closes with Batman realising that time is continuous, not discrete and he needs an uncountable infinity of URIs…

The final issue, “B-nodes and Broomsticks”, is a much more light hearted affair as Batman, gains enlightenment and realises that nothing can ever be perfectly modelled and any model should serve him not he serve the model. In the final showdown Batman gives a speech about how if we try to hold knowledge too tight it slips through our grasp. He then quickly and satisfyingly captures the Modeller and delivers him the the cops who charge him with Aggravated Wasting of Police Time (they are still bitter about reading all the w3c OWL documents).

In a twist which splits the fanbase, some love it, other hate it; we never actually find out what the FRBR was capable of doing and if it really would have  lived up to the hype.

*** * ***

OK, maybe I have a chip on my shoulder about the fact that OWL appears to be the hardest part of the whole RDF thing and for the benefit of semantic web researchers only.

I’m interested to see if I get any outraged comments. I love modelling stuff and have written OWL ontologies for fun in my own time, my issue with the whole thing is that I’m not convinced it’s a useful exercise. Everyone hates 404s but making one way links was one of the things which made the web possible. I suspect that linked data will have some similar sacrifices to make on the alter of pragmatism. The picture isn’t supposed to look like anyone in particular, my 9am drawing ability is limited.

I currently think you have to accept that URIs may be sameAs or not, depending on the task you are attempting rather than an absolute truth. Semantic relativism, baby!

I’ve spoken to a number of people who seem frustrated that the information available from http://data.gov.uk/ and other providers of RDF and SPARQL isn’t really usable by the layman yet. There seems to be a misplaced belief that as tools get better, providing raw RDF will be useful to your mum.

I’m sorry, but that’s not going to happen.

You will always need skilled people to understand the subtleties and create the mash-ups for the general public to use. Almost everybody in the UK has access to a computer. Using a computer is far more effective with even a small amount of programming skills but I’m guessing that less than 1% of the population attempt to acquire any. Maybe some people learn a little in school. Understanding how to get an answer out of a big complicated dataset is similar. With linked data these datasets may have errors, be patchy, and you may be trying to overlap two not entirely identical sets of identifiers. These require special skills which must be learned.

Information Shamen

The shaman was, in popular belief (I am hardly an expert on these things), the member of the tribe tasked with going and getting knowledge and then passing it on to the tribe. OK, maybe their approach was to chew on some dodgy fungus, but bear with me on the analogy. In our culture there’s many things we have to trust an expert on. The expert often has no more access to information than us, but they are trained to navigate it and we are not. Medical doctors and archaeologists, financial advisers and sporting commentators.

With the advent of linked data we see the need for a new breed who can venture into the complexities of the web of data and bring back an answer. Better still, bring back a recipe, or nice PHP website, which can help us get answers in a certain scope. However, any person willing to become informed and skilled enough to do this for themselves will be rare, and in doing so becomes such a specialist. They do not need to be protective of their skills, as most people will be too lazy, busy or plain unable to follow where they explore.

For an example, one of the pioneers of this new exploration is Tony Hirst. You should really read his blog. He has written up a number of his forays into the data including how he did it. His blog was a key inspiration in us keeping this one.

Data Journalism

While I like the term InfoShamen, it is a wee bit pretentious. It harks back to my days of wanting to be a cyberpunk computer programmer. These days my friends glare at me if I refer to the laptop as my “deck”.

The more acceptable term is “Data Journalist”. We’re all journalists now, so long as we have a code of integraty for our blogs. But leading the field in this is the Guardian Data Blog. Check it out.

Caveat Datanaut

This morning, over breakfast I was chatting with my housemate and cleaner about cabbages and kings and which side of the road various places drive. This led to a discussion of this very cool bridge between China and Hong Kong (irrelevant to this article but so cool it’s worth including). I then wrote a quick bit of SPARQL, between mouthfuls of bacon sandwich, to see a list of places which drive on the left and right, according to wikipedia.

I did this by first looking up a country, Japan, in Wikipedia, to see if it had the data I wanted in the infobox.. it did [View Japan on Wikipedia] as it’s in the infobox there was a good chance it would be available from DBPedia. I then went to dbpedia to find out what the predicate was [View DBPedia Data]. This gave me the predicate, dbpprop:drivesOn

Armed with this I went to http://dbpedia.org/sparql and wrote out my query:

SELECT DISTINCT ?name ?side WHERE {
   ?place dbpprop:drivesOn ?side .
   ?place foaf:name ?name .
}

Which failed as I forgot the namespaces, so a quick trip to prefix.cc got me those. I use prefix.cc often enough that I can just write http://prefix.cc/foaf,dbpprop.sparql into a browser to get the cut and paste bit I need. So I added this to the top to make it work.

PREFIX foaf:    <http://xmlns.com/foaf/0.1/>
PREFIX dbpprop: <http://dbpedia.org/property/>

All of this took about 4 minutes, compared with the 30 or so it’s taken to document.

• See my table of country to side of the road driveyness

This data was interesting for a quick view that lots more places than we thought drive on the left. However it’s flawed in a number of ways. It lists countries once for each variation of their foaf:name. All I want is any name, not all of them. Maybe there’s a predicate in dbpedia for preferred name, but it’s not worth the time to find out over breakfast. Maybe I should have just listed the URI and driveysideness, but I wanted to present it to people not used to dbpedia URIs.

My housemate asked about South Africa and it’s not in the list. I just checked and the wikipedia page does have a “drives on” element in the info box, but even when I remove the requirement for a foaf:name, it still does not show up in the data.

I think that this illustrates the current level of skill required to work with such data, and also the risks of attempting to interpret it. We can improve on this but it will be like improvements to library science, it’ll still require a skilled operator to get meaningful results.

1966

I’m hardly the first to comment on this. The following quote has been on my homepage for years and it’s exciting seeing it beginning to come into it’s own 44 years after it was said.

“Today the forerunners of these synthesists are already at work in many places. Their titles may be anything; their degrees may be in anything – or they may have no degrees.

Today the are called `operations researchers’, or sometimes `systems development engineers’, or other interim tags. But they are all interdisciplinary people, generalists, not specialists – the new Renaissance Man. The very explosion of data which forced most scholars to specialise very narrowly created the necessity which evoked this new non-specialist. So far, this `unspeciality’ is in its infancy; its methodology is inchoate, the results are sometimes trivial, and no one knows how to train to become such a man. But the results are often spectacularly brilliant, too — this new man may yet save all of us.” – Robert A. Heinlein, 1966.

That last bit isn’t a bit of an exaggeration. Once a respectable number of science datasets become available online in usefully open and interoperable formats we will have to rethink science. Some people are skilled at generating data and some at interpreting it. Currently prestige is given for collecting and interpreting some data (ie. publishing in a high impact journal) but it seems likely that these will, should and must become decoupled. Tim Berners-Lee, speaking at the Royal Society this year, said something along the lines of “Do your research in the mashuposphere but give all due credit to the datasphere”.

The world has changed in some very exciting ways over the last few years, and few people can see it yet!…

* one caveat. We may get to the point where AI can take a human question and query the web of linked data for the results, and report back with an explanation of any gaps or uncertainties in the data. Once we get to that level of AI, it becomes hard to justify the purpose of the human in the process.

A brief post on how we’re keeping our RDF accurate and up to date:

We’ve had our infrastructure data available as RDF for a while now (thanks to Nick Gibbins et al.), and have recently started pushing our datasets into a triplestore.

As time goes on, we’re consuming and re-using more and more of our RDF, with a plan to make our SPARQL endpoint public. This means that it’s increasingly important that the RDF we make available is up to date and an accurate representation of the data it’s generated from.

Keeping our RDF updated

This is the easy part. Our underlying data is mostly in MSSQL/MySQL databases. Each RDF document (e.g. http://rdf.ecs.soton.ac.uk/person/60) is built dynamically each time the URL is requested (with some caching).

Keeping our Triplestore updated

All triples from each RDF document are stored in a named graph. We’re using the URL of the RDF document as a graph name (so triples read from http://rdf.ecs.soton.ac.uk/person/60 would be stored in a the named graph http://rdf.ecs.soton.ac.uk/person/60).

Every time an RDF document is requested (and once per night), a hash of the data is calculated, and stored in a database, along with the time the hash was calculated.

A set of scripts running on the same host as the triplestore query the hash database periodically, and re-import triples from any RDF documents which have changed since it last checked.

We’re also slowly moving towards live updates. Many of our systems have a single point up change (e.g. a person’s profile pages are edited from a single form), so we’re planning to insert hooks into these that trigger a refresh of the data into the triplestore.

Keeping our Data Safe

An ongoing worry is that private information could get into our triplestore, or exposed on our RDF pages. Once incorrect/private data gets out, it’d be difficult to remove should the data ever get into external datasets.

As well as having plenty of checks throughout code, we’ve also added monitoring for this to our Nagios IT monitoring system.

At regular intervals, we build a list of anyone in the School who shouldn’t have information about themselves visible. This is then used build a number of SPARQL queries, which are execute to ensure the data hasn’t made it into our triplestore.

If something is found, we log the date this was first noticed. If this isn’t corrected within a reasonable timeframe (currently 24 hours or so), our Nagios system sends warning emails out. (The 24h delay is to allow for the delay between the data changing at source, and the triplestore picking up on that change)

Further Checks

As we deal with more (and larger) datasets, I can see a use for setting up additional checks like this – monitoring our data is just as important as monitoring our systems infrastructure in a lot of cases.

SPARQL (along with multiple datasets kept in a triplestore) provides a convenient (and consistent) way of effectively querying across multiple data sources.

A few ideas for future checks include:

• Data integrity checks (e.g. check each module has 1+ teachers)
• Spelling checks (e.g. check any literals for common misspellings)
• Link checks (e.g. check any URL/URI used actually resolves)

Unit tests for data?