{"id":658,"date":"2011-03-02T17:12:20","date_gmt":"2011-03-02T17:12:20","guid":{"rendered":"http:\/\/blog.soton.ac.uk\/webteam\/?p=658"},"modified":"2011-03-02T17:13:11","modified_gmt":"2011-03-02T17:13:11","slug":"rdf-sparql-debugging","status":"publish","type":"post","link":"https:\/\/blog.soton.ac.uk\/webteam\/2011\/03\/02\/rdf-sparql-debugging\/","title":{"rendered":"RDF \/ SPARQL debugging"},"content":{"rendered":"

While working on data.southampton.ac.uk<\/a>, we run into inevitable problems with data failing to load into our triplestore, or with SPARQL queries failing to return in a reasonable time (or at all…).<\/p>\n

I thought I’d share a few of the basic steps I take to find the root of these issues.<\/p>\n

RDF Import Problems<\/h2>\n

The main tool I use for debugging raw RDF when it fails to import into a store is rapper<\/a>.<\/p>\n

This is a small command line utility which uses the Raptor<\/a> library to parse RDF in a number of formats (currently: rdfxml, ntriples, turtle, trig, rss-tag-soup, grddl, rdfa).<\/p>\n

It’s extremely fast at parsing, so it a quick way of finding errors in a file.<\/p>\n

I’ll usually run something like:
\n\/usr\/bin\/rapper -i rdfxml input.rdf > \/dev\/null<\/code><\/p>\n

This attempts to read input.rdf as RDF\/XML, convert it to ntriples, then throw away the output.<\/p>\n

Any syntax errors it comes across are reported on the command line. It usually takes seconds to run, even on large (well, ~4m RDF\/XML triples large) datasets, and seems much more accurate in spotting mistakes than web based validators I’ve used.<\/p>\n

SPARQL Query Problems<\/h2>\n

Slow queries have accounted for the majority of the problems we’ve had so far.<\/p>\n

The thing that isn’t immediately obvious, is that a query which returns a few items can still be generating thousands of intermediate results under the hood.<\/p>\n

So far, there have been two underlying causes for this:<\/p>\n

    \n
  1. Poorly optimised queries<\/li>\n
  2. Incorrect source data imported<\/li>\n<\/ol>\n

    Triplestore query optimisation isn’t perfect yet, so it’s always worth playing around with changing the order of parts of queries, rather than assuming an optimiser will do it.<\/p>\n

    Incorrect (but valid) source data being imported into the store has caused us problems a few times. The main culprit has been where large numbers of values have been mapped to a single predicate for a single object.<\/p>\n

    One example was in our phonebook data<\/a>, where a single person URI had several hundred foaf:name, foaf:mbox and foaf:phone assigned to it.<\/p>\n

    The number of results for queries involving this URI quickly exploded for each predicate added to a query.<\/p>\n

    The easiest way to debug these problems has been to:<\/p>\n

      \n
    1. Break each query down into atomic parts<\/li>\n
    2. Run a SPARQL COUNT first part<\/li>\n
    3. Add another part to the query<\/li>\n
    4. Run a SPARQL COUNT on the combined parts<\/li>\n
    5. Look for significant changes between 2. and 4.<\/li>\n<\/ol>\n

      So in the example of our phonebook data error, the initial query was something like:
      \nSELECT * WHERE {
      \n ?person a foaf:Person ;
      \n foaf:familyName ?family_name ;
      \n foaf:givenName ?given_name ;
      \n foaf:name ?name ;
      \n foaf:mbox ?email ;
      \n foaf:phone ?phone .
      \n}<\/code><\/p>\n

      Which looked reasonable, until the query took up 10+ Gb of RAM…<\/p>\n

      Debugging then started with:
      \nSELECT (COUNT(*) AS ?count) WHERE {
      \n ?person a foaf:Person .
      \n}<\/code><\/p>\n

      Which gave ~2000 results.<\/p>\n

      Looking just at foaf:name:
      \nSELECT (COUNT(*) AS ?count) WHERE {
      \n ?person a foaf:Person .
      \n ?person foaf:name ?name .
      \n}<\/code><\/p>\n

      This gave ~70000 results rather than the expected ~2000. A quick look through the source data then showed the issue.<\/p>\n

      It’s pretty simple really, but took us a while to get a feel for how to debug query problems, and what to look out for.<\/p>\n

      I’ve also started monitoring 4store<\/a>‘s query log to look for slow queries, as this has been a great indicator of poor data or query structure.<\/p>\n","protected":false},"excerpt":{"rendered":"

      While working on data.southampton.ac.uk, we run into inevitable problems with data failing to load into our triplestore, or with SPARQL queries failing to return in a reasonable time (or at all…). I thought I’d share a few of the basic steps I take to find the root of these issues. RDF Import Problems The main […]<\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[136,411,4227],"tags":[],"class_list":["post-658","post","type-post","status-publish","format-standard","hentry","category-rdf","category-sparql","category-triplestore"],"_links":{"self":[{"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/posts\/658","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/comments?post=658"}],"version-history":[{"count":9,"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/posts\/658\/revisions"}],"predecessor-version":[{"id":667,"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/posts\/658\/revisions\/667"}],"wp:attachment":[{"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/media?parent=658"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/categories?post=658"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.soton.ac.uk\/webteam\/wp-json\/wp\/v2\/tags?post=658"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}