Our tale begins a month and a half ago, in a lab at the University of Southampton….

I was at the beginning of my internship, and we had decided one of the key jobs was fleshing out our data. The Open Data Service previously gathered data using pen and whiteboard. The issue with these archaic tools is time. Not just the time spent gathering data, but processing it all by hand at the other end.

As such, I set out on journey. A journey to create one tool that would facilitate the easy gathering of data. It’s still far from complete, but here’s what it does so far.

Overview
DISCLAIMER: None of this tool is considered “stable” at this point in time. Data formats can and will change, use at your own risk!

The aims of the tool are:

• To speed up the gathering and processing of data. More specifically, data gathered on-location.
• To enable and encourage non-technical people to contribute open data.

The result is a responsive website, written in PHP. PHP was chosen as it’s trivially integrated into the Open Data service. – https://github.com/Spoffy/OpenGather

User Interface

The current interface is extremely simple, but it gets the job done. It shows a set of object types (schemas) people can submit. Changing the schema changes the form fields shown. These can then be filled in and submitted. Any required fields that weren’t filled in are highlighted in red.

The tool currently supports text fields, dropdown fields and geolocation fields. For geolocation fields, the initial values of longitude and latitude use the phone’s GPS. It’s possible to click on the map to select a more precise location. This is especially useful when recording the location-sensitive objects such as doors.

The Database

A small subset of the tables.

The tool uses MySQL as its default backend. The details are configurable in config.php. There’s a single central table that records each data item entered. It stores an id, the time and the schema id.

Each schema has its own table of details. Each entry’s id acts a foreign key, relating an entry to the details about it in the schema’s table.

Exporting Data

Currently, the data is exportable as JSON. This format allows several schemas to exist together seamlessly. JSON is also human editable, making it easy to correct long-term data. The tool makes the export publically available at http://yourwebsite/path/to/tool/dumpjson.php. There’s no issue with making data public, as it’s designed to gather OpenData.

The data is also available through the MySQL instance. This method of access isn’t recommended.

The Schema Generator

Personally, I think this is the coolest part of the system. It allows you to quickly specify a schema using PHP. This schema is then transformed into HTML for the web forms and SQL for the database. The web interface updates the schema list when the page loads. Dynamically loading these schemas allows submissions to go straight to the database.

The upshot is that defining new schema is incredibly easy. There’s no need to mess around with HTML or SQL. Just a few PHP objects gets the job done!

The following is a sample schema I use to gather data from around the University:

//Format: 
//    new ObjectSchema($name, $fieldsArray);
//    new TextField($name, $id, $required);
//    new DropdownField($name, $id, $optionsArray, $required=true);

$schemas = array(
    new ObjectSchema("Building Entrance", array(
        new TextField("Building Number", "buildingId", true),
        new TextField("Entrance Label", "entranceId", false),
        new TextField("Description", "description", true),
        new GeoField("Latitude", "lat", true),
        new GeoField("Longitude", "long", true),
        new DropdownField("Access Method Daytime", "accessDaytime", $accessOptions),
        new DropdownField("Access Method Evening", "accessEvening", $accessOptions),
        new DropdownField("Opening Method", "openingMethod", $openingOptions)
    )),

    new ObjectSchema("Drinking Water Source", array(
        new TextField("Building Number", "buildingId", true),
        new TextField("Floor", "floor", true),
        new GeoField("Latitude", "lat", true),
        new GeoField("Longitude", "long", true)
    ))
);

In conclusion….
The tool is still very much in the early stages of development. Feel free to use it, but be wary that things may break between versions! If you’re feeling particularly adventurous, merge requests are more than welcome…

Improvements currently on the roadmap include:

• An updated, friendlier user interface.
• Versioning for the schema, including a tool to move data between schema versions. This should help with the long-term preservation for data.
• Support for image uploading. One major weakness is the need to take images seperately and link them later.
• Add a README to explain installation and usage.

The source code is available at https://github.com/Spoffy/OpenGather

(Edit) Next week Eventually, I’ll be talking about taming QGIS, building tilesets and designing GeoJSON maps!

Skills Demonstrated
– User Consultation
– Specification Acquisition
– Adding Functionality to Legacy Code

This was another successful week working on choices. Just like the previous six weeks. Due to my mangers spontaneous expedition to Iceland, Docpot was put on hold. I’ve been to Iceland before, didn’t take me a whole week to get my groceries though…(Sorry).

The medical department uses a convoluted method to allocate students to their courses. This requires a separate allocation algorithm due to the uniqueness of their demands. Last year they split the students into three cohorts depending on their background. Some students were able to pick a language in both slots; some could not as one slot was pre-allocated. This needs changing now. It turns out that choices has finally forced me to make an appointment in the hospital. Meetings are much more productive when all parties are in the same room, even if that room is in a dark corner of a large city hospital. The medical department was a pleasure to work with. It’s refreshing when an agreed solution doesn’t need a complete reworking of existing code.

Kev and I split the work between ourselves. The endgame was to ask questions about the language options so students were aware of the requirements. There were three difficulties of a language: beginner, intermediate and advanced. Each required varied qualifications. We needed to implement questions which are only asked when a student pick specific options. If the student does not pick the option, the answer should be set to a default false value. Kev worked on making actions to add choosers to groups depending on their options. I made the changes to the add and edit questions pages to allow a admin to add this functionality to their form. Using what I learnt last week, I refactored the questions controller so it was easier to work with. I cannot stress the usefulness of testing enough!

We got a working demo working by Friday. Kev was showcasing it to the medical department this morning. There are a few minor bugs to fix but they seemed happy with our progress.

I am now over half way through my internship. Choices starting to look healthier now. I know I say every week but I am looking forward to when i can move on from choices. This week TIDT are doing WAISFest, which is a hackathon hosted by the university. It should give me a good enough reason to escape choices. We shall see…

Skills Demonstrated
– Integration Testing
– Refactoring Legacy Code

So week 5 was the tail end of front-end fortnight, what is that you ask? Well I can tell you in layman’s terms as I was the only one in the whole team who had no involvement with it. Our team had two weeks to figure out which front-end framework they would use for the foreseeable future. That left me pretty much to my own devices for the duration. That meant choices for me – there is always something to fix in choices.

The more astute of my readers will have noticed that this blog post covers two weeks. These few weeks have left my blogging ability rather diminished. But I digress.

It is now where I would like to link to you my managers recent blog post “Zen and the Art of Legacy Camp Site Cleaning”. I paraphrase: “Bolting unit tests onto legacy code is pretty much impossible. Untested legacy code which has been chugging along without issue should not be rewritten. The trick is to leave the campsite cleaner than you found it”. There were two incomprehensible behemoth functions in the options controller totalling about 400 lines. Much like choices, 70% of the universe consists of dark energy. No physicist can explain what dark energy is, or how it got there; only what it does. This is where integration tests come in handy. Unlike in unit testing, the controller communicates with fake databases called fixtures. This allowed me to give this pile of code a data set and then examine how it was digested at the other end. I assumed the add and edit functions worked before I touched them and proceeded to make many integration tests using this method. It was important that I had good code coverage in the tests before starting the cleaning process. Good testing allowed for quick refactoring thanks to immediate feedback.

Since pushing choices to live, we’ve had even more requests from users. And so it goes on. A user found a bug where the ‘all pages’ button on the paginator was dysfunctional. The code for the all button was commented out, it appeared as though somebody couldn’t get it to work but then forgot to remove the button from the view. The fix was actually simple. They tried setting the options per page limit to 1000 to make the all button work. This would have worked but there is a sneaky cake setting called ‘max-limit’ which is set to 100 by default, this overrode the 1000 limit.

There are few more things which I fixed and implemented on choices this week. But this blog post is already getting rather long for my liking, and most likely yours so this is it for now.

These instructions have been deprecated, see the new version available here.

This is a quick(ish) instructional post on how to gather open data in Southampton. This is written assuming you’re using the Open Gather tool. It covers the sort of data we’re looking for and how to gather it.

Objects we’re looking for:

• Drinking water dispensers (Fountains, coolers, etc)
• Gender neutral toilets (Toilets a person of any gender could use, e.g most disabled toilets)
• Portals into buildings and between them (E.g Doors)
• Public showers a cyclist could use
• Reception Desks (And any other Points of Service)
• Images of University buildings (that we don’t already have)

It’s a bit like a game of eye-spy (or Pokemon Go). The aim is to hunt round, find each of the items above and record info about it. For some of them (Portals, Building Images) we know the data we’re missing. For the others, we have no idea, so a little urban exploration may be required!
For all of these, we’re interested in where they are. This involves a building number, floor and geo-location for most of the above. Open Gather has a clickable map to allow for precise geo-location. Otherwise, http://lemur.ecs.soton.ac.uk/~cjg/clickymap/ is available.

How to Gather Data

Preparation

• If you’re hosting your own copy of OpenGather, make sure to clear any testing data out first!
• If you’re recording portals and building images, it can be helpful to plot the things you’re looking for on a map. If you’ve retrieved a list of items using SPARQL, you can use the following to plot the items on the map.
  • Run a SPARQL query to generate the list of missing items, complete with latitude and longitude (examples to come soon!)
  • Generate a KML/CSV/GEOJSON file from the data produced by the SPARQL endpoint.
  • Host the KML/CSV?/GEOJSON file in a publically accesible location. I prefer Git, but Google Drive or an online pasting tool like Pastey also works.
  • Using umap (http://umap.openstreetmap.fr/en/) as a mapping tool, add a layer, then either import the data from the remote, or in umap, add as a remote data source.
  • (When using Umap, tick “Use Proxy” to ensure the icons load correctly)
• Print off a copy of the map. UMap doesn’t work very well in mobile browsers.
• Print off Univeristy of Southampton photograph consent forms. These are needed to use photos with people’s faces in.
• Make sure your phone and camera have adequate amounts of battery (ideally full).
• If your camera and phone are different, check the two clocks show the exact same time. This makes matching images and data far easier.

Overall Process

1. Pick a location on the map and decide which buildings to gather data from.
2. For each building, gather the data needed, using the instructions below.

General Method

This is the quick-and-easy summary of how to record data. More specific information is available below.

1. Open the OpenGather tool in your browser.
2. Select the type of object to record.
3. Fill in the appropriate fields.
4. Submit the data.
5. Take a picture of the object.

Taking a Building Image

1. Using the open data tool, select the category “Building Image”.
   • Fill in the “Building Number” field.
   • Wait for the GPS to update to the current location.
   • If the accuracy is low (say, less precise [higher] than 6m), click/touch the map to mark a more accurate position.
2. Take a picture of building, attempting to get as much of the building in frame as possible.
   • A good photo will make the building easily identifiable as you walk past it.

The geo-location data isn’t necessary for buildings that are already marked on the map, but it helps automatically match images to names later on.

Gathering Portal Data

Walk around the building looking for entrances. Try to identify all entrances that aren’t fire escapes (which we aren’t permitted to gather as of 14/07/2016).

For each entrance that you find:

1. Select the category “Building Entrance” in the OpenGather tool.
2. Record the geo-location of the entrance. This can be done by tapping on the map in the OpenGather tool.
3. Fill in the fields using the tool.
   • “Building Number” – Number of the building the entrance is attached to
   • “Entrance Label” – An arbitrary letter to identify the entrance. Typically starting from ‘A’.
   • “Description” – A brief description of the entrance, such as “Staff”, “Main”, “Side”, “North-east”.
   • “Access Method” – Is a card or key needed to get in?
   • “Opening Method” – How do you physically open the door? This is used to determine disabled accessibility.
4. Submit the data
5. Take a picture identifying the entrance. A good photo will make the entrance easily identifiable as you walk past.
6. Follow the procedure for getting consent, if any people are in your photo (an ideal photo has no people).

Recording a Drinking Water Source

Should one of these rare and majestic creatures be spotted:

1. Throw a greatball at it
2. Select the category “Drinking Water Source” in the OpenGather tool
3. Fill in the fields using the tool.
   • “Building Number” – Number of the building the water source is in
   • “Floor” – The floor the water source is on, level 1 is usually the ground floor.
4. Record geo-location using the map. Zoom in on the building you’re currently in, and try to mark your position in the building by clicking on the map.
5. Submit the data
6. Take a picture of the water source. Ideally, this will make it clear where the water source is located in that part of the building.
7. Follow the procedure for getting consent, if any people are in your photo (an ideal photo has no people).

Recording the location of Public Showers

1. Select the category “Public Showers” in the OpenGather tool
2. Fill in the fields using the tool.
   • “Building Number” – Number of the building the water source is in
   • “Floor” – The floor the water source is on, level 1 is usually the ground floor.
   • “Room Number” – Room number of the shower, if it has one.
3. Record geo-location using the map. Zoom in on the building you’re currently in, and try to mark your position in the building by clicking on the map.
4. Submit the data

Reception Desk (Point of Service)

1. Select the category “Point of Service” in the OpenGather tool
2. Fill in the fields using the tool
   • “Description” – What the Point of Service is. For example, “Library Reception Desk” or “Student Services Information Desk”
   • “Building Number” – Number of the building the point of service is in (assuming it isn’t a standalone service)
   • “Phone” – A phone number for contacting that point of service, if available.
   • “Email” – An email for contracting that point of service, if available.
   • “Opening Hours: Mon…etc” – Opening times for the point of service for that day. E.g “9:00-18:00”.
3. Record geo-location using the map. Zoom in on the building you’re currently in, and try to mark your position in the building by clicking on the map.
4. Submit the Data
5. Take a picture of the desk. It’s nice to have a friendly receptionist in the photo if possible, but don’t force anyone!
6. If anyone (including any member of staff) is in the picture, follow the procedure for gaining consent.

Requesting Consent
Attempt to get nobody in the shot, unless you’re taking pictures of a reception or Point of Service stand, where behind-the-counter staff can make it look friendlier.

If people need to be in the shot:

1. Verbally ask permission before taking the picture, explaining that you represent the Open Data Service, and what that is. Ensure they’re okay signing a consent form.
2. Take the photo.
3. Ask them to fill in an entry on the consent form.

Cross buildings off as you go, to mark them as completed.

Week 5 already? It feels like the time has flown by. This is a particularly interesting week for me, as Chris is away story-ing. I have plenty to do, but it all needs doing under my own steam. That’s why this is a fantastic time to talk about time management. More specifically, how woeful I am and how to improve.

Thoughts on Time

To begin, I want to talk about task lists. An easy-to-use task list has single-handedly made the biggest difference to my time management. It’s done that by ensuring I never forget a task. I can always see the tasks I have to do, prioritise them and set deadlines. By prioritising them, I can always work on the most important thing.

The trick to using a task list effectively is to put everything on it. Every little job, no matter how small, needs to go on it. It needs to be an authoritive list of everything that needs doing.

When the list contains everything, you can work purely off the contents of the list. You just pick off the most important task you can have time to work on and get to work.

My two biggest time management issues are tunnel vision and getting distracted. The former is where you get caught-up working on one thing. The mistake is not stepping back and asking “What’s the right thing to be working on?” periodically. As for getting distracted… it’s hard not to, in a lab filled with shiny objects and fantastic people.  Hopefully, these will be solved in later blog post.

Work for the Week

This week, I’ll be mainly working on creating a SPARQL URL checker. This will build a database of all the URLs a SPARQL endpoint knows about (URLs in this context meaning website addresses, rather than RDF URIs). It will then launch requests to each of those URLs, reporting the status of each. The aim is to identify any broken links that need repairing.

Chris and I spent Friday planning the system, which should look something like this:

A high-level plan of SPARQL-Detective

The code will be available at https://www.github.com/Spoffy/SPARQL-Detective in the near future.

I’ll also be working to implement some of the changes to OpenGather I mentioned in my previous blog post. The main focus is to implement a schema for each type of data to be gathered.

PHP – Or as I now call it, “Interpreted C”

As promised, a short rant. So, this internship has been my first time working with PHP. I used it for the OpenGather tool, and up until now it hasn’t been so bad.

However, I have since been introduced to the joys of the cURL library. Here’s a short snippet to query a URL in cURL.

$curlHandle = curl_init($url);
//Force it to use get requests
curl_setopt($curlHandle, CURLOPT_HTTPGET, true);
//Force a fresh connection for each request. Not sure if this is needed...
curl_setopt($curlHandle, CURLOPT_FRESH_CONNECT, true);
//Get Headers in case we need Location or other.
curl_setopt($curlHandle, CURLOPT_HEADER, true);
//Attempt to follow redirects
curl_setopt($curlHandle, CURLOPT_FOLLOWLOCATION, true);
//Do we care about SSL certificates when checking a link is broken?
//...Possibly if there's SSL errors. V2.
curl_setopt($curlHandle, CURLOPT_SSL_VERIFYPEER, false);

//Don't actually care about the output...
ob_start();
$result = curl_exec($curlHandle);
ob_end_clean();

$link_status[$url] = curl_getinfo($curlHandle, CURLINFO_HTTP_CODE);
curl_close($curlHandle);

Oh, sorry, did I say short? I lied. This bit of code summarises my thoughts on PHP perfectly. The code is verbose, unnecessarily so. It’s missing high level abstractions (You have to manually parse the returned string for header data). And it all somehow feels… clunky. Thankfully, at least memory management isn’t a problem.. right?

For comparison, here’s the same snippet in Python.

link_status[url] = urlopen(url).getcode()

….The joys of cURL!