Unpacking large image collections: Polynomial Texture Map (PTM) Extension workflow and implementation

This project’s YouTube Import Plugin performs what we called the ‘grunt’ work of large file import and metadata completion, saving the depositor time and effort. A video import will typically be a single item, but if your large file contains a series of images, say up to 100 images, how would the repository organise, describe and present these? That was the problem posed by archaeologists at the University of Southampton. To begin to answer it in this post we describe the workflow in use of another tool produced by Tim Brody for the DepositMOre project, also available in the EPrints Bazaar, the Polynomial Texture Map (PTM) Extension.

If this sounds like a more specialised app, it is, but one which nevertheless demonstrates some exemplary features for wider deposit applications that involve the capture and description of multiple objects within a single repository item. It is deposit at-scale issues like these that will characterise the extension of institutional repositories to handle research data outputs, of which the archaeology case here is one example.

What is a PTM?

First we need to know what a PTM is and explain the process by which these are produced. The detail of objects from archaeology finds are intensively imaged and documented in situ in the field or in the lab. One can imagine a series of camera shots from different angles. In fact, an object is instead photographed using a fixed-position camera, using a camera rig or dome-like apparatus, while the position of the light source is varied, thus recording a series of surface details by a process called Reflectance Transformation Imaging (RTI). A good explanation of the process can be found in this paper.

If the hardware is clever, so is the software, which processes the recorded images in a sequence of output formats:

Camera-raw -> digital negatives -> JPEG (for processing) -> RTI (using RTI Builder) -> PTM archive

The series of camera images is first processed to produce conventional JPEG. This is then transformed into an RTI using special software and can be disseminated as a PTM archive, which is what we use to upload to our repository in this case. For our purposes, RTI can be seen as a general case including PTMs. The critical step performed by the RTI software is to transform the image set into a form that supports analysis, description and presentation. What we need to do is enable that functionality from a repository that additionally supports secure long-term storage and distribution.

Uploading and unpacking a PTM archive in a repository

Let’s see what we have when we upload a PTM archive to a test EPrints repository without the PTM Extension app. The repository recognises a zip file, just like any other compressed package, but knows nothing about what is inside or how to unpack this file.

For the next stage we will install two apps from the Bazaar on our test repository. First, the ReCollect app to enhance the use of the repository for collecting standards-based metadata for describing research data. We will classify our images as research data. This app adds a New Dataset button to begin to deposit research data.

Second, the PTM Extension is installed. This time when we deposit our data archive it is recognised as a PTM, as shown by the appearance of a PTM action tool to the right, among the familiar action tools available for an EPrints file.

If we click on the right-hand PTM icon it offers the chance to Unpack the PTM archive into its constituent components.

Below we can see those unpacked components. This is a relatively simple archive, just three components, but a typical archive could contain many more.

If we look at the metadata for the dataset, in contrast to the video import example which had largely completed the required fields automatically, here our file-level and dataset fields are empty. File-level information is critical in an archive containing many files and should be available, we are advised, from the XML camera output. We shall discover how more progress can be made towards adding metadata and completion in the next post when we present user feedback on use of this tool for serious PTM archives.

Our archaeology colleagues at Southampton have worked with the project throughout both phases, the original DepositMO project as well as DepositMOre, and acted as testers for the Watch Folder and Word Add-In tools. It was those insights that led, eventually, to the PTM extension tool described here. We are grateful to Graeme Earl, who initiated, inspired and maintained the archaeology interest in this work, and to Hembo Pagi and Gareth Beale for their expert description of the required application, and their subsequent feedback on the implementation of the PTM Extension.