User testing results: a simple analysis

The series of blog posts on user testing requires some summary, even if it is best not to be over-analytical about the results of user tests, especially if you were responsible for designing the tests. That is perhaps best left to others, and this post will include some points highlighted by Dave Tarrant, the developer of one of the deposit tools tested.

First, let’s recap some of the key issues, constraints and highlights of the user tests already noted in earlier posts.

The first post on the tests made the following claim: “What I can tell you now is that we had no disasters in organising the tests, from my point of view, and I think I would have known. Therefore, like an auditor, I shall present the report as a valid representation of the tools under test.”

We were clear about what was being tested: “In this case we are testing the process of repository deposit using two new tools, that is, it is a usability test. In these tests we will not be testing the users’ capacity to install the tools themselves, but to use tools already set up for use.” Interpretations of the results have to take care not to stretch the limits of the test, for example, we can make no claims about the wider applicability, uptake or impact of the tools, simply the ability of users to use them for the specified tasks.

“User testing depends on providing clear instructions to users.” By providing the full instruction document, not only does this show what users were asked to do in the tests, but the clarity of the instructions and the possible impact on the tests can be judged. The structure of the test also points directly to the structure of the results: what users said in response to before-and-after questionnaires, what users did in the test, times taken to perform the specified tasks and completion rates.

We learned from the before-test questions that users included “a mix of experienced and new repository users, a reasonable profile for this test since the tools are aimed at both types of users.”

With reference to times and task completion we were able to compare the relative performance of the two tools under test, the Word Add-in and the Watch Folder: “The total time taken to complete image deposits was in all cases longer than the time taken to deposit and update Word docs. There was more variability in times of image 1 deposit, but less so for subsequent image deposits when the procedure was more familiar. Some image deposit cases were not completed, but all doc deposits were completed.” This is perhaps the result that might attract most attention, and is the area that Dave Tarrant focusses on below.

The detail of the test results can be found in what users did. We had two ways of recording what users did, based on the repository record indicating process and the degree of task completion, and notes taken by a test observer providing additional insights on process and spontaneous user reactions.

But what did the users say about the tools they tested? The after-test questions gave us some insights: “These summary results suggest that on balance use of these tools might encourage more deposit, just, most clearly in the case of the Word tool, but it won’t be so easy to wean users off the standard repository deposit interface on the basis of these tools.”

Developer’s view of the test results, by Dave Tarrant

Contradicting the mantra “repositories are built for finished publications”, the ‘own’ content brought to the tests for deposit by these users was mostly content that might not be considered as formal ‘publications’. It is these more diverse content-owning communities that are least supported by current repository software. As we see demand for data publication in repositories, for example, the types of content deposited in repositories is only going to grow beyond traditional publications.

During the testing carried out in DepositMO, users were asked to use both the new clients and the existing repository interfaces to deposit a number of different content types. The results from these tests are clear: on average, both direct deposit clients (Word Add-in and Watch Folder) took less time for deposit.

Using the Microsoft Word Add-In developed as part of DepositMO, the average deposit time for a document (from opening Word to completed deposit) was done in less than half the time that it took to deposit the same item via the native repository interface.

The Watch Folder client also resulted in speedier deposit, with a 12% gain in time to deposit over the standard repository interface. In its current form the Watch Folder client does not provide the simplest means of metadata control and thus in some cases additional time was incurred to complete this stage using the repository metadata interfaces.

Some of the quotes recorded during testing suggest at an attitude change is possible, with one user finding the experience “Quite fun”. Over 50% of the users reflected that the tools would encourage them to submit more of their own content to the repository.

With users who are already experienced in repository deposit it is clear that combining new and existing tools into one experience can be confusing, particularly as a user’s collection size grows. All users suggested that the tools “need some work”, specifically in informing the user about the actions being implemented both locally and remotely.

Omission of DSpace user testing

The results reported apply to a controlled test environment “with Web-connected laptops running Windows 7 and connecting to a demonstrator EPrints repository running the SWORDv2 extensions.” Similar extensions have been added to DSpace, and we intended to run the same user tests with a demonstrator DSpace repository as we had with EPrints. We had a possible test site and users lined up. In pre-tests, the setup ran successfully with the Watch Folder tool, but we were unable to to complete a deposit process with the Word Add-in tool and the DSpace demo repository. What this shows is the complexity of managing and synchronising a communication process between a series of tools – Word Add-in, repository software, repository instance, SWORDv2, DepositMO extensions – where three of the components are new and under test for the first time. Perhaps we should rejoice that the EPrints user tests worked at all given these odds. This would have been resolvable with more time, but the project had already been granted an extension to complete the SWORDv2 and DepositMO extensions for DSpace, and further time could not be justified. We decided against a shortened DSpace test with just the Watch Folder tool.

We can draw no further implications from this, although we note that Marco Fabiani at Queen Mary University of London, ironically and by complete coincidence our target DSpace test site, has reported issues with installing the Watch Folder to work with DSpace. Users did not test installation of the deposit tools, and this is known to be an area requiring further attention, particularly for the Watch Folder.

Summary

The wow! factor seen during a demonstrator presentation is, unsurprisingly, harder to sustain in practice, and noticeably harder as a user’s collection size grows. It’s not just about the initial deposit. As collection size grows, issues of metadata control and versioning become more critical. This is why there are no simple comparisons to be made between the instant deposit tools (Word Add-in and Watch Folder) and the more structured native repository deposit interfaces. Instant deposit may be at the expense of providing careful metadata now, although further refinements to the tools might be able to improve metadata control without losing deposit time.

The critical point is not in the comparisons. We can now see more clearly how different repository deposit tools can support different users with different deposit demands – “these more diverse content-owning communities that are least supported by current repository software” – widening the base of repository users. With the growing emphasis on research data management, especially using data repositories, the need for choice in repository deposit – offering tradeoffs between time to deposit and degree of documentation – is only going to become more acute. After all, repositories wish to acquire more content without adding unnecessary barriers to deposit.