User testing: design of the test procedure

We began this current series of posts with a promise to present the results of user tests of the new deposit tools developed in the DepositMO project. Via a convoluted route we are finally ready to present those results.

In this series of posts we have described the target of the user tests: the DepositMO tools for Word (Add-in) and for the file manager (Watch Folder), and the extended SWORD infrastructure, to deposit in-progress works from the desktop to the user’s specified repository. To design and perform the tests, however, we need to be more precise about what is being tested and how. That is what we will map out in this post, covering purpose of the test, materials to be used, target users, and measures and outcomes.

Purpose. With any user test it is vital to be clear what you are, and are not, testing. In this case we are testing the process of repository deposit using two new tools, that is, it is a usability test.

Ultimately, if these tools are to succeed then they will have to be available to general users in ways that enable them to be downloaded and installed by those users. Here there are particular limitations and difficulties with installing each tool, given the early stage of development reached. So in these tests we will not be testing the users’ capacity to install the tools themselves, but to use tools already set up for use. This is thus a controlled test environment, which was a pair of Web-connected laptops running Windows 7, and connecting to a demonstrator EPrints repository running the SWORDv2 extensions.

Materials. If this is a test solely of repository deposit, what is to be deposited? The aim is to provide users with content suitable for use with the new deposit tools, and to be used as a basis, initially, for exploring the features of the tools. Beyond that we don’t want to lead the user in terms of choice of deposit tool. Also, for comparative purposes, we introduced a task that involves using the native repository deposit interface. For subsequent deposit tasks the choice of deposit tool is left open for the user.

Given that we have a Word deposit tool, we provide Word documents for deposit. The file manager deposit tool is intended to be a simple and general-purpose tool that might be used to deposit materials considered unconventional in current repository collections. Images are perhaps the simplest case here and were provided in the test corpus. Also, as part of the test we invited users to bring their own content, as a way of motivating interest in particular features that users might have identified during the set test.

By the end of the test users would have deposited quite a large and diverse set of materials in the demonstrator repository. This is not a typical deposit scenario for current repositories, but the test seeks to anticipate new ways of working and deposit made possible by the tools and when such scenarios may not be unexpected. Even so, the extent of deposit in the time available for the test may be ambitious and complex, and in this sense could be seen as subjecting the tools to maximum stress.

Users. Who are the target users? They could be any members of an institution with a repository and who might need to deposit content. Since the new tools are intended to open new deposit scenarios, the users could be new or experienced repository users.

In the case of the DepositMO project we have a number of interested groups participating in the project committee, representing different repositories hoping to implement the tools, and academic subject groups representing authors with expanding needs for repository deposit. To be specific, our users were invited and coordinated by representatives for the e-Prints Soton and EdShare repositories, and the archaeology and chemistry departments at the University of Southampton. In addition, we had interest in the tools from the Kultivate project, a companion JISC repository deposit project, and we worked with two representatives of that project as well.

Since the number of users is small, 13 users in total across four user groups, the tests are sufficient to reveal usability issues with the tools, but to draw any further conclusions about repository deposit would be speculation.

Measures and outcomes. A user test needs to have a way to measure the outcomes. From a general perspective the purpose of repository deposit is to transfer a copy of the designated content to the repository in the most timely and efficient manner, to ensure that the integrity of the content is maintained through copying and transfer, and that it is sufficiently described to aid retrieval by a variety of queries.

In this test we focus on the first issue, the timeliness of deposit, to some extent on the description issue, and not at all on the integrity issue as that is not a function of the new tools but is left to existing repository processes.

In most use cases involving new technology or services it is better to measure what the user does rather than what the user thinks, although the latter can be helpful and revealing in the context of the former. So here we have a record of what the user does, in terms of the repository history for that user’s account, and allied to the history timings we independently timed, in quite a rough manner, how long it took to complete each task in the specified test process.

To discover what users thought of the tools we included short questionnaires in the test document, to be completed just before and immediately after the test process. The first questionnaire inquired about the user’s relevant experience with repositories, if any, and the second prompted their reactions to the test and the tools. During the test observers recorded more instinctive and spontaneous, perhaps less considered, reactions. Most users, apart from one, worked in pairs to provide the opportunity for these reactions to be vocalised, since otherwise the observers were not permitted to be directly involved in the test or to influence it in any way.

The test. Taking all the above factors into account we produced the test specification document. Users were invited, by the coordinators, to a test session at a specified time and place, as local to the users as possible. Four sessions were held, allowing up to 1.5 hours for the test each time. As already mentioned, users were encouraged to bring some of their own content for deposit. Other than that, users did not have to bring any equipment. Pre-installed laptops were provided for each user or pair, and they were not briefed in advance on the form of the test.

Apart from some introductory words by the test moderator, the test is intended to be self-contained and self-explanatory. The purpose of the introduction was to put users at ease, and to ensure that the setup and arrangements were sufficient to allow them to focus on the test and not be distracted.

In the next post we will begin to set out the instructions for performing the test as presented to the test users.