User testing results 2: times taken to perform tasks

Based on the tests presented to users we have a number of ways to analyse the results:

1. Times taken to perform the tasks
2. What users did, based on the repository record and observer notes
3. What users said, based on feedback from the test form

Subsequent posts will reveal what users did and said. Here we focus on what the timings and task completion tell us about the usability of the deposit tools.

User tests were performed over a series of dates during June and July 2011. Some tests were simultaneous and co-located. In total 13 users performed the test in 6 pairs and one singly (recall, tests were set up for pairs, to provide the opportunity for reactions to be vocalised between users, and to avoid equipment limitations for the controlled test environment).

Users 1, 2: June 3
Users 3, 4: June 17
User 5: June 20
Users 6, 7: July 5

All users completed the same tasks, guided by the same instruction document. Minor amendments were made to the document between tests to improve and clarify procedure, as pointed out by users, where problems were deemed to be due to the test documentation rather than the tasks and tools under test.

Procedure For reference, this is a quick chronology of the deposit test process outlined in detail in previous posts:

• Deposit a Word document (using Word tool) Doc 1
• Deposit an image (using file manager tool) Image 1
• Deposit a pdf using the standard EPrints repository interface (PDF)
• Deposit a second Word document (tool chosen by user) Doc 2
• Deposit a second image (tool chosen by user) Image 2
• Update the second Word document (Update doc)
• Add an image to a collection including the second image (Add image)
• Deposit own content (Own content)

Results

Time taken to perform set tasks

Table 1 shows times grouped by process and tool rather than chronologically.

Table 1. Time taken by users to perform set tasks in deposit test process

Notes on Table 1

Times were recorded by observers for selected stages in the overall process. The repository timestamps, including the EPrints history for each record, were used to calculate times of individual tasks.

Where a time duration is unknown, or where a time range is given, the observer notes are incomplete.

Where no time is given, the procedure was not completed.

Where the process appears to be continuous, it is assumed that the last recorded deposit time for an item is the start time of the next deposit process, even if the first recorded deposit time for the subsequent item is some minutes later. This assumption is based on the need for users to assess the instruction and decide which actions to take, and that the time taken to do this is an implicit part of the deposit process for each item.

Not all user processes were continuous or followed the order of the instructions. Where this happens timings are likely to be less reliable, based on an assessment of the record and a judgement of how users proceeded. Such instances will be specified.

e.g. User 1: the deposit of images and the PDF appear to be interleaved, so apportionment of the times between images and pdf are approximate.

Deposit time ≠ Total doc + Total image + PDF + Own. Allow ± 1 min for each action.

Deposit time ≠ Total test time. Additional time is taken to read the documentation for the procedure, and to complete questions before and after the deposit process.

Repository clock set 1h slow. (This is only relevant when comparing observer timings with repository record.)

General observations on times

The total time taken to complete image deposits was in all cases longer than the time taken to deposit and update Word docs. There was more variability in times of image 1 deposit, but less so for subsequent image deposits when the procedure was more familiar. Some image deposit cases were not completed, but all doc deposits were completed.

In the next post we will look at what users did, based on the repository record and observer notes.