User testing results 3: what users did

Based on the tests we have a number of ways to analyse the results. Previously we considered what the timings and task completion tell us about the usability of the deposit tools. Here we get some insights into the times taken to perform the tasks by using the records of what users did, taken from the observer notes and an assessment of the repository records they created.

For completeness, and since it did not seem appropriate to break the post between users, this is necessarily a much longer post than usual on this blog.

User 1

Pair. Experienced repository users, many deposits of a variety of content types over years.

From repository record

Hard to match record with observer notes and intended test chronology. Ended with 5 deposits of doc 1. Took a long time to deposit image 1. Appears to make some false starts with image deposit, did PDF, then returned to image 1 and ended with a collection of 4 images, including all 3 images provided in the sample data for the test and an additional image not provided. Deposited doc 2 (no metadata) and updated by adding a new item (with metadata).

Extracts from observer record

Deposit item 1 (doc 1) Clicked submit x2 so resulted in 2 deposits.  “Would have expected an update” – both agreed. It’s like a ‘save’ button – expect new copy of the doc. Item type – should be doc not unspecified

Deposit item 2 (image 1) Didn’t work first time. Then did work when refreshed screen for manage items = Revision 11 though! But there doesn’t seem to be anything actually there? Ghost image and also actual file. Went back 3^rd time for item 2 -> added another image as the revision of the first Separate folder for each item? How would it tell which version? Well that worked = Rev 14 did what was expected

PDF Subjects -> LC/Dewey – choice? Tried to search for several subjects.

Image 2 Seems odd that would need to create a separate folder for each item. If I have to create a new folder for each item I am never going to use this!! Cut item from original folder. Waited for metadata file. Revised metadata – abbey by night. Resulted in a deposit with 4 files

Doc 2 MS Word error message about the file because it seemed to take the title and corrupt the file. Going back and try again – create new folder. Doc not corrupt at moment. Used short cuts to copy and paste -> my doc which I’ve spent months working on may have been corrupted … Titles seem to be switching around … and the doc seems to be fine when opened up in the repository

Updating doc Error message – forgotten details which had been provided – username and password but no location -> went to item which was locked. Hopelessly lost. No confidence about which document we are working on

Click on’ Submit’ – then this created another copy Characterising and preserving

How does it know which doc I want to update rather than creating a new version?

User 2

Single. Experienced repository user, many deposits of a variety of content types over years in different repositories.

From repository record

Appears to have completed all stages in correct order, and deposited a substantial number of own items including a variety of document and format types, although the collections appear to have become mixed within records, some duplication of items. All repository records can be identified by title, with supplementary metadata added to describe items within records. Looks like an exemplary result.

Extracts from observer record

Opening Word user panel. Missed (the Word deposit tab) at first; looked at repository. Needed prompt to ‘show’ the panel.

Image 1 Had closed depositMO folder after first use. Eventually realized it had to be reopened.

Doc 2 Reset user panel correctly. Added more metadata through EPrints deposit interface

Image 2 Named folder (‘Bath’) this time. Again added more metadata through repo. interface.

Update doc ‘lost some of my added metadata’ e.g. item type. Went back to add again

Add image User waited to see if more VIEW, METADATA files were generated. Noted they didn’t. ‘lost metadata again’. Added again through repo. interface

Own content Deposited 2x pdf + ppt via file manager tool. Added metadata via repo. interface. Added Word doc via panel, added metadata. Then copied ALL own content into one folder. Added one extra item to collection which overwrote previous title and added title of new item.

Parting comment: ‘Like Word’. Would like other (Office apps). That would encourage. Don’t like losing metadata.

User 3

Pair. One is an occasional repository user, the other a non-user.

From repository record

We don’t have an end timing for the short break in the process prior to update doc, hence the span of time for this task and for the total deposit time. A lengthy time for first deposits of both doc and image. No image deposits can be found in the repository workspace for this user. On inspection, these deposits can be traced to the workspace of User 1, probably due to an issue with the user configuration file in the ‘watch folder’, which directs content to the user’s login space in the repository. Image 2 was at first added to same folder as image 1, but then deposited separately in own folder. The docs and pdf do not use the config direction and appear correctly in this user’s repository workspace, with a duplicate entry for doc 1, all identified with metadata. No own content.

Extracts from observer record

Opening CONGIG file. Doesn’t automatically open – users had to open this in Notepad

Opening Word user panel. No label to say it is ‘user panel’. Author panel had a label, which caused some confusion.

Resetting repository location in Word user panel. My testers missed out this step, so the process didn’t work.  They received an error message ‘couldn’t understand response from server endpoint’ and realized their mistake.

Setting login in Word user panel. Password defaults to admin – confusing.  No confirmation when new password is entered to say that the username and password works.

Doc 1. Two identical records were added to DepositMO. . Users updated one, so there may be two on the screenshot with different times. Some confusion over ‘item type’ and what this meant. The testers had a discussion at this point about metadata, and how the repository found the title, as it wasn’t the title of the document.

Image 1. Testers not entirely sure where the depositMO folder was. Users said they didn’t understand what the script was meant to do.  They misunderstood and thought they had to copy the image into the repository itself. On adding metadata to the image, testers were disappointed with this – they didn’t understand exactly what they were doing.  They expressed doubts that academics would do this (XML text edit). Image 1 didn’t appear in the repository, although clicking on the View_item.html took them through to the appropriate record.  It appears that drag and drop was adding the record to the ‘user1’ account, even though they had updated the Config file correctly.

Doc 2 Deposited using the Word App

Image 2 Deposited using the Drag & Drop method. The testers didn’t understand the point of the different folders. Testers didn’t set up different content folders at first, but realized their mistake and made a second attempt.

Testers had remembered most steps at this point. Testers commented that there are too many steps, it is not straightforward, and it is not that different to the existing eprints interface in that regard.

Add image. Testers didn’t have time for this, but had inadvertently done this already.

Testers queried again how much more metadata you would have to add if using the Word app and the Drag and Drop options.

User 4

Pair. Both are non-users of repositories.

From repository record

As with user 3, which was simultaneous with this user test, image 1 and the added image appear in the repository workspace of the user 2 account, although this was avoided for deposit of image 2 by using the conventional EPrints deposit interface. Image 3 was added to image 1 and so also ended in user 2 account. As with user 3, this may partly account for the time taken to deposit image 1. All repository entries have at least basic metadata description and are identifiable in the list of records, including two items of own content but excepting the drag-and-drop items that ended in the user 2 account.

Extracts from observer record

Doc 1 “Show’ panel not obvious – “need explanation” Prompted

Image 1 Just deposited item, not in list of records but can view item

Doc 2 Word user panel – need to update again “find it hard to remember what have done”

Image 2 Used EPrints deposit interface

Update doc Went to repository ‘Edit item’ button. Prompted – back to Word panel. “Check if it worked”

Own content 1 Word doc – deposited, then added more metadata through EPrints deposit interface

Own content 2 pdf – using EPrints deposit

User 5

Pair. Test users only to date, but expect to be more regular repository users in future.

From repository record

An unspecified item appeared in this user account just prior to deposit of doc 1.

Image 2 is described, but image 1 with added image 3 are not.

Unusually for these tests the pdf is fully and correctly described, this taking slightly longer on this task than other users.

A large amount of own content was added using the file manager tool. These were packaged content – a ‘web site’ and zipped archive – and the aim was to see how the tool extracted and presented the contents. No metadata was added.

Extracts from observer record

Doc 1 Two unspecified objects in repo at this point + deposited doc. Picked up metadata from doc inc abstract: “Good”. Does doc need to be “tagged” for this to happen? “Good to have item type.” “Could improve description.” Started to add metadata via EPrints interface – “off piste”.

Image 1 Copied doc by mistake. Deleted – but it returned. Then successfully  copied intended image. “Quite fun.” “Could be more user-friendly – populate metadata rather than XML.”

PDF Couldn’t copy-paste abstract. Worked 3^rd time. Added metadata carefully

Doc 2 Want Word deposit set to default URL. “Like the way it flags up” missing metadata. Checked what’s there/not there: “still quite a few required fields not populated”

Image 2 Chose to use EPrints interface – “like to be able to add keywords, contextualise, but don’t know creator, etc.

Update doc Users checked this replacement but couldn’t identify which document to open at first

Add image This instruction assumes they would have used the file deposit tool to add image 2 – they hadn’t (see above). Added image to existing folder. Then didn’t know which item it had been added to – at first.

Own content “Quite like to test uploading a Web site.” The file tool grabbed lots of pdf files. Zip file: “Maybe it doesn’t like zip files”

User 6

Pair. Mixed experience pair: one had not previously used a repository; the other has used a media and image repository rather than an open access institutional repository such as EPrints.

From repository record

We do not have definitive observer time checks for start of deposit, hence the span of time for doc 1.

Doc 1 is listed with a title, but there is no other metadata in the record and full-text is not available.

Image 2 was deposited twice, accounting for the extended deposit time for this item. Added to same folder as image 1, then deposited as separate item as required. Title metadata added

PDF added with minimal metadata.

Own content included a large video (700 MB) and further attempts to deposit image 1 (an archaeology picture by a colleague)

Extracts from observer record

File manager deposit tool setup: user A didn’t understand the config file set up although user B went straight to it and just did it with no comments

Doc 1 Options did not seem straightforward, there was clicking to find things and then going back. “We can’t follow simple instructions” – they are too impatient they think. I think they were looking for a way to upload a new description but couldn’t find one. There is confusion around the terminology of “submit” and “deposit”. Tried to submit it twice because of confusion but recognised that when they got a warning or error message.

Image 1 Confusion about folders for deposit but reference back to the instructions helped. Some uncertainty about what to put in the title

Doc 2 Error message for address to deposit – some confusion until the right place found. Some confusion from two people working together not knowing what button the other has pressed. Some problems when didn’t submit properly. They tried to submit again. Some confusion about how much information you HAD to put in before it would allow you to submit. What does “user work area” mean?

Image 2 Managed to submit things and then made the new folders which was considered to be confusing. Metadata file appears to have been rewritten – it did not contain the information they had submitted. Different file structure for this part of the test. It takes a while for the metadata file to be written. The metadata options generated some uncertainty with data deleted or overwritten and so the required things to enter could not be found. They lost the object that they had put in by putting it in the same folder as something else. They put it in again. Considered to be very fiddly, and creating a new folder for every file at this speed is too time consuming with the delay in the creation of the metadata file. AND then another object appeared in the folder – seemed to be a serious time lag for this to happen i.e. reentering stuff that is there but hasn’t appeared yet… “This is evil”.

Update doc Success first time.

Add image Worry and concern that this would break the system again! But some uncertainty about how this as a system would work, i.e. illogical? Problem – they think that the program keeps rewriting their metadata. The program is confused about the files and the metadata. Trying to sort it out took some time – Had to use the details to work out what was happening.  Altered through the web browser (EPrints interface) which a takes a long time to update the metadata file itself.

Own content Using the EPrints interface as it was considered to be the easiest way to do it. Video took a long time to upload but no problems.

User 7

Pair. Both are regular and experienced users of the Blackboard repository, and have deposited a range of content types.

From repository record

An ‘unspecified’ and unexplained record appears first, as for some other users. The first doc deposit has metadata, but doc 2 unusually has no description, although it is updated correctly. The PDF is deposited with made-up metadata. All texts are deposited into the repository review area. From the summary timings, only the image addition is notably longer than other tasks. Deposit of image 1 appears to be unsuccessful, but it appears with image 2 (not according to instructions), both attempts listed as ‘unspecified’. Then image 2 is joined with image 3 (as instructed) but with the title ‘Waterwheel’. Finally, own content is a series of 6 images collected, again, under the name ‘Waterwheel’.

Extracts from observer record

Setup demo repository Clicked ‘New Item’. Added an unspecified object?

Setup file deposit tool, open watch_folder Incomplete

Image 1 Needed to advise opening watch_folder. “What are the unspecifieds that keep popping up?”

PDF Performed deposit without reference to the actual document provided

Image 2 Copied to same folder as image1, then set up new folder. Added metadata via XML page (did not display)

Update doc Opened new copy of doc2 in new Word window. Update didn’t work. Not previously submitted version – opened new? Found path in original eprint, updated successfully. “Hasn’t identified title” Wiped metadata, “really annoying”

Add image “went in straight away”

Own content 6 images copied in one folder. “Worked well”. Opened XML metadata page for folder. Then added metadata via EPrints form.

Found there were two items in repo! Renamed wrong item. Re-copied. Deleted generated metadata.xml. Failed to recreate. Copied again. Added metadata via EPrints form.

The next post will show what users said about the test and tools they used, based on feedback from the test form completed immediately at the conclusion of the test.