{"id":988,"date":"2012-02-03T16:43:16","date_gmt":"2012-02-03T15:43:16","guid":{"rendered":"http:\/\/blog.soton.ac.uk\/depositmo\/?p=988"},"modified":"2012-02-03T17:53:50","modified_gmt":"2012-02-03T16:53:50","slug":"user-testing-results-3-what-users-did","status":"publish","type":"post","link":"https:\/\/blog.soton.ac.uk\/depositmo\/2012\/02\/03\/user-testing-results-3-what-users-did\/","title":{"rendered":"User testing results 3: what users did"},"content":{"rendered":"

Based on the tests we have a number of ways to analyse the results.\u00a0Previously we considered what the timings and task completion<\/a> tell us about the usability of the deposit tools. Here we get some insights into the times taken to perform the tasks by using the records of what users did, taken from the observer notes and an assessment of the repository records they created.<\/p>\n

For completeness, and since it did not seem appropriate to break the post between users, this is necessarily a much longer post than usual on this blog.<\/p>\n

<\/strong><\/p>\n

\"\"<\/a>User 1<\/strong><\/h2>\n

Pair. Experienced repository users, many deposits of a variety of content types over years.<\/p>\n

<\/strong><\/p>\n

From repository record<\/strong><\/p>\n

Hard to match record with observer notes and intended test chronology. Ended with 5 deposits of doc 1. Took a long time to deposit image 1. Appears to make some false starts with image deposit, did PDF, then returned to image 1 and ended with a collection of 4 images, including all 3 images provided in the sample data for the test and an additional image not provided. Deposited doc 2 (no metadata) and updated by adding a new item (with metadata).<\/p>\n

Extracts from observer record<\/strong><\/p>\n

Deposit item 1 (doc 1) Clicked submit x2 so resulted in 2 deposits.\u00a0 \u201cWould have expected an update\u201d \u2013 both agreed. It\u2019s like a \u2018save\u2019 button \u2013 expect new copy of the doc. Item type \u2013 should be doc not unspecified<\/p>\n

Deposit item 2 (image 1) Didn\u2019t work first time. Then did work when refreshed screen for manage items = Revision 11 though! But there doesn\u2019t seem to be anything actually there? Ghost image and also actual file. Went back 3rd<\/sup> time for item 2 -> added another image as the revision of the first Separate folder for each item? How would it tell which version? Well that worked = Rev 14 did what was expected<\/p>\n

PDF Subjects -> LC\/Dewey – choice? Tried to search for several subjects.<\/p>\n

Image 2 Seems odd that would need to create a separate folder for each item. If I have to create a new folder for each item I am never going to use this!! Cut item from original folder. Waited for metadata file. Revised metadata – abbey by night. Resulted in a deposit with 4 files<\/p>\n

Doc 2 MS Word error message about the file because it seemed to take the title and corrupt the file. Going back and try again – create new folder. Doc not corrupt at moment. Used short cuts to copy and paste -> my doc which I’ve spent months working on may have been corrupted … Titles seem to be switching around … and the doc seems to be fine when opened up in the repository<\/p>\n

Updating doc Error message \u2013 forgotten details which had been provided \u2013 username and password but no location -> went to item which was locked. Hopelessly lost<\/em>. No confidence about which document we are working on<\/p>\n

Click on\u2019 Submit\u2019 \u2013 then this created another copy Characterising and preserving<\/p>\n

How does it know which doc I want to update rather than creating a new version?<\/p>\n

\"\"<\/a>User 2<\/strong><\/h2>\n

Single. Experienced repository user, many deposits of a variety of content types over years in different repositories.<\/p>\n

From repository record<\/strong><\/p>\n

Appears to have completed all stages in correct order, and deposited a substantial number of own items including a variety of document and format types, although the collections appear to have become mixed within records, some duplication of items. All repository records can be identified by title, with supplementary metadata added to describe items within records. Looks like an exemplary result.<\/p>\n

Extracts from observer record<\/strong><\/p>\n

Opening Word user panel. Missed (the Word deposit tab) at first; looked at repository. Needed prompt to \u2018show\u2019 the panel.<\/p>\n

Image 1 Had closed depositMO folder after first use. Eventually realized it had to be reopened.<\/p>\n

Doc 2 Reset user panel correctly. Added more metadata through EPrints deposit interface<\/p>\n

Image 2 Named folder (\u2018Bath\u2019) this time. Again added more metadata through repo. interface.<\/p>\n

Update doc \u2018lost some of my added metadata\u2019 e.g. item type. Went back to add again<\/p>\n

Add image User waited to see if more VIEW, METADATA files were generated. Noted they didn\u2019t. \u2018lost metadata again\u2019. Added again through repo. interface<\/p>\n

Own content Deposited 2x pdf + ppt via file manager tool. Added metadata via repo. interface. Added Word doc via panel, added metadata. Then copied ALL own content into one folder. Added one extra item to collection which overwrote previous title and added title of new item.<\/p>\n

Parting comment: \u2018Like Word\u2019. Would like other (Office apps). That would encourage. Don\u2019t like losing metadata.<\/p>\n

\"\"<\/a>User 3 <\/strong><\/h2>\n

<\/strong>Pair. One is an occasional repository user, the other a non-user.<\/p>\n

From repository record<\/strong><\/p>\n

We don\u2019t have an end timing for the short break in the process prior to update doc, hence the span of time for this task and for the total deposit time. A lengthy time for first deposits of both doc and image. No image deposits can be found in the repository workspace for this user. On inspection, these deposits can be traced to the workspace of User 1, probably due to an issue with the user configuration file in the \u2018watch folder\u2019, which directs content to the user\u2019s login space in the repository. Image 2 was at first added to same folder as image 1, but then deposited separately in own folder. The docs and pdf do not use the config direction and appear correctly in this user\u2019s repository workspace, with a duplicate entry for doc 1, all identified with metadata. No own content.<\/p>\n

Extracts from observer record<\/strong><\/p>\n

Opening CONGIG file. Doesn\u2019t automatically open \u2013 users had to open this in Notepad<\/p>\n

Opening Word user panel. No label to say it is \u2018user panel\u2019. Author panel had a label, which caused some confusion.<\/p>\n

Resetting repository location in Word user panel. My testers missed out this step, so the process didn\u2019t work.\u00a0 They received an error message \u2018couldn\u2019t understand response from server endpoint\u2019 and realized their mistake.<\/p>\n

Setting login in Word user panel. Password defaults to admin \u2013 confusing.\u00a0 No confirmation when new password is entered to say that the username and password works.<\/p>\n

Doc 1. Two identical records were added to DepositMO. . Users updated one, so there may be two on the screenshot with different times. Some confusion over \u2018item type\u2019 and what this meant. The testers had a discussion at this point about metadata, and how the repository found the title, as it wasn\u2019t the title of the document.<\/p>\n

Image 1. Testers not entirely sure where the depositMO folder was. Users said they didn\u2019t understand what the script was meant to do.\u00a0 They misunderstood and thought they had to copy the image into the repository itself. On adding metadata to the image, testers were disappointed with this \u2013 they didn\u2019t understand exactly what they were doing.\u00a0 They expressed doubts that academics would do this (XML text edit). Image 1 didn\u2019t appear in the repository, although clicking on the View_item.html took them through to the appropriate record.\u00a0 It appears that drag and drop was adding the record to the \u2018user1\u2019 account, even though they had updated the Config file correctly.<\/p>\n

Doc 2 Deposited using the Word App<\/p>\n

Image 2 Deposited using the Drag & Drop method. The testers didn\u2019t understand the point of the different folders. Testers didn\u2019t set up different content folders at first, but realized their mistake and made a second attempt.<\/p>\n

Testers had remembered most steps at this point. Testers commented that there are too many steps, it is not straightforward, and it is not that different to the existing eprints interface in that regard.<\/p>\n

Add image. Testers didn\u2019t have time for this, but had inadvertently done this already.<\/p>\n

Testers queried again how much more metadata you would have to add if using the Word app and the Drag and Drop options.<\/p>\n

\"\"<\/a>User 4 <\/strong><\/h2>\n

<\/strong>Pair. Both are non-users of repositories.<\/p>\n

From repository record<\/strong><\/p>\n

As with user 3, which was simultaneous with this user test, image 1 and the added image appear in the repository workspace of the user 2 account, although this was avoided for deposit of image 2 by using the conventional EPrints deposit interface. Image 3 was added to image 1 and so also ended in user 2 account. As with user 3, this may partly account for the time taken to deposit image 1. All repository entries have at least basic metadata description and are identifiable in the list of records, including two items of own content but excepting the drag-and-drop items that ended in the user 2 account.<\/p>\n

Extracts from observer record<\/strong><\/p>\n

Doc 1 \u201cShow\u2019 panel not obvious \u2013 \u201cneed explanation\u201d Prompted<\/p>\n

Image 1 Just deposited item, not in list of records but can view item<\/p>\n

Doc 2 Word user panel \u2013 need to update again \u201cfind it hard to remember what have done\u201d<\/p>\n

Image 2 Used EPrints deposit interface<\/p>\n

Update doc Went to repository \u2018Edit item\u2019 button. Prompted \u2013 back to Word panel. \u201cCheck if it worked\u201d<\/p>\n

Own content 1 Word doc \u2013 deposited, then added more metadata through EPrints deposit interface<\/p>\n

Own content 2 pdf \u2013 using EPrints deposit<\/p>\n

\"\"<\/a>User 5 <\/strong><\/h2>\n

<\/strong>Pair. Test users only to date, but expect to be more regular repository users in future.<\/p>\n

From repository record<\/strong><\/p>\n

An unspecified item appeared in this user account just prior to deposit of doc 1.<\/p>\n

Image 2 is described, but image 1 with added image 3 are not.<\/p>\n

Unusually for these tests the pdf is fully and correctly described, this taking slightly longer on this task than other users.<\/p>\n

A large amount of own content was added using the file manager tool. These were packaged content \u2013 a \u2018web site\u2019 and zipped archive \u2013 and the aim was to see how the tool extracted and presented the contents. No metadata was added.<\/p>\n

Extracts from observer record<\/strong><\/p>\n

Doc 1 Two unspecified objects in repo at this point + deposited doc. Picked up metadata from doc inc abstract: \u201cGood\u201d. Does doc need to be \u201ctagged\u201d for this to happen? \u201cGood to have item type.\u201d \u201cCould improve description.\u201d Started to add metadata via EPrints interface \u2013 \u201coff piste\u201d.<\/p>\n

Image 1 Copied doc by mistake. Deleted \u2013 but it returned. Then successfully\u00a0 copied intended image. \u201cQuite fun.\u201d \u201cCould be more user-friendly \u2013 populate metadata rather than XML.\u201d<\/p>\n

PDF Couldn\u2019t copy-paste abstract. Worked 3rd<\/sup> time. Added metadata carefully<\/p>\n

Doc 2 Want Word deposit set to default URL. \u201cLike the way it flags up\u201d missing metadata. Checked what\u2019s there\/not there: \u201cstill quite a few required fields not populated\u201d<\/p>\n

Image 2 Chose to use EPrints interface \u2013 \u201clike to be able to add keywords, contextualise, but don\u2019t know creator, etc.<\/p>\n

Update doc Users checked this replacement but couldn\u2019t identify which document to open at first<\/p>\n

Add image This instruction assumes they would have used the file deposit tool to add image 2 \u2013 they hadn\u2019t (see above). Added image to existing folder. Then didn\u2019t know which item it had been added to \u2013 at first.<\/p>\n

Own content \u201cQuite like to test uploading a Web site.\u201d The file tool grabbed lots of pdf files. Zip file: \u201cMaybe it doesn\u2019t like zip files\u201d<\/p>\n

\"\"<\/a>User 6 <\/strong><\/h2>\n

<\/strong>Pair. Mixed experience pair: one had not previously used a repository; the other has used a media and image repository rather than an open access institutional repository such as EPrints.<\/p>\n

From repository record<\/strong><\/p>\n

We do not have definitive observer time checks for start of deposit, hence the span of time for doc 1.<\/p>\n

Doc 1 is listed with a title, but there is no other metadata in the record and full-text is not available.<\/p>\n

Image 2 was deposited twice, accounting for the extended deposit time for this item. Added to same folder as image 1, then deposited as separate item as required. Title metadata added<\/p>\n

PDF added with minimal metadata.<\/p>\n

Own content included a large video (700 MB) and further attempts to deposit image 1 (an archaeology picture by a colleague)<\/p>\n

Extracts from observer record<\/strong><\/p>\n

File manager deposit tool setup: user A didn\u2019t understand the config file set up although user B went straight to it and just did it with no comments<\/p>\n

Doc 1 Options did not seem straightforward, there was clicking to find things and then going back. \u201cWe can\u2019t follow simple instructions\u201d \u2013 they are too impatient they think. I think they were looking for a way to upload a new description but couldn\u2019t find one. There is confusion around the terminology of \u201csubmit\u201d and \u201cdeposit\u201d. Tried to submit it twice because of confusion but recognised that when they got a warning or error message.<\/p>\n

Image 1 Confusion about folders for deposit but reference back to the instructions helped. Some uncertainty about what to put in the title<\/p>\n

Doc 2 Error message for address to deposit \u2013 some confusion until the right place found. Some confusion from two people working together not knowing what button the other has pressed. Some problems when didn\u2019t submit properly. They tried to submit again. Some confusion about how much information you HAD to put in before it would allow you to submit. What does \u201cuser work area\u201d mean?<\/p>\n

Image 2 Managed to submit things and then<\/span> made the new folders which was considered to be confusing. Metadata file appears to have been rewritten \u2013 it did not contain the information they had submitted. Different file structure for this part of the test. It takes a while for the metadata file to be written. The metadata options generated some uncertainty with data deleted or overwritten and so the required things to enter could not be found. They lost the object that they had put in by putting it in the same folder as something else. They put it in again. Considered to be very fiddly, and creating a new folder for every file at this speed is too time consuming with the delay in the creation of the metadata file. AND then another object appeared in the folder \u2013 seemed to be a serious time lag for this to happen i.e. reentering stuff that is there but hasn\u2019t appeared yet\u2026 \u201cThis is evil\u201d.<\/p>\n

Update doc Success first time.<\/p>\n

Add image Worry and concern that this would break the system again! But some uncertainty about how this as a system would work, i.e. illogical? Problem \u2013 they think that the program keeps rewriting their metadata. The program is confused about the files and the metadata. Trying to sort it out took some time \u2013 Had to use the details to work out what was happening.\u00a0 Altered through the web browser (EPrints interface) which a takes a long time to update the metadata file itself.<\/p>\n

Own content Using the EPrints interface as it was considered to be the easiest way to do it. Video took a long time to upload but no problems.<\/p>\n

\"\"<\/a>User 7 <\/strong><\/h2>\n

<\/strong>Pair. Both are regular and experienced users of the Blackboard repository, and have deposited a range of content types.<\/p>\n

From repository record<\/strong><\/p>\n

An \u2018unspecified\u2019 and unexplained record appears first, as for some other users. The first doc deposit has metadata, but doc 2 unusually has no description, although it is updated correctly. The PDF is deposited with made-up metadata. All texts are deposited into the repository review area. From the summary timings, only the image addition is notably longer than other tasks. Deposit of image 1 appears to be unsuccessful, but it appears with image 2 (not according to instructions), both attempts listed as \u2018unspecified\u2019. Then image 2 is joined with image 3 (as instructed) but with the title \u2018Waterwheel\u2019. Finally, own content is a series of 6 images collected, again, under the name \u2018Waterwheel\u2019.<\/p>\n

Extracts from observer record<\/strong><\/p>\n

Setup demo repository Clicked \u2018New Item\u2019. Added an unspecified object?<\/p>\n

Setup file deposit tool, open watch_folder Incomplete<\/p>\n

Image 1 Needed to advise opening watch_folder. \u201cWhat are the unspecifieds that keep popping up?\u201d<\/p>\n

PDF Performed deposit without reference to the actual document provided<\/p>\n

Image 2 Copied to same folder as image1, then set up new folder. Added metadata via XML page (did not display)<\/p>\n

Update doc Opened new copy of doc2 in new Word window. Update didn\u2019t work. Not previously submitted version \u2013 opened new? Found path in original eprint, updated successfully. \u201cHasn\u2019t identified title\u201d Wiped metadata, \u201creally annoying\u201d<\/p>\n

Add image \u201cwent in straight away\u201d<\/p>\n

Own content 6 images copied in one folder. \u201cWorked well\u201d. Opened XML metadata page for folder. Then added metadata via EPrints form.<\/p>\n

Found there were two items in repo! Renamed wrong item. Re-copied. Deleted generated metadata.xml. Failed to recreate. Copied again. Added metadata via EPrints form.<\/p>\n

The next post will show what users said about the test and tools they used, based on feedback from the test form completed immediately at the conclusion of the test.<\/p>\n","protected":false},"excerpt":{"rendered":"

Based on the tests we have a number of ways to analyse the results.\u00a0Previously we considered what the timings and task completion tell us about the usability of the deposit tools. Here we get some insights into the times taken to perform the tasks by using the records of what users did, taken from the […]<\/p>\n","protected":false},"author":25,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[332,334,143,15641,15648,341],"class_list":["post-988","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-depositmo","tag-inf11","tag-jisc","tag-user-testing","tag-user-testing-results","tag-users"],"_links":{"self":[{"href":"https:\/\/blog.soton.ac.uk\/depositmo\/wp-json\/wp\/v2\/posts\/988","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.soton.ac.uk\/depositmo\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.soton.ac.uk\/depositmo\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.soton.ac.uk\/depositmo\/wp-json\/wp\/v2\/users\/25"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.soton.ac.uk\/depositmo\/wp-json\/wp\/v2\/comments?post=988"}],"version-history":[{"count":17,"href":"https:\/\/blog.soton.ac.uk\/depositmo\/wp-json\/wp\/v2\/posts\/988\/revisions"}],"predecessor-version":[{"id":1099,"href":"https:\/\/blog.soton.ac.uk\/depositmo\/wp-json\/wp\/v2\/posts\/988\/revisions\/1099"}],"wp:attachment":[{"href":"https:\/\/blog.soton.ac.uk\/depositmo\/wp-json\/wp\/v2\/media?parent=988"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.soton.ac.uk\/depositmo\/wp-json\/wp\/v2\/categories?post=988"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.soton.ac.uk\/depositmo\/wp-json\/wp\/v2\/tags?post=988"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}