Based on the tests we have a number of ways to analyse the results. Previously we considered what the timings and task completion tell us about the usability of the deposit tools. Here we get some insights into the times taken to perform the tasks by using the records of what users did, taken from the observer notes and an assessment of the repository records they created.
For completeness, and since it did not seem appropriate to break the post between users, this is necessarily a much longer post than usual on this blog.
User 1
Pair. Experienced repository users, many deposits of a variety of content types over years.
From repository record
Hard to match record with observer notes and intended test chronology. Ended with 5 deposits of doc 1. Took a long time to deposit image 1. Appears to make some false starts with image deposit, did PDF, then returned to image 1 and ended with a collection of 4 images, including all 3 images provided in the sample data for the test and an additional image not provided. Deposited doc 2 (no metadata) and updated by adding a new item (with metadata).
Extracts from observer record
Deposit item 1 (doc 1) Clicked submit x2 so resulted in 2 deposits. âWould have expected an updateâ â both agreed. Itâs like a âsaveâ button â expect new copy of the doc. Item type â should be doc not unspecified
Deposit item 2 (image 1) Didnât work first time. Then did work when refreshed screen for manage items = Revision 11 though! But there doesnât seem to be anything actually there? Ghost image and also actual file. Went back 3rd time for item 2 -> added another image as the revision of the first Separate folder for each item? How would it tell which version? Well that worked = Rev 14 did what was expected
PDF Subjects -> LC/Dewey – choice? Tried to search for several subjects.
Image 2 Seems odd that would need to create a separate folder for each item. If I have to create a new folder for each item I am never going to use this!! Cut item from original folder. Waited for metadata file. Revised metadata – abbey by night. Resulted in a deposit with 4 files
Doc 2 MS Word error message about the file because it seemed to take the title and corrupt the file. Going back and try again – create new folder. Doc not corrupt at moment. Used short cuts to copy and paste -> my doc which I’ve spent months working on may have been corrupted … Titles seem to be switching around … and the doc seems to be fine when opened up in the repository
Updating doc Error message â forgotten details which had been provided â username and password but no location -> went to item which was locked. Hopelessly lost. No confidence about which document we are working on
Click onâ Submitâ â then this created another copy Characterising and preserving
How does it know which doc I want to update rather than creating a new version?
User 2
Single. Experienced repository user, many deposits of a variety of content types over years in different repositories.
From repository record
Appears to have completed all stages in correct order, and deposited a substantial number of own items including a variety of document and format types, although the collections appear to have become mixed within records, some duplication of items. All repository records can be identified by title, with supplementary metadata added to describe items within records. Looks like an exemplary result.
Extracts from observer record
Opening Word user panel. Missed (the Word deposit tab) at first; looked at repository. Needed prompt to âshowâ the panel.
Image 1 Had closed depositMO folder after first use. Eventually realized it had to be reopened.
Doc 2 Reset user panel correctly. Added more metadata through EPrints deposit interface
Image 2 Named folder (âBathâ) this time. Again added more metadata through repo. interface.
Update doc âlost some of my added metadataâ e.g. item type. Went back to add again
Add image User waited to see if more VIEW, METADATA files were generated. Noted they didnât. âlost metadata againâ. Added again through repo. interface
Own content Deposited 2x pdf + ppt via file manager tool. Added metadata via repo. interface. Added Word doc via panel, added metadata. Then copied ALL own content into one folder. Added one extra item to collection which overwrote previous title and added title of new item.
Parting comment: âLike Wordâ. Would like other (Office apps). That would encourage. Donât like losing metadata.
User 3
Pair. One is an occasional repository user, the other a non-user.
From repository record
We donât have an end timing for the short break in the process prior to update doc, hence the span of time for this task and for the total deposit time. A lengthy time for first deposits of both doc and image. No image deposits can be found in the repository workspace for this user. On inspection, these deposits can be traced to the workspace of User 1, probably due to an issue with the user configuration file in the âwatch folderâ, which directs content to the userâs login space in the repository. Image 2 was at first added to same folder as image 1, but then deposited separately in own folder. The docs and pdf do not use the config direction and appear correctly in this userâs repository workspace, with a duplicate entry for doc 1, all identified with metadata. No own content.
Extracts from observer record
Opening CONGIG file. Doesnât automatically open â users had to open this in Notepad
Opening Word user panel. No label to say it is âuser panelâ. Author panel had a label, which caused some confusion.
Resetting repository location in Word user panel. My testers missed out this step, so the process didnât work. They received an error message âcouldnât understand response from server endpointâ and realized their mistake.
Setting login in Word user panel. Password defaults to admin â confusing. No confirmation when new password is entered to say that the username and password works.
Doc 1. Two identical records were added to DepositMO. . Users updated one, so there may be two on the screenshot with different times. Some confusion over âitem typeâ and what this meant. The testers had a discussion at this point about metadata, and how the repository found the title, as it wasnât the title of the document.
Image 1. Testers not entirely sure where the depositMO folder was. Users said they didnât understand what the script was meant to do. They misunderstood and thought they had to copy the image into the repository itself. On adding metadata to the image, testers were disappointed with this â they didnât understand exactly what they were doing. They expressed doubts that academics would do this (XML text edit). Image 1 didnât appear in the repository, although clicking on the View_item.html took them through to the appropriate record. It appears that drag and drop was adding the record to the âuser1â account, even though they had updated the Config file correctly.
Doc 2 Deposited using the Word App
Image 2 Deposited using the Drag & Drop method. The testers didnât understand the point of the different folders. Testers didnât set up different content folders at first, but realized their mistake and made a second attempt.
Testers had remembered most steps at this point. Testers commented that there are too many steps, it is not straightforward, and it is not that different to the existing eprints interface in that regard.
Add image. Testers didnât have time for this, but had inadvertently done this already.
Testers queried again how much more metadata you would have to add if using the Word app and the Drag and Drop options.
User 4
Pair. Both are non-users of repositories.
From repository record
As with user 3, which was simultaneous with this user test, image 1 and the added image appear in the repository workspace of the user 2 account, although this was avoided for deposit of image 2 by using the conventional EPrints deposit interface. Image 3 was added to image 1 and so also ended in user 2 account. As with user 3, this may partly account for the time taken to deposit image 1. All repository entries have at least basic metadata description and are identifiable in the list of records, including two items of own content but excepting the drag-and-drop items that ended in the user 2 account.
Extracts from observer record
Doc 1 âShowâ panel not obvious â âneed explanationâ Prompted
Image 1 Just deposited item, not in list of records but can view item
Doc 2 Word user panel â need to update again âfind it hard to remember what have doneâ
Image 2 Used EPrints deposit interface
Update doc Went to repository âEdit itemâ button. Prompted â back to Word panel. âCheck if it workedâ
Own content 1 Word doc â deposited, then added more metadata through EPrints deposit interface
Own content 2 pdf â using EPrints deposit
User 5
Pair. Test users only to date, but expect to be more regular repository users in future.
From repository record
An unspecified item appeared in this user account just prior to deposit of doc 1.
Image 2 is described, but image 1 with added image 3 are not.
Unusually for these tests the pdf is fully and correctly described, this taking slightly longer on this task than other users.
A large amount of own content was added using the file manager tool. These were packaged content â a âweb siteâ and zipped archive â and the aim was to see how the tool extracted and presented the contents. No metadata was added.
Extracts from observer record
Doc 1 Two unspecified objects in repo at this point + deposited doc. Picked up metadata from doc inc abstract: âGoodâ. Does doc need to be âtaggedâ for this to happen? âGood to have item type.â âCould improve description.â Started to add metadata via EPrints interface â âoff pisteâ.
Image 1 Copied doc by mistake. Deleted â but it returned. Then successfully copied intended image. âQuite fun.â âCould be more user-friendly â populate metadata rather than XML.â
PDF Couldnât copy-paste abstract. Worked 3rd time. Added metadata carefully
Doc 2 Want Word deposit set to default URL. âLike the way it flags upâ missing metadata. Checked whatâs there/not there: âstill quite a few required fields not populatedâ
Image 2 Chose to use EPrints interface â âlike to be able to add keywords, contextualise, but donât know creator, etc.
Update doc Users checked this replacement but couldnât identify which document to open at first
Add image This instruction assumes they would have used the file deposit tool to add image 2 â they hadnât (see above). Added image to existing folder. Then didnât know which item it had been added to â at first.
Own content âQuite like to test uploading a Web site.â The file tool grabbed lots of pdf files. Zip file: âMaybe it doesnât like zip filesâ
User 6
Pair. Mixed experience pair: one had not previously used a repository; the other has used a media and image repository rather than an open access institutional repository such as EPrints.
From repository record
We do not have definitive observer time checks for start of deposit, hence the span of time for doc 1.
Doc 1 is listed with a title, but there is no other metadata in the record and full-text is not available.
Image 2 was deposited twice, accounting for the extended deposit time for this item. Added to same folder as image 1, then deposited as separate item as required. Title metadata added
PDF added with minimal metadata.
Own content included a large video (700 MB) and further attempts to deposit image 1 (an archaeology picture by a colleague)
Extracts from observer record
File manager deposit tool setup: user A didnât understand the config file set up although user B went straight to it and just did it with no comments
Doc 1 Options did not seem straightforward, there was clicking to find things and then going back. âWe canât follow simple instructionsâ â they are too impatient they think. I think they were looking for a way to upload a new description but couldnât find one. There is confusion around the terminology of âsubmitâ and âdepositâ. Tried to submit it twice because of confusion but recognised that when they got a warning or error message.
Image 1 Confusion about folders for deposit but reference back to the instructions helped. Some uncertainty about what to put in the title
Doc 2 Error message for address to deposit â some confusion until the right place found. Some confusion from two people working together not knowing what button the other has pressed. Some problems when didnât submit properly. They tried to submit again. Some confusion about how much information you HAD to put in before it would allow you to submit. What does âuser work areaâ mean?
Image 2 Managed to submit things and then made the new folders which was considered to be confusing. Metadata file appears to have been rewritten â it did not contain the information they had submitted. Different file structure for this part of the test. It takes a while for the metadata file to be written. The metadata options generated some uncertainty with data deleted or overwritten and so the required things to enter could not be found. They lost the object that they had put in by putting it in the same folder as something else. They put it in again. Considered to be very fiddly, and creating a new folder for every file at this speed is too time consuming with the delay in the creation of the metadata file. AND then another object appeared in the folder â seemed to be a serious time lag for this to happen i.e. reentering stuff that is there but hasnât appeared yet⊠âThis is evilâ.
Update doc Success first time.
Add image Worry and concern that this would break the system again! But some uncertainty about how this as a system would work, i.e. illogical? Problem â they think that the program keeps rewriting their metadata. The program is confused about the files and the metadata. Trying to sort it out took some time â Had to use the details to work out what was happening. Altered through the web browser (EPrints interface) which a takes a long time to update the metadata file itself.
Own content Using the EPrints interface as it was considered to be the easiest way to do it. Video took a long time to upload but no problems.
User 7
Pair. Both are regular and experienced users of the Blackboard repository, and have deposited a range of content types.
From repository record
An âunspecifiedâ and unexplained record appears first, as for some other users. The first doc deposit has metadata, but doc 2 unusually has no description, although it is updated correctly. The PDF is deposited with made-up metadata. All texts are deposited into the repository review area. From the summary timings, only the image addition is notably longer than other tasks. Deposit of image 1 appears to be unsuccessful, but it appears with image 2 (not according to instructions), both attempts listed as âunspecifiedâ. Then image 2 is joined with image 3 (as instructed) but with the title âWaterwheelâ. Finally, own content is a series of 6 images collected, again, under the name âWaterwheelâ.
Extracts from observer record
Setup demo repository Clicked âNew Itemâ. Added an unspecified object?
Setup file deposit tool, open watch_folder Incomplete
Image 1 Needed to advise opening watch_folder. âWhat are the unspecifieds that keep popping up?â
PDF Performed deposit without reference to the actual document provided
Image 2 Copied to same folder as image1, then set up new folder. Added metadata via XML page (did not display)
Update doc Opened new copy of doc2 in new Word window. Update didnât work. Not previously submitted version â opened new? Found path in original eprint, updated successfully. âHasnât identified titleâ Wiped metadata, âreally annoyingâ
Add image âwent in straight awayâ
Own content 6 images copied in one folder. âWorked wellâ. Opened XML metadata page for folder. Then added metadata via EPrints form.
Found there were two items in repo! Renamed wrong item. Re-copied. Deleted generated metadata.xml. Failed to recreate. Copied again. Added metadata via EPrints form.
The next post will show what users said about the test and tools they used, based on feedback from the test form completed immediately at the conclusion of the test.
Recent Comments