First, let’s recap some of the key issues, constraints and highlights of the user tests already noted in earlier posts.

The first post on the tests made the following claim: “What I can tell you now is that we had no disasters in organising the tests, from my point of view, and I think I would have known. Therefore, like an auditor, I shall present the report as a valid representation of the tools under test.”

We were clear about what was being tested: “In this case we are testing the process of repository deposit using two new tools, that is, it is a usability test. In these tests we will not be testing the users’ capacity to install the tools themselves, but to use tools already set up for use.” Interpretations of the results have to take care not to stretch the limits of the test, for example, we can make no claims about the wider applicability, uptake or impact of the tools, simply the ability of users to use them for the specified tasks.

“User testing depends on providing clear instructions to users.” By providing the full instruction document, not only does this show what users were asked to do in the tests, but the clarity of the instructions and the possible impact on the tests can be judged. The structure of the test also points directly to the structure of the results: what users said in response to before-and-after questionnaires, what users did in the test, times taken to perform the specified tasks and completion rates.

We learned from the before-test questions that users included “a mix of experienced and new repository users, a reasonable profile for this test since the tools are aimed at both types of users.”

With reference to times and task completion we were able to compare the relative performance of the two tools under test, the Word Add-in and the Watch Folder: “The total time taken to complete image deposits was in all cases longer than the time taken to deposit and update Word docs. There was more variability in times of image 1 deposit, but less so for subsequent image deposits when the procedure was more familiar. Some image deposit cases were not completed, but all doc deposits were completed.” This is perhaps the result that might attract most attention, and is the area that Dave Tarrant focusses on below.

The detail of the test results can be found in what users did. We had two ways of recording what users did, based on the repository record indicating process and the degree of task completion, and notes taken by a test observer providing additional insights on process and spontaneous user reactions.

But what did the users say about the tools they tested? The after-test questions gave us some insights: “These summary results suggest that on balance use of these tools might encourage more deposit, just, most clearly in the case of the Word tool, but it won’t be so easy to wean users off the standard repository deposit interface on the basis of these tools.”

Developer’s view of the test results, by Dave Tarrant

Contradicting the mantra “repositories are built for finished publications”, the ‘own’ content brought to the tests for deposit by these users was mostly content that might not be considered as formal ‘publications’. It is these more diverse content-owning communities that are least supported by current repository software. As we see demand for data publication in repositories, for example, the types of content deposited in repositories is only going to grow beyond traditional publications.

During the testing carried out in DepositMO, users were asked to use both the new clients and the existing repository interfaces to deposit a number of different content types. The results from these tests are clear: on average, both direct deposit clients (Word Add-in and Watch Folder) took less time for deposit.

Using the Microsoft Word Add-In developed as part of DepositMO, the average deposit time for a document (from opening Word to completed deposit) was done in less than half the time that it took to deposit the same item via the native repository interface.

The Watch Folder client also resulted in speedier deposit, with a 12% gain in time to deposit over the standard repository interface. In its current form the Watch Folder client does not provide the simplest means of metadata control and thus in some cases additional time was incurred to complete this stage using the repository metadata interfaces.

Some of the quotes recorded during testing suggest at an attitude change is possible, with one user finding the experience “Quite fun”. Over 50% of the users reflected that the tools would encourage them to submit more of their own content to the repository.

With users who are already experienced in repository deposit it is clear that combining new and existing tools into one experience can be confusing, particularly as a user’s collection size grows. All users suggested that the tools “need some work”, specifically in informing the user about the actions being implemented both locally and remotely.

Omission of DSpace user testing

The results reported apply to a controlled test environment “with Web-connected laptops running Windows 7 and connecting to a demonstrator EPrints repository running the SWORDv2 extensions.” Similar extensions have been added to DSpace, and we intended to run the same user tests with a demonstrator DSpace repository as we had with EPrints. We had a possible test site and users lined up. In pre-tests, the setup ran successfully with the Watch Folder tool, but we were unable to to complete a deposit process with the Word Add-in tool and the DSpace demo repository. What this shows is the complexity of managing and synchronising a communication process between a series of tools – Word Add-in, repository software, repository instance, SWORDv2, DepositMO extensions – where three of the components are new and under test for the first time. Perhaps we should rejoice that the EPrints user tests worked at all given these odds. This would have been resolvable with more time, but the project had already been granted an extension to complete the SWORDv2 and DepositMO extensions for DSpace, and further time could not be justified. We decided against a shortened DSpace test with just the Watch Folder tool.

We can draw no further implications from this, although we note that Marco Fabiani at Queen Mary University of London, ironically and by complete coincidence our target DSpace test site, has reported issues with installing the Watch Folder to work with DSpace. Users did not test installation of the deposit tools, and this is known to be an area requiring further attention, particularly for the Watch Folder.

Summary

The wow! factor seen during a demonstrator presentation is, unsurprisingly, harder to sustain in practice, and noticeably harder as a user’s collection size grows. It’s not just about the initial deposit. As collection size grows, issues of metadata control and versioning become more critical. This is why there are no simple comparisons to be made between the instant deposit tools (Word Add-in and Watch Folder) and the more structured native repository deposit interfaces. Instant deposit may be at the expense of providing careful metadata now, although further refinements to the tools might be able to improve metadata control without losing deposit time.

The critical point is not in the comparisons. We can now see more clearly how different repository deposit tools can support different users with different deposit demands – “these more diverse content-owning communities that are least supported by current repository software” – widening the base of repository users. With the growing emphasis on research data management, especially using data repositories, the need for choice in repository deposit – offering tradeoffs between time to deposit and degree of documentation – is only going to become more acute. After all, repositories wish to acquire more content without adding unnecessary barriers to deposit.

Based on the tests we have a number of ways to analyse the results. Previously we considered what the timings and task completion tell us about the usability of the deposit tools, and what users did, based on the observer notes and an assessment of the repository records they created. Here we discover what they thought of the tools they were testing, prompted by a series of questions in the user’s test documentation and placed to be answered immediately at the conclusion of the practical part of the test.

We present just the summary results and selected extracts from this user feedback, so we do not have another over-extended post. If you wish to see the unexpurgated comments, here is the original report of the full test results (pdf).

Summary results

1 Would these tools encourage you to deposit more of your own content in a repository?

Yes (8) No (5)

2 Would these tools encourage you to deposit types of content that you have not previously deposited in a repository? Which type?

Yes 5 (inc. images, multiple presentations/teaching resources, computing files) No (6) ? (2)

3 Which of the two new deposit tools (in Word, and in the file manager) used here are you likely to use if it was generally available?

Word (7), File manager (1), both (3), neither (1), repository (1)

6 Or are you more likely to continue to use the standard repository deposit interface?

Yes (6), No (0), ? (4), N/A (3)

These summary results suggest that on balance use of these tools might encourage more deposit, just, most clearly in the case of the Word tool, but it won’t be so easy to wean users off the standard repository deposit interface on the basis of these tools.

Selected extracts from fully transcribed results

Here are some extracts to provide a flavour of the user comments. These are not intended to be fully representative, for which please refer to summary results above and the full test results document.

1 Would these tools encourage you to deposit more of your own content in a repository?

User2 (EdShare) Yes For me the Word tool. The MO (file manager) deposit was initially exciting but the difficulty of differentiating between many files was frustrating as filenames were not carried over. The updating via Word for single files and also the addition of new items via MO leading to either loss or overwriting of previous metadata is v. frustrating.

User 3/1 (Library) The Word deposit seems very simple, provided it’s easy to get the DepositMO add-in. I would like to know if I need to construct my Word docs in a particular way. Image deposit (file manager) seemed complicated and a bit confusing – not sure I’d use this.

User 5/1 (Kultivate) If uploading Word files would definitely use the DepositMO tool. Not sure about other content.

2 Would these tools encourage you to deposit types of content that you have not previously deposited in a repository? Which type?

User 1/1 (EdShare) No. Folder deposit confusing

User 1/2 (EdShare) No. Word integration confusing

User 3/2 (Library) In theory – yes – but possibly only Word as that was easiest!

User 5/2 (Kultivate) I think yes for PDFs (and Word docs) – of course the images still seem to require more work/metadata

3 Which of the two new deposit tools (in Word, and in the file manager) used here are you likely to use if it was generally available?

User 1 (EdShare) File manager. Word version confusing in version control, settings and location of files

User 6/1 (Archaeology) The Word one is nicer and a good idea, but all are confusing

User 7/2 (Archaeology) Word was very useful for text. The file manager was good for images but possibly required a bit more prior knowledge.

4 How would you improve either of the tools?

User 1/2 (EdShare) Make actions of file manager clearer (what is being updated and where)

User 2 (EdShare) More metadata extracted from original file e.g. title from name, creator from owner (if available) and also type (Word added-both showed as HTML files)

User 3/1 (Library) Word – I wouldn’t require user to add/edit URL; Image – I think this needs to be made more user friendly

User 3/2 (Library) Metadata requirement (i.e. having to add title in HTML) in file manager prob off-putting to some.

5 Are there features of the new deposit tools that would deter you from using them for repository deposit?

User 2 (EdShare) Overwriting metadata which had been hand-edited when updating or adding additional files is a real turn-off.

User 3/1 (Library) Image deposit is too complicated. I wouldn’t remember the URL needed for Word deposit

User 3/2 (Library) I have nothing to compare this with, but I did find too many windows open caused a degree of confusion

User 5/1 (Kultivate) With uploading images etc … I think the time taken in standard way is about the same, due to adding metadata. I think the ‘watch folders’ idea is brilliant but needs to be developed further.

User 6/2 (Archaeology) The fact that each file’s metadata had to be in its own folder with that file -> hard to deposit large directories of files.

6 Or are you more likely to continue to use the standard repository deposit interface?

User 1/2 (EdShare) Yes. Better/clearer control. Less ambiguous.

User 2 (EdShare) Would like to be able to use the new deposit tools more as the traditional, although v. good for library geeks like me, is long-winded for the normal academic.

User 6/1 (Archaeology) I think yes because it is proven to be efficient

Any other comments?

User 5/2 (Kultivate) Really helpful for documents – I can see that for repository managers and even self-depositors this could be helpful, saving quite a bit of time.

User 6/1 (Archaeology) Good ideas but they are not easy enough to use. They seem slow and frustrating.

Once again, we thank our users for engaging fully with these tests, for the time and effort they gave to participate, for their rigour and objectivity. As we saw earlier in the live audience demonstrators, it’s when you put software tools in front of users that they come to life.

Our test users were: I. Atkins, Joy Caisley, Grant Cox, David Davies, Helena Dmetriou, Harry Gibbs, Nick Graffy, Marie-Therese Gramstadt, Matthew Harrison, Alisa Miller, John O’Hagan, Isobel Stark and Adam Warren.

I must also thank the organisers of the user test teams, who doubled as test observers to report on individual tests. Our observers were: Penny Copeland, Debra Morris and Kate Walker.

This completes the recorded results from the DepositMO user testing. In the remaining posts, as the project ends we will seek to draw some lessons and conclusions from the tests and from the project overall. These will be our interpretations of the results. If you have a view on these results and what they might tell us about the deposit tools developed by the project, or on the prospects and needs for future repository deposit tools, please leave a comment.

For completeness, and since it did not seem appropriate to break the post between users, this is necessarily a much longer post than usual on this blog.

User 1

Pair. Experienced repository users, many deposits of a variety of content types over years.

From repository record

Hard to match record with observer notes and intended test chronology. Ended with 5 deposits of doc 1. Took a long time to deposit image 1. Appears to make some false starts with image deposit, did PDF, then returned to image 1 and ended with a collection of 4 images, including all 3 images provided in the sample data for the test and an additional image not provided. Deposited doc 2 (no metadata) and updated by adding a new item (with metadata).

Extracts from observer record

Deposit item 1 (doc 1) Clicked submit x2 so resulted in 2 deposits.  “Would have expected an update” – both agreed. It’s like a ‘save’ button – expect new copy of the doc. Item type – should be doc not unspecified

Deposit item 2 (image 1) Didn’t work first time. Then did work when refreshed screen for manage items = Revision 11 though! But there doesn’t seem to be anything actually there? Ghost image and also actual file. Went back 3^rd time for item 2 -> added another image as the revision of the first Separate folder for each item? How would it tell which version? Well that worked = Rev 14 did what was expected

PDF Subjects -> LC/Dewey – choice? Tried to search for several subjects.

Image 2 Seems odd that would need to create a separate folder for each item. If I have to create a new folder for each item I am never going to use this!! Cut item from original folder. Waited for metadata file. Revised metadata – abbey by night. Resulted in a deposit with 4 files

Doc 2 MS Word error message about the file because it seemed to take the title and corrupt the file. Going back and try again – create new folder. Doc not corrupt at moment. Used short cuts to copy and paste -> my doc which I’ve spent months working on may have been corrupted … Titles seem to be switching around … and the doc seems to be fine when opened up in the repository

Updating doc Error message – forgotten details which had been provided – username and password but no location -> went to item which was locked. Hopelessly lost. No confidence about which document we are working on

Click on’ Submit’ – then this created another copy Characterising and preserving

How does it know which doc I want to update rather than creating a new version?

User 2

Single. Experienced repository user, many deposits of a variety of content types over years in different repositories.

From repository record

Appears to have completed all stages in correct order, and deposited a substantial number of own items including a variety of document and format types, although the collections appear to have become mixed within records, some duplication of items. All repository records can be identified by title, with supplementary metadata added to describe items within records. Looks like an exemplary result.

Extracts from observer record

Opening Word user panel. Missed (the Word deposit tab) at first; looked at repository. Needed prompt to ‘show’ the panel.

Image 1 Had closed depositMO folder after first use. Eventually realized it had to be reopened.

Doc 2 Reset user panel correctly. Added more metadata through EPrints deposit interface

Image 2 Named folder (‘Bath’) this time. Again added more metadata through repo. interface.

Update doc ‘lost some of my added metadata’ e.g. item type. Went back to add again

Add image User waited to see if more VIEW, METADATA files were generated. Noted they didn’t. ‘lost metadata again’. Added again through repo. interface

Own content Deposited 2x pdf + ppt via file manager tool. Added metadata via repo. interface. Added Word doc via panel, added metadata. Then copied ALL own content into one folder. Added one extra item to collection which overwrote previous title and added title of new item.

Parting comment: ‘Like Word’. Would like other (Office apps). That would encourage. Don’t like losing metadata.

User 3

Pair. One is an occasional repository user, the other a non-user.

From repository record

We don’t have an end timing for the short break in the process prior to update doc, hence the span of time for this task and for the total deposit time. A lengthy time for first deposits of both doc and image. No image deposits can be found in the repository workspace for this user. On inspection, these deposits can be traced to the workspace of User 1, probably due to an issue with the user configuration file in the ‘watch folder’, which directs content to the user’s login space in the repository. Image 2 was at first added to same folder as image 1, but then deposited separately in own folder. The docs and pdf do not use the config direction and appear correctly in this user’s repository workspace, with a duplicate entry for doc 1, all identified with metadata. No own content.

Extracts from observer record

Opening CONGIG file. Doesn’t automatically open – users had to open this in Notepad

Opening Word user panel. No label to say it is ‘user panel’. Author panel had a label, which caused some confusion.

Resetting repository location in Word user panel. My testers missed out this step, so the process didn’t work.  They received an error message ‘couldn’t understand response from server endpoint’ and realized their mistake.

Setting login in Word user panel. Password defaults to admin – confusing.  No confirmation when new password is entered to say that the username and password works.

Doc 1. Two identical records were added to DepositMO. . Users updated one, so there may be two on the screenshot with different times. Some confusion over ‘item type’ and what this meant. The testers had a discussion at this point about metadata, and how the repository found the title, as it wasn’t the title of the document.

Image 1. Testers not entirely sure where the depositMO folder was. Users said they didn’t understand what the script was meant to do.  They misunderstood and thought they had to copy the image into the repository itself. On adding metadata to the image, testers were disappointed with this – they didn’t understand exactly what they were doing.  They expressed doubts that academics would do this (XML text edit). Image 1 didn’t appear in the repository, although clicking on the View_item.html took them through to the appropriate record.  It appears that drag and drop was adding the record to the ‘user1’ account, even though they had updated the Config file correctly.

Doc 2 Deposited using the Word App

Image 2 Deposited using the Drag & Drop method. The testers didn’t understand the point of the different folders. Testers didn’t set up different content folders at first, but realized their mistake and made a second attempt.

Testers had remembered most steps at this point. Testers commented that there are too many steps, it is not straightforward, and it is not that different to the existing eprints interface in that regard.

Add image. Testers didn’t have time for this, but had inadvertently done this already.

Testers queried again how much more metadata you would have to add if using the Word app and the Drag and Drop options.

User 4

Pair. Both are non-users of repositories.

From repository record

As with user 3, which was simultaneous with this user test, image 1 and the added image appear in the repository workspace of the user 2 account, although this was avoided for deposit of image 2 by using the conventional EPrints deposit interface. Image 3 was added to image 1 and so also ended in user 2 account. As with user 3, this may partly account for the time taken to deposit image 1. All repository entries have at least basic metadata description and are identifiable in the list of records, including two items of own content but excepting the drag-and-drop items that ended in the user 2 account.

Extracts from observer record

Doc 1 “Show’ panel not obvious – “need explanation” Prompted

Image 1 Just deposited item, not in list of records but can view item

Doc 2 Word user panel – need to update again “find it hard to remember what have done”

Image 2 Used EPrints deposit interface

Update doc Went to repository ‘Edit item’ button. Prompted – back to Word panel. “Check if it worked”

Own content 1 Word doc – deposited, then added more metadata through EPrints deposit interface

Own content 2 pdf – using EPrints deposit

User 5

Pair. Test users only to date, but expect to be more regular repository users in future.

From repository record

An unspecified item appeared in this user account just prior to deposit of doc 1.

Image 2 is described, but image 1 with added image 3 are not.

Unusually for these tests the pdf is fully and correctly described, this taking slightly longer on this task than other users.

A large amount of own content was added using the file manager tool. These were packaged content – a ‘web site’ and zipped archive – and the aim was to see how the tool extracted and presented the contents. No metadata was added.

Extracts from observer record

Doc 1 Two unspecified objects in repo at this point + deposited doc. Picked up metadata from doc inc abstract: “Good”. Does doc need to be “tagged” for this to happen? “Good to have item type.” “Could improve description.” Started to add metadata via EPrints interface – “off piste”.

Image 1 Copied doc by mistake. Deleted – but it returned. Then successfully  copied intended image. “Quite fun.” “Could be more user-friendly – populate metadata rather than XML.”

PDF Couldn’t copy-paste abstract. Worked 3^rd time. Added metadata carefully

Doc 2 Want Word deposit set to default URL. “Like the way it flags up” missing metadata. Checked what’s there/not there: “still quite a few required fields not populated”

Image 2 Chose to use EPrints interface – “like to be able to add keywords, contextualise, but don’t know creator, etc.

Update doc Users checked this replacement but couldn’t identify which document to open at first

Add image This instruction assumes they would have used the file deposit tool to add image 2 – they hadn’t (see above). Added image to existing folder. Then didn’t know which item it had been added to – at first.

Own content “Quite like to test uploading a Web site.” The file tool grabbed lots of pdf files. Zip file: “Maybe it doesn’t like zip files”

User 6

Pair. Mixed experience pair: one had not previously used a repository; the other has used a media and image repository rather than an open access institutional repository such as EPrints.

From repository record

We do not have definitive observer time checks for start of deposit, hence the span of time for doc 1.

Doc 1 is listed with a title, but there is no other metadata in the record and full-text is not available.

Image 2 was deposited twice, accounting for the extended deposit time for this item. Added to same folder as image 1, then deposited as separate item as required. Title metadata added

PDF added with minimal metadata.

Own content included a large video (700 MB) and further attempts to deposit image 1 (an archaeology picture by a colleague)

Extracts from observer record

File manager deposit tool setup: user A didn’t understand the config file set up although user B went straight to it and just did it with no comments

Doc 1 Options did not seem straightforward, there was clicking to find things and then going back. “We can’t follow simple instructions” – they are too impatient they think. I think they were looking for a way to upload a new description but couldn’t find one. There is confusion around the terminology of “submit” and “deposit”. Tried to submit it twice because of confusion but recognised that when they got a warning or error message.

Image 1 Confusion about folders for deposit but reference back to the instructions helped. Some uncertainty about what to put in the title

Doc 2 Error message for address to deposit – some confusion until the right place found. Some confusion from two people working together not knowing what button the other has pressed. Some problems when didn’t submit properly. They tried to submit again. Some confusion about how much information you HAD to put in before it would allow you to submit. What does “user work area” mean?

Image 2 Managed to submit things and then made the new folders which was considered to be confusing. Metadata file appears to have been rewritten – it did not contain the information they had submitted. Different file structure for this part of the test. It takes a while for the metadata file to be written. The metadata options generated some uncertainty with data deleted or overwritten and so the required things to enter could not be found. They lost the object that they had put in by putting it in the same folder as something else. They put it in again. Considered to be very fiddly, and creating a new folder for every file at this speed is too time consuming with the delay in the creation of the metadata file. AND then another object appeared in the folder – seemed to be a serious time lag for this to happen i.e. reentering stuff that is there but hasn’t appeared yet… “This is evil”.

Update doc Success first time.

Add image Worry and concern that this would break the system again! But some uncertainty about how this as a system would work, i.e. illogical? Problem – they think that the program keeps rewriting their metadata. The program is confused about the files and the metadata. Trying to sort it out took some time – Had to use the details to work out what was happening.  Altered through the web browser (EPrints interface) which a takes a long time to update the metadata file itself.

Own content Using the EPrints interface as it was considered to be the easiest way to do it. Video took a long time to upload but no problems.

User 7

Pair. Both are regular and experienced users of the Blackboard repository, and have deposited a range of content types.

From repository record

An ‘unspecified’ and unexplained record appears first, as for some other users. The first doc deposit has metadata, but doc 2 unusually has no description, although it is updated correctly. The PDF is deposited with made-up metadata. All texts are deposited into the repository review area. From the summary timings, only the image addition is notably longer than other tasks. Deposit of image 1 appears to be unsuccessful, but it appears with image 2 (not according to instructions), both attempts listed as ‘unspecified’. Then image 2 is joined with image 3 (as instructed) but with the title ‘Waterwheel’. Finally, own content is a series of 6 images collected, again, under the name ‘Waterwheel’.

Extracts from observer record

Setup demo repository Clicked ‘New Item’. Added an unspecified object?

Setup file deposit tool, open watch_folder Incomplete

Image 1 Needed to advise opening watch_folder. “What are the unspecifieds that keep popping up?”

PDF Performed deposit without reference to the actual document provided

Image 2 Copied to same folder as image1, then set up new folder. Added metadata via XML page (did not display)

Update doc Opened new copy of doc2 in new Word window. Update didn’t work. Not previously submitted version – opened new? Found path in original eprint, updated successfully. “Hasn’t identified title” Wiped metadata, “really annoying”

Add image “went in straight away”

Own content 6 images copied in one folder. “Worked well”. Opened XML metadata page for folder. Then added metadata via EPrints form.

Found there were two items in repo! Renamed wrong item. Re-copied. Deleted generated metadata.xml. Failed to recreate. Copied again. Added metadata via EPrints form.

The next post will show what users said about the test and tools they used, based on feedback from the test form completed immediately at the conclusion of the test.

1. Times taken to perform the tasks
2. What users did, based on the repository record and observer notes
3. What users said, based on feedback from the test form

Subsequent posts will reveal what users did and said. Here we focus on what the timings and task completion tell us about the usability of the deposit tools.

User tests were performed over a series of dates during June and July 2011. Some tests were simultaneous and co-located. In total 13 users performed the test in 6 pairs and one singly (recall, tests were set up for pairs, to provide the opportunity for reactions to be vocalised between users, and to avoid equipment limitations for the controlled test environment).

Users 1, 2: June 3
Users 3, 4: June 17
User 5: June 20
Users 6, 7: July 5

All users completed the same tasks, guided by the same instruction document. Minor amendments were made to the document between tests to improve and clarify procedure, as pointed out by users, where problems were deemed to be due to the test documentation rather than the tasks and tools under test.

Procedure For reference, this is a quick chronology of the deposit test process outlined in detail in previous posts:

• Deposit a Word document (using Word tool) Doc 1
• Deposit an image (using file manager tool) Image 1
• Deposit a pdf using the standard EPrints repository interface (PDF)
• Deposit a second Word document (tool chosen by user) Doc 2
• Deposit a second image (tool chosen by user) Image 2
• Update the second Word document (Update doc)
• Add an image to a collection including the second image (Add image)
• Deposit own content (Own content)

Results

Time taken to perform set tasks

Table 1 shows times grouped by process and tool rather than chronologically.

Table 1. Time taken by users to perform set tasks in deposit test process

Notes on Table 1

Times were recorded by observers for selected stages in the overall process. The repository timestamps, including the EPrints history for each record, were used to calculate times of individual tasks.

Where a time duration is unknown, or where a time range is given, the observer notes are incomplete.

Where no time is given, the procedure was not completed.

Where the process appears to be continuous, it is assumed that the last recorded deposit time for an item is the start time of the next deposit process, even if the first recorded deposit time for the subsequent item is some minutes later. This assumption is based on the need for users to assess the instruction and decide which actions to take, and that the time taken to do this is an implicit part of the deposit process for each item.

Not all user processes were continuous or followed the order of the instructions. Where this happens timings are likely to be less reliable, based on an assessment of the record and a judgement of how users proceeded. Such instances will be specified.

e.g. User 1: the deposit of images and the PDF appear to be interleaved, so apportionment of the times between images and pdf are approximate.

Deposit time ≠ Total doc + Total image + PDF + Own. Allow ± 1 min for each action.

Deposit time ≠ Total test time. Additional time is taken to read the documentation for the procedure, and to complete questions before and after the deposit process.

Repository clock set 1h slow. (This is only relevant when comparing observer timings with repository record.)

General observations on times

The total time taken to complete image deposits was in all cases longer than the time taken to deposit and update Word docs. There was more variability in times of image 1 deposit, but less so for subsequent image deposits when the procedure was more familiar. Some image deposit cases were not completed, but all doc deposits were completed.

In the next post we will look at what users did, based on the repository record and observer notes.

To understand the context of the test results we need to know something about the users who performed the tests. All users, apart from two, are based at the University of Southampton, the home of the DepositMO project. We are grateful to them all for their participation, and will acknowledge them individually at the conclusion of these results.

The test instruction document included two questionnaire sections:

1. Before any deposit actions, to find out the prior experience of testers in using repositories.
2. After deposit tests, to discover their reactions to the tools used in the test.

This post summarises the findings from part 1.

1. Have you deposited content in an institutional repository? Yes 9 No 4
2. In which repository do you deposit most? EdShare (3 users), EPrints (3), Blackboard (2), MediaBin (1), none (4)
3. Do you deposit content regularly? Approximately how many items have you deposited in this repository? Over how long a period have you been depositing? See Table 1
4. What type of data have you deposited? Presentation slides (4 users), research papers (PDF) (3), images (3), video (3), docs (2), theses (2), audio (1), research data (1), spreadsheets (1), handouts (1), reports (1), essays (2), computer project (1), database (1).

Table 1. Brief history of repository deposits by our test users

This presents a picture of a mix of experienced and new repository users, a reasonable profile for this test since the tools are aimed at both types of users. Among the experienced users, none claim to deposit on a regular basis, but some show many deposits of a wide range of data types over an extended period. In this presentation we will not be seeking to stratify the results between users and non-users.

In the next post we start to learn how these users got on with the tasks set for them in this test of new deposit tools.