Skip to content

Categories:

DepositMOre: new deposit tools follow the content

DepositMO 2, Spiderman 3, Scream 4? There’s a reason JISC doesn’t do sequels, and not just because it isn’t in the business of funding films. And there’s a reason this project is DepositMOre and not DepositMO 2: it’s in the call.

Where DepositMO developed new deposit tools, DepositMOre is intended to be about applying those tools to test if they can result in a measurable increase in content for real repositories – within 6 months – because that is what the JISC call for projects specified. We are learning early in this work that in the era of ‘big data’ institutions are identifying eligible content by their researchers in external sources and finding they need help to import this content to their repositories, that is, the deposit tools follow the content rather than the other way round. A follow-on project, but with a twist, not just more of the same.

EasyChair deposit tool: identifying content from the EasyChair conference system with a one-click button for deposit in a repository

The tools developed in DepositMO, such as the Word Add-in, although visionary and original, are intended to support deposit of newly created content. Since we don’t expect to be able to stimulate an orgy of new research writing in 6 months specifically as a result of this project, the more pragmatic approach is to try and identify existing content that should be in a repository but isn’t. This sort of approach – finding the ‘low hanging fruit‘ – is hardly new to repositories.

So the emphasis in DepositMOre is on tools for content discovery, of extant content that should have a copy in a repository, and the means of simplifying the heavy lifting - and thus saving time - of deposit .

If you read the project proposal you will find an example using the EasyChair conference system. EasyChair manages papers submitted to conferences. In many cases, conference proceedings are published, and once papers have been reviewed and selected then EasyChair has done its job. Not all papers are published, however, and many remain, unintentionally, exclusively in the EasyChair files, but these papers are searchable.

The aim of the DepositMOre EasyChair tool is to search the EasyChair corpus, and then check against the contents of a specified repository to find if a given paper has already been deposited there. As can be seen in the illustration above, if an EasyChair paper is not matched in the repository then a deposit button appears; when a match is found a green tick is placed against ‘Deposited’ and the deposit button is greyed out to prevent duplication. Metadata describing the paper is displayed for the author or repository administrator to decide whether to activate the deposit, and is used to populate the repository record for the selected paper deposit.

Combining EasyChair search and repository comparison search is important, if probably not infallible, for expediting deposit. It’s the time-saving feature that will be at the heart of DepositMOre tools offered to repositories.

A similar approach can be envisaged for another discovery service linking to academic papers, Microsoft Academic Search (MAS): find papers and perform a match against current repository content, with an option to deposit, again using a simple button. Why not Google Scholar (GS)? MAS has an API, which means you can perform machine-based searches and process and import the results to another service; GS does not currently support this.

Deposit tools that work with EasyChair or MAS or other services are likely to be made available in the EPrints Bazaar. That’s another change from DepositMO – at release tools will work with EPrints and not DSpace. That again is dictated by the pragmatics of the project’s short timescale and the need to work with real repositories. All our partners manage, or have access to, an EPrints repository.

The evolution of the project does not end there. Our initial meetings with partners have begun to point at extant content they have already identified, but which is not easy to import to their repositories. In this case the potential attractions for repository managers of a DepositMOre tool are less discovery and more location and heavy lifting (time saving). Examples uncovered so far include video sources for arts repositories, and large collections of structured image data for an archaeology partner.

So the target of the project becomes clearer – it’s foremost about partner repositories and content, and the tools should follow those leads. That is the big change from DepositMO.

In the next posts we will introduce our repository and content partners, and show how they are leading us towards the development of new deposit tools for specified content.

Posted in Uncategorized. Tagged with , , , .

By Steve Hitchcock
November 27, 2012

No comments

DepositMOre: the timeline so far

Que Sera, Sera, but timing is everything, so they say. DepositMOre follows on directly from the DepositMO project, but temporally the linkage was not quite as expected. Here we review how the project finally got to the start line, to begin to understand whatever it will be and whatever small changes delay may have wrought.

We last blogged here about DepositMO on 14 February. At that time we had submitted a bid to a JISC call for proposals – Deposit Projects: Benefits Realisation – to build on the work through a new project, already identified in the bid as DepositMOre. The aim of that call was to offer some direct continuation for projects in the original JISC Repository Deposit programme strand, for successful bids. For that reason there would be a fast-track presentation and evaluation of bids, with a view to new projects starting in March 2012, and finishing within 6 months.

While we heard positive feedback from the evaluation committee, we didn’t get the green light to begin in March. Instead the light turned green in July. Bigger wheels than this decision have been turning at JISC, so some indecision was understandable, even if we were left in a period of uncertainty, not sure when or if the new project would get approval. Despite the delay, it says a great deal about the commitment of JISC and programme manager Balviar Notay to this area of work, and we are grateful that she persisted and delivered the project to us.

Projects emerge from calls, and are directed by those calls. The requirement of this call was to show deposit of more content in real repositories, using the tools developed in the first phase of the work, in our case DepositMO. In other words, a focus on deposit rather than development. A consequent requirement was that we partner with some institutional repositories: in DepositMO we had partnered with technical developers. These would not be the same partners, although the small core project  team at Southampton would be the same.

This, then, is what the project looks like, as described in the project proposal. This was the final version to be submitted to JISC and includes changes responding to feedback from the evaluation committee. Although this was the version that was finally accepted, and forms the basis of the project, the outcome remained uncertain at this stage. It can be seen from the cover page that this version was anticipating a schedule starting in June 2012.

So with repository partners on board and the bid submitted, we waited to report the formal outcome to them, believing throughout that the outcome was imminent. When that decision arrived some months after the original submission and presentation we realised we would have a job to keep all the partners with us on a revised schedule, and that we would have to be flexible on a that schedule. It was unlikely we would be able to start immediately.

Does anyone remember the summer of 2012? In the UK it was essentially the Olympics, then everyone went away to recover. I’m pleased to say, however, that we were able to use that time to work with the respective repository partners to put together a revised project timescale that they could work with. As a result the project will run for 6 months to end March 2013.

There has been one significant change. I was fortunate to work with Dave Tarrant as the axis of DepositMO at Southampton. Dave motivated the shape and development profile of both projects, but in the intervening period has committed to representing Southampton in other big projects. In his place we are delighted to welcome Tim Brody to DepositMOre. Tim is known for his distinguished work in the repository field, most recently as lead developer for EPrints but also for related repository services such as ROAR, Citebase, and others.

Given the time that elapsed since the bid, the subsequent space that created for discussion and thinking about the work, and surrounding developments, not everything about the project will be exactly as envisaged. We will explore what has changed in terms of technical development, and introduce each of our repository partners, in the next posts.

Posted in Uncategorized. Tagged with , , , , .

By Steve Hitchcock
November 23, 2012

No comments

From DepositMO to DepositMOre: what’s in a name?

Hands up those who know what the MO stands for in DepositMO. I see a few familiar hands, from people who know the project or have been attentive to this blog. What about those new to the project – can you work out what it stands for? Well, it is Modus Operandi, and refers to the aim of the project to change the modus operandi of repository deposit. Obvious really, if you connected deposit to repositories, and from experience knew that it might be a good thing if the process of repository deposit could be improved, from a user perspective.

Alright, we agree, it’s not the most intuitive of project names. But with one leap to a new project we are free, and the genius of the original name becomes apparent. Simply appending two characters to the name we have DepositMOre, and all becomes clear. Well yes, you still have to make the connection with repositories, but once that is known the purpose of the project is clear: deposit more content in real repositories. Need we say MOre?

Yes, we do, but that’s for another post, where we’ll bring you up to date with the story of the new project so far.

Posted in Uncategorized. Tagged with , .

By Steve Hitchcock
November 2, 2012

No comments

User testing results: a simple analysis

Results, by ntr23The series of blog posts on user testing requires some summary, even if it is best not to be over-analytical about the results of user tests, especially if you were responsible for designing the tests. That is perhaps best left to others, and this post will include some points highlighted by Dave Tarrant, the developer of one of the deposit tools tested.

First, let’s recap some of the key issues, constraints and highlights of the user tests already noted in earlier posts.

The first post on the tests made the following claim: “What I can tell you now is that we had no disasters in organising the tests, from my point of view, and I think I would have known. Therefore, like an auditor, I shall present the report as a valid representation of the tools under test.”

We were clear about what was being tested: “In this case we are testing the process of repository deposit using two new tools, that is, it is a usability test. In these tests we will not be testing the users’ capacity to install the tools themselves, but to use tools already set up for use.” Interpretations of the results have to take care not to stretch the limits of the test, for example, we can make no claims about the wider applicability, uptake or impact of the tools, simply the ability of users to use them for the specified tasks.

“User testing depends on providing clear instructions to users.” By providing the full instruction document, not only does this show what users were asked to do in the tests, but the clarity of the instructions and the possible impact on the tests can be judged. The structure of the test also points directly to the structure of the results: what users said in response to before-and-after questionnaires, what users did in the test, times taken to perform the specified tasks and completion rates.

We learned from the before-test questions that users included “a mix of experienced and new repository users, a reasonable profile for this test since the tools are aimed at both types of users.”

With reference to times and task completion we were able to compare the relative performance of the two tools under test, the Word Add-in and the Watch Folder: “The total time taken to complete image deposits was in all cases longer than the time taken to deposit and update Word docs. There was more variability in times of image 1 deposit, but less so for subsequent image deposits when the procedure was more familiar. Some image deposit cases were not completed, but all doc deposits were completed.” This is perhaps the result that might attract most attention, and is the area that Dave Tarrant focusses on below.

The detail of the test results can be found in what users did. We had two ways of recording what users did, based on the repository record indicating process and the degree of task completion, and notes taken by a test observer providing additional insights on process and spontaneous user reactions.

But what did the users say about the tools they tested? The after-test questions gave us some insights: “These summary results suggest that on balance use of these tools might encourage more deposit, just, most clearly in the case of the Word tool, but it won’t be so easy to wean users off the standard repository deposit interface on the basis of these tools.”

Developer’s view of the test results, by Dave Tarrant

Contradicting the mantra “repositories are built for finished publications”, the ‘own’ content brought to the tests for deposit by these users was mostly content that might not be considered as formal ‘publications’. It is these more diverse content-owning communities that are least supported by current repository software. As we see demand for data publication in repositories, for example, the types of content deposited in repositories is only going to grow beyond traditional publications.

During the testing carried out in DepositMO, users were asked to use both the new clients and the existing repository interfaces to deposit a number of different content types. The results from these tests are clear: on average, both direct deposit clients (Word Add-in and Watch Folder) took less time for deposit.

Using the Microsoft Word Add-In developed as part of DepositMO, the average deposit time for a document (from opening Word to completed deposit) was done in less than half the time that it took to deposit the same item via the native repository interface.

The Watch Folder client also resulted in speedier deposit, with a 12% gain in time to deposit over the standard repository interface. In its current form the Watch Folder client does not provide the simplest means of metadata control and thus in some cases additional time was incurred to complete this stage using the repository metadata interfaces.

Some of the quotes recorded during testing suggest at an attitude change is possible, with one user finding the experience “Quite fun”. Over 50% of the users reflected that the tools would encourage them to submit more of their own content to the repository.

With users who are already experienced in repository deposit it is clear that combining new and existing tools into one experience can be confusing, particularly as a user’s collection size grows. All users suggested that the tools “need some work”, specifically in informing the user about the actions being implemented both locally and remotely.

Omission of DSpace user testing

The results reported apply to a controlled test environment “with Web-connected laptops running Windows 7 and connecting to a demonstrator EPrints repository running the SWORDv2 extensions.” Similar extensions have been added to DSpace, and we intended to run the same user tests with a demonstrator DSpace repository as we had with EPrints. We had a possible test site and users lined up. In pre-tests, the setup ran successfully with the Watch Folder tool, but we were unable to to complete a deposit process with the Word Add-in tool and the DSpace demo repository. What this shows is the complexity of managing and synchronising a communication process between a series of tools – Word Add-in, repository software, repository instance, SWORDv2, DepositMO extensions – where three of the components are new and under test for the first time. Perhaps we should rejoice that the EPrints user tests worked at all given these odds. This would have been resolvable with more time, but the project had already been granted an extension to complete the SWORDv2 and DepositMO extensions for DSpace, and further time could not be justified. We decided against a shortened DSpace test with just the Watch Folder tool.

We can draw no further implications from this, although we note that Marco Fabiani at Queen Mary University of London, ironically and by complete coincidence our target DSpace test site, has reported issues with installing the Watch Folder to work with DSpace. Users did not test installation of the deposit tools, and this is known to be an area requiring further attention, particularly for the Watch Folder.

Summary

The wow! factor seen during a demonstrator presentation is, unsurprisingly, harder to sustain in practice, and noticeably harder as a user’s collection size grows. It’s not just about the initial deposit. As collection size grows, issues of metadata control and versioning become more critical. This is why there are no simple comparisons to be made between the instant deposit tools (Word Add-in and Watch Folder) and the more structured native repository deposit interfaces. Instant deposit may be at the expense of providing careful metadata now, although further refinements to the tools might be able to improve metadata control without losing deposit time.

The critical point is not in the comparisons. We can now see more clearly how different repository deposit tools can support different users with different deposit demands – “these more diverse content-owning communities that are least supported by current repository software” – widening the base of repository users. With the growing emphasis on research data management, especially using data repositories, the need for choice in repository deposit – offering tradeoffs between time to deposit and degree of documentation – is only going to become more acute. After all, repositories wish to acquire more content without adding unnecessary barriers to deposit.

Posted in Uncategorized. Tagged with , , , , , .

By Steve Hitchcock
February 14, 2012

No comments

DepositMO – enabling the user-repository conversation

Barbara Claussen, Modern Monoliths migrating, by Northern SparkAs the DepositMO project concludes, lead developer Dave Tarrant summarises his thoughts on the impact and effect of the project’s technical developments.

DepositMO has contributed to reducing the distance between users and content held in digital repositories. In a world where people are ‘always connected’, the Internet is now being used to both store data as well as transfer it. Content stored ‘online’ can be accessed from any device, such as smartphones, and from desktop applications. ‘Disks’ are disappearing in favour of constantly connected clouds of data. Although the physical distance to the data bits increases, for the user, because their content is available on any ‘connected’ device, the distance to their content is simply that to the nearest connected device.

The repository is one item in this connected cloud and must provide the same types of services. The SWORDv2 and DepositMO extensions replicate much of the functionality of the proprietary web interfaces of repository systems and allow control to be decoupled from the content storage. This decoupling is the key enabler to reducing the distance between content and users.

During the course of the project, two platforms – EPrints and DSpace - were enabled with SWORDv2 and DepositMO extensions, and two clients developed (Watch Folder and the Word Add-in). The Watch Folder client was developed as a result of comments about services not reflecting the usability of Dropbox “Drag Drop Share”. Watch Folder was developed as a demonstration that this type of client could be implemented on top of the SWORDv2 and DepositMO protocols. The initial feedback was positive, but it is not DropBox, and the client still needs some polishing. The protocol is in place, however, and would support a more fully-fledged client.

Since the implementation on EPrints, two developments which utilise both protocols have been undertaken for customers by EPrints Services, a repository services provider: one to harvest tweets into a repository, and the other to automatically manage internal and external repositories of the same items. Significantly, both took less than a week to develop (combined!), demonstrating the understandability of not only the specification documents but also the implementations against the repository platform.

DepositMO set out to enable a conversation between a user and a repository. Unlike generic data storage, a repository is able to provide specific services to a user. Through creating a conversation with the repository it is envisaged that an environment can be created in which the repository is providing added value to the user. Within the project this conversation allows distributed item curation and continued editing. Essentially all this boils down to making users’ lives easier and making these services appeal to new users. Why click 20 buttons and fill in three forms when you can just click one button?

Posted in Uncategorized. Tagged with , , .

By davetaz
February 6, 2012

No comments

User testing results 4: what users said

We are about to complete the results of the user tests of the novel and original repository deposit tools developed in the DepositMO project.

Based on the tests we have a number of ways to analyse the results. Previously we considered what the timings and task completion tell us about the usability of the deposit tools, and what users did, based on the observer notes and an assessment of the repository records they created. Here we discover what they thought of the tools they were testing, prompted by a series of questions in the user’s test documentation and placed to be answered immediately at the conclusion of the practical part of the test.

We present just the summary results and selected extracts from this user feedback, so we do not have another over-extended post. If you wish to see the unexpurgated comments, here is the original report of the full test results (pdf).

Summary results

1 Would these tools encourage you to deposit more of your own content in a repository?

Yes (8) No (5)

2 Would these tools encourage you to deposit types of content that you have not previously deposited in a repository? Which type?

Yes 5 (inc. images, multiple presentations/teaching resources, computing files) No (6) ? (2)

3 Which of the two new deposit tools (in Word, and in the file manager) used here are you likely to use if it was generally available?

Word (7), File manager (1), both (3), neither (1), repository (1)

6 Or are you more likely to continue to use the standard repository deposit interface?

Yes (6), No (0), ? (4), N/A (3)

These summary results suggest that on balance use of these tools might encourage more deposit, just, most clearly in the case of the Word tool, but it won’t be so easy to wean users off the standard repository deposit interface on the basis of these tools.

Selected extracts from fully transcribed results

Here are some extracts to provide a flavour of the user comments. These are not intended to be fully representative, for which please refer to summary results above and the full test results document.

1 Would these tools encourage you to deposit more of your own content in a repository?

User2 (EdShare) Yes For me the Word tool. The MO (file manager) deposit was initially exciting but the difficulty of differentiating between many files was frustrating as filenames were not carried over. The updating via Word for single files and also the addition of new items via MO leading to either loss or overwriting of previous metadata is v. frustrating.

User 3/1 (Library) The Word deposit seems very simple, provided it’s easy to get the DepositMO add-in. I would like to know if I need to construct my Word docs in a particular way. Image deposit (file manager) seemed complicated and a bit confusing – not sure I’d use this.

User 5/1 (Kultivate) If uploading Word files would definitely use the DepositMO tool. Not sure about other content.

2 Would these tools encourage you to deposit types of content that you have not previously deposited in a repository? Which type?

User 1/1 (EdShare) No. Folder deposit confusing

User 1/2 (EdShare) No. Word integration confusing

User 3/2 (Library) In theory – yes – but possibly only Word as that was easiest!

User 5/2 (Kultivate) I think yes for PDFs (and Word docs) – of course the images still seem to require more work/metadata

3 Which of the two new deposit tools (in Word, and in the file manager) used here are you likely to use if it was generally available?

User 1 (EdShare) File manager. Word version confusing in version control, settings and location of files

User 6/1 (Archaeology) The Word one is nicer and a good idea, but all are confusing

User 7/2 (Archaeology) Word was very useful for text. The file manager was good for images but possibly required a bit more prior knowledge.

4 How would you improve either of the tools?

User 1/2 (EdShare) Make actions of file manager clearer (what is being updated and where)

User 2 (EdShare) More metadata extracted from original file e.g. title from name, creator from owner (if available) and also type (Word added-both showed as HTML files)

User 3/1 (Library) Word – I wouldn’t require user to add/edit URL; Image – I think this needs to be made more user friendly

User 3/2 (Library) Metadata requirement (i.e. having to add title in HTML) in file manager prob off-putting to some.

5 Are there features of the new deposit tools that would deter you from using them for repository deposit?

User 2 (EdShare) Overwriting metadata which had been hand-edited when updating or adding additional files is a real turn-off.

User 3/1 (Library) Image deposit is too complicated. I wouldn’t remember the URL needed for Word deposit

User 3/2 (Library) I have nothing to compare this with, but I did find too many windows open caused a degree of confusion

User 5/1 (Kultivate) With uploading images etc … I think the time taken in standard way is about the same, due to adding metadata. I think the ‘watch folders’ idea is brilliant but needs to be developed further.

User 6/2 (Archaeology) The fact that each file’s metadata had to be in its own folder with that file -> hard to deposit large directories of files.

6 Or are you more likely to continue to use the standard repository deposit interface?

User 1/2 (EdShare) Yes. Better/clearer control. Less ambiguous.

User 2 (EdShare) Would like to be able to use the new deposit tools more as the traditional, although v. good for library geeks like me, is long-winded for the normal academic.

User 6/1 (Archaeology) I think yes because it is proven to be efficient

Any other comments?

User 5/2 (Kultivate) Really helpful for documents – I can see that for repository managers and even self-depositors this could be helpful, saving quite a bit of time.

User 6/1 (Archaeology) Good ideas but they are not easy enough to use. They seem slow and frustrating.

Once again, we thank our users for engaging fully with these tests, for the time and effort they gave to participate, for their rigour and objectivity. As we saw earlier in the live audience demonstrators, it’s when you put software tools in front of users that they come to life.

Our test users were: I. Atkins, Joy Caisley, Grant Cox, David Davies, Helena Dmetriou, Harry Gibbs, Nick Graffy, Marie-Therese Gramstadt, Matthew Harrison, Alisa Miller, John O’Hagan, Isobel Stark and Adam Warren.

I must also thank the organisers of the user test teams, who doubled as test observers to report on individual tests. Our observers were: Penny Copeland, Debra Morris and Kate Walker.

This completes the recorded results from the DepositMO user testing. In the remaining posts, as the project ends we will seek to draw some lessons and conclusions from the tests and from the project overall. These will be our interpretations of the results. If you have a view on these results and what they might tell us about the deposit tools developed by the project, or on the prospects and needs for future repository deposit tools, please leave a comment.

Posted in Uncategorized. Tagged with , , , , , .

By Steve Hitchcock
February 3, 2012

No comments

User testing results 3: what users did

Based on the tests we have a number of ways to analyse the results. Previously we considered what the timings and task completion tell us about the usability of the deposit tools. Here we get some insights into the times taken to perform the tasks by using the records of what users did, taken from the observer notes and an assessment of the repository records they created.

For completeness, and since it did not seem appropriate to break the post between users, this is necessarily a much longer post than usual on this blog.

User 1

Pair. Experienced repository users, many deposits of a variety of content types over years.

From repository record

Hard to match record with observer notes and intended test chronology. Ended with 5 deposits of doc 1. Took a long time to deposit image 1. Appears to make some false starts with image deposit, did PDF, then returned to image 1 and ended with a collection of 4 images, including all 3 images provided in the sample data for the test and an additional image not provided. Deposited doc 2 (no metadata) and updated by adding a new item (with metadata).

Extracts from observer record

Deposit item 1 (doc 1) Clicked submit x2 so resulted in 2 deposits.  “Would have expected an update” – both agreed. It’s like a ‘save’ button – expect new copy of the doc. Item type – should be doc not unspecified

Deposit item 2 (image 1) Didn’t work first time. Then did work when refreshed screen for manage items = Revision 11 though! But there doesn’t seem to be anything actually there? Ghost image and also actual file. Went back 3rd time for item 2 -> added another image as the revision of the first Separate folder for each item? How would it tell which version? Well that worked = Rev 14 did what was expected

PDF Subjects -> LC/Dewey – choice? Tried to search for several subjects.

Image 2 Seems odd that would need to create a separate folder for each item. If I have to create a new folder for each item I am never going to use this!! Cut item from original folder. Waited for metadata file. Revised metadata – abbey by night. Resulted in a deposit with 4 files

Doc 2 MS Word error message about the file because it seemed to take the title and corrupt the file. Going back and try again – create new folder. Doc not corrupt at moment. Used short cuts to copy and paste -> my doc which I’ve spent months working on may have been corrupted … Titles seem to be switching around … and the doc seems to be fine when opened up in the repository

Updating doc Error message – forgotten details which had been provided – username and password but no location -> went to item which was locked. Hopelessly lost. No confidence about which document we are working on

Click on’ Submit’ – then this created another copy Characterising and preserving

How does it know which doc I want to update rather than creating a new version?

User 2

Single. Experienced repository user, many deposits of a variety of content types over years in different repositories.

From repository record

Appears to have completed all stages in correct order, and deposited a substantial number of own items including a variety of document and format types, although the collections appear to have become mixed within records, some duplication of items. All repository records can be identified by title, with supplementary metadata added to describe items within records. Looks like an exemplary result.

Extracts from observer record

Opening Word user panel. Missed (the Word deposit tab) at first; looked at repository. Needed prompt to ‘show’ the panel.

Image 1 Had closed depositMO folder after first use. Eventually realized it had to be reopened.

Doc 2 Reset user panel correctly. Added more metadata through EPrints deposit interface

Image 2 Named folder (‘Bath’) this time. Again added more metadata through repo. interface.

Update doc ‘lost some of my added metadata’ e.g. item type. Went back to add again

Add image User waited to see if more VIEW, METADATA files were generated. Noted they didn’t. ‘lost metadata again’. Added again through repo. interface

Own content Deposited 2x pdf + ppt via file manager tool. Added metadata via repo. interface. Added Word doc via panel, added metadata. Then copied ALL own content into one folder. Added one extra item to collection which overwrote previous title and added title of new item.

Parting comment: ‘Like Word’. Would like other (Office apps). That would encourage. Don’t like losing metadata.

User 3

Pair. One is an occasional repository user, the other a non-user.

From repository record

We don’t have an end timing for the short break in the process prior to update doc, hence the span of time for this task and for the total deposit time. A lengthy time for first deposits of both doc and image. No image deposits can be found in the repository workspace for this user. On inspection, these deposits can be traced to the workspace of User 1, probably due to an issue with the user configuration file in the ‘watch folder’, which directs content to the user’s login space in the repository. Image 2 was at first added to same folder as image 1, but then deposited separately in own folder. The docs and pdf do not use the config direction and appear correctly in this user’s repository workspace, with a duplicate entry for doc 1, all identified with metadata. No own content.

Extracts from observer record

Opening CONGIG file. Doesn’t automatically open – users had to open this in Notepad

Opening Word user panel. No label to say it is ‘user panel’. Author panel had a label, which caused some confusion.

Resetting repository location in Word user panel. My testers missed out this step, so the process didn’t work.  They received an error message ‘couldn’t understand response from server endpoint’ and realized their mistake.

Setting login in Word user panel. Password defaults to admin – confusing.  No confirmation when new password is entered to say that the username and password works.

Doc 1. Two identical records were added to DepositMO. . Users updated one, so there may be two on the screenshot with different times. Some confusion over ‘item type’ and what this meant. The testers had a discussion at this point about metadata, and how the repository found the title, as it wasn’t the title of the document.

Image 1. Testers not entirely sure where the depositMO folder was. Users said they didn’t understand what the script was meant to do.  They misunderstood and thought they had to copy the image into the repository itself. On adding metadata to the image, testers were disappointed with this – they didn’t understand exactly what they were doing.  They expressed doubts that academics would do this (XML text edit). Image 1 didn’t appear in the repository, although clicking on the View_item.html took them through to the appropriate record.  It appears that drag and drop was adding the record to the ‘user1’ account, even though they had updated the Config file correctly.

Doc 2 Deposited using the Word App

Image 2 Deposited using the Drag & Drop method. The testers didn’t understand the point of the different folders. Testers didn’t set up different content folders at first, but realized their mistake and made a second attempt.

Testers had remembered most steps at this point. Testers commented that there are too many steps, it is not straightforward, and it is not that different to the existing eprints interface in that regard.

Add image. Testers didn’t have time for this, but had inadvertently done this already.

Testers queried again how much more metadata you would have to add if using the Word app and the Drag and Drop options.

User 4

Pair. Both are non-users of repositories.

From repository record

As with user 3, which was simultaneous with this user test, image 1 and the added image appear in the repository workspace of the user 2 account, although this was avoided for deposit of image 2 by using the conventional EPrints deposit interface. Image 3 was added to image 1 and so also ended in user 2 account. As with user 3, this may partly account for the time taken to deposit image 1. All repository entries have at least basic metadata description and are identifiable in the list of records, including two items of own content but excepting the drag-and-drop items that ended in the user 2 account.

Extracts from observer record

Doc 1 “Show’ panel not obvious – “need explanation” Prompted

Image 1 Just deposited item, not in list of records but can view item

Doc 2 Word user panel – need to update again “find it hard to remember what have done”

Image 2 Used EPrints deposit interface

Update doc Went to repository ‘Edit item’ button. Prompted – back to Word panel. “Check if it worked”

Own content 1 Word doc – deposited, then added more metadata through EPrints deposit interface

Own content 2 pdf – using EPrints deposit

User 5

Pair. Test users only to date, but expect to be more regular repository users in future.

From repository record

An unspecified item appeared in this user account just prior to deposit of doc 1.

Image 2 is described, but image 1 with added image 3 are not.

Unusually for these tests the pdf is fully and correctly described, this taking slightly longer on this task than other users.

A large amount of own content was added using the file manager tool. These were packaged content – a ‘web site’ and zipped archive – and the aim was to see how the tool extracted and presented the contents. No metadata was added.

Extracts from observer record

Doc 1 Two unspecified objects in repo at this point + deposited doc. Picked up metadata from doc inc abstract: “Good”. Does doc need to be “tagged” for this to happen? “Good to have item type.” “Could improve description.” Started to add metadata via EPrints interface – “off piste”.

Image 1 Copied doc by mistake. Deleted – but it returned. Then successfully  copied intended image. “Quite fun.” “Could be more user-friendly – populate metadata rather than XML.”

PDF Couldn’t copy-paste abstract. Worked 3rd time. Added metadata carefully

Doc 2 Want Word deposit set to default URL. “Like the way it flags up” missing metadata. Checked what’s there/not there: “still quite a few required fields not populated”

Image 2 Chose to use EPrints interface – “like to be able to add keywords, contextualise, but don’t know creator, etc.

Update doc Users checked this replacement but couldn’t identify which document to open at first

Add image This instruction assumes they would have used the file deposit tool to add image 2 – they hadn’t (see above). Added image to existing folder. Then didn’t know which item it had been added to – at first.

Own content “Quite like to test uploading a Web site.” The file tool grabbed lots of pdf files. Zip file: “Maybe it doesn’t like zip files”

User 6

Pair. Mixed experience pair: one had not previously used a repository; the other has used a media and image repository rather than an open access institutional repository such as EPrints.

From repository record

We do not have definitive observer time checks for start of deposit, hence the span of time for doc 1.

Doc 1 is listed with a title, but there is no other metadata in the record and full-text is not available.

Image 2 was deposited twice, accounting for the extended deposit time for this item. Added to same folder as image 1, then deposited as separate item as required. Title metadata added

PDF added with minimal metadata.

Own content included a large video (700 MB) and further attempts to deposit image 1 (an archaeology picture by a colleague)

Extracts from observer record

File manager deposit tool setup: user A didn’t understand the config file set up although user B went straight to it and just did it with no comments

Doc 1 Options did not seem straightforward, there was clicking to find things and then going back. “We can’t follow simple instructions” – they are too impatient they think. I think they were looking for a way to upload a new description but couldn’t find one. There is confusion around the terminology of “submit” and “deposit”. Tried to submit it twice because of confusion but recognised that when they got a warning or error message.

Image 1 Confusion about folders for deposit but reference back to the instructions helped. Some uncertainty about what to put in the title

Doc 2 Error message for address to deposit – some confusion until the right place found. Some confusion from two people working together not knowing what button the other has pressed. Some problems when didn’t submit properly. They tried to submit again. Some confusion about how much information you HAD to put in before it would allow you to submit. What does “user work area” mean?

Image 2 Managed to submit things and then made the new folders which was considered to be confusing. Metadata file appears to have been rewritten – it did not contain the information they had submitted. Different file structure for this part of the test. It takes a while for the metadata file to be written. The metadata options generated some uncertainty with data deleted or overwritten and so the required things to enter could not be found. They lost the object that they had put in by putting it in the same folder as something else. They put it in again. Considered to be very fiddly, and creating a new folder for every file at this speed is too time consuming with the delay in the creation of the metadata file. AND then another object appeared in the folder – seemed to be a serious time lag for this to happen i.e. reentering stuff that is there but hasn’t appeared yet… “This is evil”.

Update doc Success first time.

Add image Worry and concern that this would break the system again! But some uncertainty about how this as a system would work, i.e. illogical? Problem – they think that the program keeps rewriting their metadata. The program is confused about the files and the metadata. Trying to sort it out took some time – Had to use the details to work out what was happening.  Altered through the web browser (EPrints interface) which a takes a long time to update the metadata file itself.

Own content Using the EPrints interface as it was considered to be the easiest way to do it. Video took a long time to upload but no problems.

User 7

Pair. Both are regular and experienced users of the Blackboard repository, and have deposited a range of content types.

From repository record

An ‘unspecified’ and unexplained record appears first, as for some other users. The first doc deposit has metadata, but doc 2 unusually has no description, although it is updated correctly. The PDF is deposited with made-up metadata. All texts are deposited into the repository review area. From the summary timings, only the image addition is notably longer than other tasks. Deposit of image 1 appears to be unsuccessful, but it appears with image 2 (not according to instructions), both attempts listed as ‘unspecified’. Then image 2 is joined with image 3 (as instructed) but with the title ‘Waterwheel’. Finally, own content is a series of 6 images collected, again, under the name ‘Waterwheel’.

Extracts from observer record

Setup demo repository Clicked ‘New Item’. Added an unspecified object?

Setup file deposit tool, open watch_folder Incomplete

Image 1 Needed to advise opening watch_folder. “What are the unspecifieds that keep popping up?”

PDF Performed deposit without reference to the actual document provided

Image 2 Copied to same folder as image1, then set up new folder. Added metadata via XML page (did not display)

Update doc Opened new copy of doc2 in new Word window. Update didn’t work. Not previously submitted version – opened new? Found path in original eprint, updated successfully. “Hasn’t identified title” Wiped metadata, “really annoying”

Add image “went in straight away”

Own content 6 images copied in one folder. “Worked well”. Opened XML metadata page for folder. Then added metadata via EPrints form.

Found there were two items in repo! Renamed wrong item. Re-copied. Deleted generated metadata.xml. Failed to recreate. Copied again. Added metadata via EPrints form.

The next post will show what users said about the test and tools they used, based on feedback from the test form completed immediately at the conclusion of the test.

Posted in Uncategorized. Tagged with , , , , , .

By Steve Hitchcock
February 3, 2012

No comments

User testing results 2: times taken to perform tasks

Based on the tests presented to users we have a number of ways to analyse the results:

  1. Times taken to perform the tasks
  2. What users did, based on the repository record and observer notes
  3. What users said, based on feedback from the test form

Subsequent posts will reveal what users did and said. Here we focus on what the timings and task completion tell us about the usability of the deposit tools.

User tests were performed over a series of dates during June and July 2011. Some tests were simultaneous and co-located. In total 13 users performed the test in 6 pairs and one singly (recall, tests were set up for pairs, to provide the opportunity for reactions to be vocalised between users, and to avoid equipment limitations for the controlled test environment).

Users 1, 2: June 3
Users 3, 4: June 17
User 5: June 20
Users 6, 7: July 5

All users completed the same tasks, guided by the same instruction document. Minor amendments were made to the document between tests to improve and clarify procedure, as pointed out by users, where problems were deemed to be due to the test documentation rather than the tasks and tools under test.

Procedure For reference, this is a quick chronology of the deposit test process outlined in detail in previous posts:

  • Deposit a Word document (using Word tool) Doc 1
  • Deposit an image (using file manager tool) Image 1
  • Deposit a pdf using the standard EPrints repository interface (PDF)
  • Deposit a second Word document (tool chosen by user) Doc 2
  • Deposit a second image (tool chosen by user) Image 2
  • Update the second Word document (Update doc)
  • Add an image to a collection including the second image (Add image)
  • Deposit own content (Own content)

Results

Time taken to perform set tasks

Table 1 shows times grouped by process and tool rather than chronologically.

Table 1. Time taken by users to perform set tasks in deposit test process

User 1 User 2 User 3 User 4 User 5 User 6 User 7
Total test time 73 mins 75 mins ? 84 mins 80 mins 71 mins 64 mins
Deposit time 61 67 44-54 63 56 46-51 47
1st doc 4 3 10 5 7 2-7 3
2nd doc 5 5 2 5 6 3 2
Update doc 7 3 1-10 6 2 3 4
Total doc time 16 11 10-19 16 15 8-13 9
1st image 20 6 12 14 4 5 6
2nd image 5 5 6 3 6 10 3
Add image - 4 - 4 5 7 11
Total image time 25 15 18 21 15 22 20
PDF 8 10 14 13 17 5 8
Own content 10

two videos

 30

doc ppt xlsx pdf

 - 10

doc, pdf

 8

zip, web site

 8

video

 8

images

Notes on Table 1

Times were recorded by observers for selected stages in the overall process. The repository timestamps, including the EPrints history for each record, were used to calculate times of individual tasks.

Where a time duration is unknown, or where a time range is given, the observer notes are incomplete.

Where no time is given, the procedure was not completed.

Where the process appears to be continuous, it is assumed that the last recorded deposit time for an item is the start time of the next deposit process, even if the first recorded deposit time for the subsequent item is some minutes later. This assumption is based on the need for users to assess the instruction and decide which actions to take, and that the time taken to do this is an implicit part of the deposit process for each item.

Not all user processes were continuous or followed the order of the instructions. Where this happens timings are likely to be less reliable, based on an assessment of the record and a judgement of how users proceeded. Such instances will be specified.

e.g. User 1: the deposit of images and the PDF appear to be interleaved, so apportionment of the times between images and pdf are approximate.

Deposit time ≠ Total doc + Total image + PDF + Own. Allow ± 1 min for each action.

Deposit time ≠ Total test time. Additional time is taken to read the documentation for the procedure, and to complete questions before and after the deposit process.

Repository clock set 1h slow. (This is only relevant when comparing observer timings with repository record.)

General observations on times

The total time taken to complete image deposits was in all cases longer than the time taken to deposit and update Word docs. There was more variability in times of image 1 deposit, but less so for subsequent image deposits when the procedure was more familiar. Some image deposit cases were not completed, but all doc deposits were completed.

In the next post we will look at what users did, based on the repository record and observer notes.

Posted in Uncategorized. Tagged with , , , , , .

By Steve Hitchcock
February 3, 2012

No comments

User testing results 1: user profiles

We are about to learn the results of the user tests of the novel and original repository deposit tools developed in the DepositMO project.

To understand the context of the test results we need to know something about the users who performed the tests. All users, apart from two, are based at the University of Southampton, the home of the DepositMO project. We are grateful to them all for their participation, and will acknowledge them individually at the conclusion of these results.

The test instruction document included two questionnaire sections:

  1. Before any deposit actions, to find out the prior experience of testers in using repositories.
  2. After deposit tests, to discover their reactions to the tools used in the test.

This post summarises the findings from part 1.

  1. Have you deposited content in an institutional repository? Yes 9 No 4
  2. In which repository do you deposit most? EdShare (3 users), EPrints (3), Blackboard (2), MediaBin (1), none (4)
  3. Do you deposit content regularly? Approximately how many items have you deposited in this repository? Over how long a period have you been depositing? See Table 1
  4. What type of data have you deposited? Presentation slides (4 users), research papers (PDF) (3), images (3), video (3), docs (2), theses (2), audio (1), research data (1), spreadsheets (1), handouts (1), reports (1), essays (2), computer project (1), database (1).

Table 1. Brief history of repository deposits by our test users

Regular deposit Number of deposits Time period
No 70 2 years
No 30(ish) 2 years
No Bursts 1-4 years
No 2 Ad hoc
-
-
-
-
-
? 100s (images) 3 months
-
? 4 10 months
? 15 1 year

This presents a picture of a mix of experienced and new repository users, a reasonable profile for this test since the tools are aimed at both types of users. Among the experienced users, none claim to deposit on a regular basis, but some show many deposits of a wide range of data types over an extended period. In this presentation we will not be seeking to stratify the results between users and non-users.

In the next post we start to learn how these users got on with the tasks set for them in this test of new deposit tools.

Posted in Uncategorized. Tagged with , , , , , .

By Steve Hitchcock
February 3, 2012

No comments

User testing: instructions for users 3: depositing content

User testing depends on providing clear instructions to users. This is the third of three posts providing key sections of those instructions, intended to inform readers interested in understanding the results of user tests of the new deposit tools developed in the DepositMO project, and taking a critical view. If you have followed the first two posts presenting the instructions for users, you may have decided already not to download the full instruction document (pdf) as originally presented to the test users, but you can still do so.

In this final extract we learn exactly what users were asked to do for the test, as it is time to start depositing some content. Why is the test structured in this way? At this point it may be worth referring back to the earlier post on the design of the test procedure.

Control ribbon in Word 2010

Figure 2. Revealing the control ribbon in Word 2010 for the deposit application

Time to start depositing some content

By now you have set up the drag-and-drop deposit tool, you have some content, and you are logged in and ready to deposit something in the demo repository.

Support and assistance is not available for this part of the test, except to clarify the instructions given here.

If you are working in a controlled group you may be observed during this process. Both your actions and comments may contribute to the results. Again, we should stress the results from this test are intended to inform about the tools rather than the users, so don’t be shy of commenting. You may make comments to the observers, but during this part of the test they are unable to engage in dialogue. If you are working in a pair your discussion may be helpful in providing recordable evidence for the results.

Deposit item 1

From the sample_data folder open doc1 in Word. Now we have opened Word we can use the deposit tool that has been added to Word. First we will check the setup of the tool.

  • Open the DepositMO (alpha) tab and ‘show’ the user panel (Figure 2)
  • In the panel ensure that Repository location is set to http://depositmo.eprints.org/id/contents
  • Set the panel to your correct login details – same username and password  as your repository login.

Using the panel, ‘Submit’ the document to the repository. Go to your Web browser with the demo repository open, and click on (or reload) Manage deposits (Figure 1) to see the content you have just deposited. How is it described? Could you improve the description?

Deposit item 2

This time we will use the file manager tool, which we set up earlier. Find image1 in your sample_data folder. Copy this file using the short-cut keys for your machine. Return to your depositMO folder on the desktop, and add a new content folder (name it as you wish). Using the short-cut keys, paste image1 into the new content folder. The tool should now automatically deposit the copied image into the repository.

Open the new content folder and you should see two new files appear: VIEW_ITEM.html and (after another 10s or so) METADATA.xml. Clicking on VIEW_ITEM.html will take you to the repository, showing the record for the item you just deposited. Open the METADATA file by right-clicking and selecting a text editor (e.g. Notepad) to show the editable metadata extracted from the file for the repository. Add a title by including this line (e.g. just above <author>):
<title>Archaeology image</title>
Then ‘Save’ the file.

Again, click on Manage deposits in the demo repository to see your current list of deposited items. You should see an item listed with the title you just added.

Deposit item 3

We want you to deposit pdf1 using the standard repository deposit interface. From your Manage deposits screen in the demo repository, click on the New Item button (Figure 1). Deposit item 3 in the usual way using the EPrints interface.

Hint. After clicking the Choose File button browse to the sample_data folder via Recent Places. When adding metadata using the deposit form you only need fill in the required fields to complete the deposit.

Deposit items 4, 5

You have used both new deposit tools and the conventional repository deposit interface. Now deposit the next two items (doc2, image2) in your sample_data folder, using the deposit tool you think is most appropriate.

Hint. If you copy to the depositMO folder you may wish to set up a new content folder for each item deposited.

Short pause. At this point please ask the observer or test moderator to take a screenshot of the Manage deposits list in the repository window.

Moderator’s note. We need a time check here, before some of the items are updated and the Last Modified time changes. How long did it take to deposit each item up to this point? Timings are a rough-and-ready indicator of the ease and speed of the different deposit processes.

Updating a document

In this case we want to update the content of doc2, which you have already deposited.

Open item 7 (doc3) and copy the paragraph of text. Return to doc2 in Word – unless you closed this it should still be open in a separate window. Paste the copied paragraph where indicated in doc2. ‘Update’ the document in the repository using the appropriate button in the panel. In this way the updated document should replace the original.

Updating an item

This time we want to update an item in the repository by adding another image of Bath (image3) to the item containing image2. Unlike the case above, we are not replacing the original content, and the updated item should display both images.

Hint. You will need to return to the depositMO folder but in this case you do not need to set up a new content folder. Instead add the latest image to the appropriate folder among those you have already set up.

Depositing your own chosen items

Did you bring your own content? Given an open choice, what content do you want to be able to deposit in a repository? As with items 4 and 5, deposit your content using the appropriate tool.

Improving description

Your Manage deposits view of the repository should now contain a reasonably extensive list of contents, but can you tell which is which? Is each item sufficiently described for identification by you, or for discovery by others?

This concludes our extracts from the instruction document presented to test users. In the next posts we will find out how those users fared with the test, as we begin to analyse the results.

Posted in Uncategorized. Tagged with , , , , , .

By Steve Hitchcock
January 31, 2012

No comments