Lessons learned

Integrating usability into your work patterns means being able to demonstrate that you are in touch with the full range of users in a non-jargonistic way that excludes no-one. If you want to bootstrap usability, you will find that it has to influence and inform every single step of the development process and the relationship the project has with the wider organisational/business context in which it exists.

The biggest cultural shift is to make the usability analysis usable, i.e. to make the projects outputs extensible, transparent and traceable. The project will now be able to demonstrate an existing framework of usability into which any new development can be placed, discussed and tracked. Informal relationships with key users can now become formalised, events such as workshops can be opened up appealing to different audiences. Perhaps these workshops could also be taken out 'on the road' to visit institutions which development teams would not ordinarily get the chance to go to.

One potential dividend would be to build up experiences on the development of generic pieces of functionality, e.g. a registration form, a search form. New projects could then build these items of the back of hundreds (thousands?) of hours of research and development into these functions within the HE sector - an enviable knowledge base of good practice into which their own process could be recorded.

Usability practice has been integrated into our approach with no consultancy input - whilst there may be a need to create resources to define what constitutes the usability toolkit, this is liable to change over even a short period of time. Therefore, for usability to self-propagate, development teams across the HE sector need ways of communicating and discussing these approaches through worked examples. From a personal standpoint, there's no shortage of advice on the matter - what would be more useful if how other HE teams, facing the same institutional pressures, have actually run projects.

The funding for this project has generated a large amount of raw data on usage patterns which could inform other projects (e.g.  library catalogues) at their outset. The opportunity is there to for this to be a two-way process - to give results back to the community upon project closure and so build up a corpus of material which could potentially be used in a longitudinal study of systems support in HE.

The usability process seeks to make clear specific obstacles and impediments which your users face; it would be ironic if the approach taken to more closely integrate usability practice into HE systems development relied on generalised case studies and theory.

How successful have we been?

Our project plan lays down two kinds of success metric: quantitative and qualitative. However, a secondary aim is looking at identifying a method whereby these practices could be built into existing projects without much additional cost and enhance the quality of the next generation of software being built in UK HE. This is essential for niche areas of scholarship for which supporting software needs a high degree of innovation and no other product currently meets the needs of researchers.

Quantitative ('what people do')

Each identified issue have been separately reported; a digest appears below.

• Issue 1: met
• Issue 2: partially met
• Issue 3: met
• Issue 4: partially met
• Issue 5: partially met
• Issue 6: no change
• Issue 7: unmet
• Issue 8: no change
• Issue 9: met

Overall, this list most closely reflects my personal view of the success of this project - that in articulating user needs, it has created a mandate for change which extends far beyond what is possible within this project's timescale.

Qualitative ('what people say')

It was not possible to schedule re-interviews following the modifications being made live but here are the differences in the results from the System Usability Scale (SUS), a link to which appears on every page in BHO.


SUS before development

• Best imaginable: 40
• Excellent: 33
• Good: 28
• OK: 22
• Poor: 16
• Awful: 14
• Worst imaginable: 14

SUS after modification

• Excellent: 40
• Good: 28
• Poor: 21

More time is needed to build the level of response as the project had to report within 3 months (extended to 4) and perhaps we tried to cram too much in. Also, the SUS is not promoted actively (the click tests above were, for instance, publicised through the site news and blog); as a result, a lower level of response is to be expected and so a greater period of time is required to build up an impression of any change in the pattern of satisfaction.

Approach

We have successfully implemented a usability-centric approach, without consultant input, which covers the needs of a temporary project revolving around one set of software updates as well as providing the means for an ongoing inclusive dialogue across all functional departments (technical, editorial, managerial, marketing etc). There have been virtually no direct costs; the process has been devised, developed, implemented and reported on by existing staff, with the intention of making the results and as much of the raw data available for re-use by the HE community.

In addition, it is extensible; given the resources, each issue could be revisited, redeveloped and retested; new issues in other areas could be added to the issue list document. By publishing the information openly using a blog platform, the entire process is opened up for discussion and analysis.

It lends itself to networking/discussion and gives development teams the opportunity to discuss approaches to improving software beyond institutional technical environments.

Bootstrapping usability

As more and more content was added to British History Online (BHO), the listings pages and search results became longer and users were confronted by growing amounts of information which they would need to sift through to help them decide on which sources were going to be relevant to their research.

To prevent users becoming overwhelmed by the volume of information (thus impairing their ability to use the site for research), we undertook a usability project as part of the JISC 01/11B funding round. Our intention was to modify our way of working, building in usability practices, to find a self-sustaining way by which usability could be built into our working pattern and persist after the funding round had ended, i.e. bootstrapping.

Many project teams will be wondering how to employ usability without a budget  - and that's the model we followed. Our only direct cost, a rolling subscription with VerifyApp.com, was USD10 per month but enabled the crucial quantitative aspect to our investigation.

Our plan envisioned us researching qualitative and quantitative feedback, altering the BHO interface according to recommendations, and then re-testing both sets afterwards to give a before and after style report. It wasn't all possible (we weren't able to schedule interviews after the development) but most of it is in place and we reported the results through this blog.

Here is an outline of the work:

• interviews with historians
• focus group with the Survey of London at English Heritage
• benchmarking of System Usability Scale (SUS) before developments
• production of the issue list
• Click testing for each issue
• Development and deployment of new and modified functions on BHO
• Click testing for each issue using new modified interfaces
• Review of SUS results after modified interface went live
• Publishing the findings

At its heart, the project produced an issue list - a jargon-free document containing a clear description of the issues, how to reproduce them, screen shots and suggested questions to ask users to test any remedial development - it was the most important managerial output from the exercise because it could be understood by anybody thus providing the means for support for the changes to come from a range of different departments.