Lessons learned

Integrating usability into your work patterns means being able to demonstrate that you are in touch with the full range of users in a non-jargonistic way that excludes no-one. If you want to bootstrap usability, you will find that it has to influence and inform every single step of the development process and the relationship the project has with the wider organisational/business context in which it exists.

The biggest cultural shift is to make the usability analysis usable, i.e. to make the projects outputs extensible, transparent and traceable. The project will now be able to demonstrate an existing framework of usability into which any new development can be placed, discussed and tracked. Informal relationships with key users can now become formalised, events such as workshops can be opened up appealing to different audiences. Perhaps these workshops could also be taken out 'on the road' to visit institutions which development teams would not ordinarily get the chance to go to.

One potential dividend would be to build up experiences on the development of generic pieces of functionality, e.g. a registration form, a search form. New projects could then build these items of the back of hundreds (thousands?) of hours of research and development into these functions within the HE sector - an enviable knowledge base of good practice into which their own process could be recorded.

Usability practice has been integrated into our approach with no consultancy input - whilst there may be a need to create resources to define what constitutes the usability toolkit, this is liable to change over even a short period of time. Therefore, for usability to self-propagate, development teams across the HE sector need ways of communicating and discussing these approaches through worked examples. From a personal standpoint, there's no shortage of advice on the matter - what would be more useful if how other HE teams, facing the same institutional pressures, have actually run projects.

The funding for this project has generated a large amount of raw data on usage patterns which could inform other projects (e.g.  library catalogues) at their outset. The opportunity is there to for this to be a two-way process - to give results back to the community upon project closure and so build up a corpus of material which could potentially be used in a longitudinal study of systems support in HE.

The usability process seeks to make clear specific obstacles and impediments which your users face; it would be ironic if the approach taken to more closely integrate usability practice into HE systems development relied on generalised case studies and theory.

How successful have we been?

Our project plan lays down two kinds of success metric: quantitative and qualitative. However, a secondary aim is looking at identifying a method whereby these practices could be built into existing projects without much additional cost and enhance the quality of the next generation of software being built in UK HE. This is essential for niche areas of scholarship for which supporting software needs a high degree of innovation and no other product currently meets the needs of researchers.

Quantitative ('what people do')

Each identified issue have been separately reported; a digest appears below.

• Issue 1: met
• Issue 2: partially met
• Issue 3: met
• Issue 4: partially met
• Issue 5: partially met
• Issue 6: no change
• Issue 7: unmet
• Issue 8: no change
• Issue 9: met

Overall, this list most closely reflects my personal view of the success of this project - that in articulating user needs, it has created a mandate for change which extends far beyond what is possible within this project's timescale.

Qualitative ('what people say')

It was not possible to schedule re-interviews following the modifications being made live but here are the differences in the results from the System Usability Scale (SUS), a link to which appears on every page in BHO.


SUS before development

• Best imaginable: 40
• Excellent: 33
• Good: 28
• OK: 22
• Poor: 16
• Awful: 14
• Worst imaginable: 14

SUS after modification

• Excellent: 40
• Good: 28
• Poor: 21

More time is needed to build the level of response as the project had to report within 3 months (extended to 4) and perhaps we tried to cram too much in. Also, the SUS is not promoted actively (the click tests above were, for instance, publicised through the site news and blog); as a result, a lower level of response is to be expected and so a greater period of time is required to build up an impression of any change in the pattern of satisfaction.

Approach

We have successfully implemented a usability-centric approach, without consultant input, which covers the needs of a temporary project revolving around one set of software updates as well as providing the means for an ongoing inclusive dialogue across all functional departments (technical, editorial, managerial, marketing etc). There have been virtually no direct costs; the process has been devised, developed, implemented and reported on by existing staff, with the intention of making the results and as much of the raw data available for re-use by the HE community.

In addition, it is extensible; given the resources, each issue could be revisited, redeveloped and retested; new issues in other areas could be added to the issue list document. By publishing the information openly using a blog platform, the entire process is opened up for discussion and analysis.

It lends itself to networking/discussion and gives development teams the opportunity to discuss approaches to improving software beyond institutional technical environments.

Bootstrapping usability

As more and more content was added to British History Online (BHO), the listings pages and search results became longer and users were confronted by growing amounts of information which they would need to sift through to help them decide on which sources were going to be relevant to their research.

To prevent users becoming overwhelmed by the volume of information (thus impairing their ability to use the site for research), we undertook a usability project as part of the JISC 01/11B funding round. Our intention was to modify our way of working, building in usability practices, to find a self-sustaining way by which usability could be built into our working pattern and persist after the funding round had ended, i.e. bootstrapping.

Many project teams will be wondering how to employ usability without a budget  - and that's the model we followed. Our only direct cost, a rolling subscription with VerifyApp.com, was USD10 per month but enabled the crucial quantitative aspect to our investigation.

Our plan envisioned us researching qualitative and quantitative feedback, altering the BHO interface according to recommendations, and then re-testing both sets afterwards to give a before and after style report. It wasn't all possible (we weren't able to schedule interviews after the development) but most of it is in place and we reported the results through this blog.

Here is an outline of the work:

• interviews with historians
• focus group with the Survey of London at English Heritage
• benchmarking of System Usability Scale (SUS) before developments
• production of the issue list
• Click testing for each issue
• Development and deployment of new and modified functions on BHO
• Click testing for each issue using new modified interfaces
• Review of SUS results after modified interface went live
• Publishing the findings

At its heart, the project produced an issue list - a jargon-free document containing a clear description of the issues, how to reproduce them, screen shots and suggested questions to ask users to test any remedial development - it was the most important managerial output from the exercise because it could be understood by anybody thus providing the means for support for the changes to come from a range of different departments.

BHO Issue 9: 'Users have no way of re-ordering a list'

Page

Listings and source

Heuristic

Subjective

Description

Listings pages are sequenced by an editorial weighting: if a user remembers a source by time period, or by name, they have no way of re-ordering the list. Although source pages are arranged alphabetically, they may be structured by time period but authored in a non-linear sequence; alternatively, later volumes may be more comprehensive than earlier ones.

Impact severity

High

Recommend

Add in-page options to change the sort order. Configure some sources to be sequenced differently by default.

Examples

The Survey of London is being added to but newer volumes currently appear at the bottom of the listings, well below the fold.

New volumes appear right at the end (large descriptions also don't help scrolling)



Quantitative measure

[Use the Lords Journals series]: if you wanted to see the most recent volumes first, click where you expect to be able to re-order this list

Actual question

Please click on where you'd expect to re-order this list to see the most recent volumes FIRST

Initial click test result ('before')

July 2011, 200 responses.

'Before'



Development change

Added toggling pair of links, Sort A-Z and Sort Z-A, right-aligned beneath search text box (Survey of London on British History Online).

Modified interface



Follow-up click test result ('after')

September 2011, 154 responses.


'After'



Reflection

Clear grouping on correct sort function but still a few looking for this feature on the left hand side. In hindsight, that is where the search sort functions were located (see earlier Issue 3) and it compounds the recurring theme that the site needs a clearly designed and consistent layout for a control panel, to be used wherever appropriate.

BHO Issue 8: 'It is not clear what the top sources are'

Page

Listings 

Heuristic

Learnability

Description

A page listing sources for a specific taxonomy dimension will list the most relevant sources at the top. However, they are visually indistinct from sources which contain a lower quantity of relevant content.

Impact severity

Medium

Recommend

Prominence to be given to sources which occupy the top n positions, possibly through a gallery device, after which sources have clearly lower relevance.

Examples

For Scotland, the list of sources moves from the Topographical Dictionary of Scotland to local records from Cumberland without any presentational cue. The underlying weighting figure drops by nearly 90% at that point.

Green indicates the most relevant for Scotland



Quantitative measure

Click on ALL of the records on this page which you feel are directly relevant to Scottish affairs

Actual question

Please click on the LOWEST record on this page which you feel is directly relevant to Scottish affairs

Initial click test result ('before')

July 2011, 200 responses.

'Before'



Development change

Descriptions of the five major source types were amended (Sources on British History Online).



Using a background colour (no doubt an accessibility issue)

Follow-up click test result ('after')

September 2011, 154 responses.


'After'



Reflection

Almost identical click patterns so the visual cue seems to be unnecessary here; plus, it would difficult to explain why sources appear in the highlighted panel or not without recourse to editorial guidelines (which themselves are often a matter of personal taste amongst users) and statistical calculation. It looks thankfully like this is less of an issue than we believed but we do need to trace this issue back to ensure we haven't missed something.

BHO Issue 7: 'Source pages are text heavy but little of that is "explanation"'

Page

Listings 

Heuristic

Learnability

Description

First time users, who might arrive through a search engine, have very little introductory text with which to 'ground' themselves. What explanation there is tells users where to find things, not what those things are. Returning users know what they are looking for and rarely use any descriptive signpost material. Screen real estate can be repurposed safely for first time users.

Impact severity

High

Recommend

Descriptive material suited to first time users to be created. Potential for a new verb-less strap line to go beneath the header.

Examples

The description for Primary sources confirms where you are, but doesn't explain what they are.

An issue of signposting



Quantitative measure

Click on the category where you would expect to find the Journal of the House of Commons

Actual question

Please click on where you'd expect to find the letters and papers of HENRY VIII

Initial click test result ('before')

July 2011, 200 responses.

'Before'



Development change

Descriptions of the five major source types were amended (Sources on British History Online).



Descriptions modified

Follow-up click test result ('after')

September 2011, 154 responses.




'After'

Reflection

Unfortunately, the third row contains the correct link and there's been a move away from it. In principle, this has been unsuccessful; however, the heterogeneity of sources on BHO defies generalisation. This list could easily be expanded to show the top 5 sources from each type to give a clearer idea of content without overwhelming the user.

It looks like we were too conservative on this issue - the follow-up click test has made that clear.

BHO Issue 6: 'Data structured geographically is not represented visually'

Page

Source

Heuristic

Memorability

Description

For a set of local history volumes, there is no visual way to assess coverage over what could be a substantial amount of data.

Impact severity

High

Recommend

If the source contains a map showing coverage, then provide that map at the source level with positioned links to the individual volumes or articles as appropriate.

Examples

The Survey of London volumes have been created over the course of 120 years and cover some of the most iconic locations in the capital. At the moment, the user would have to remember the descriptions of over 50 volumes to go straight to the one they want because the subtitles can be quite general.

Volumes which cover adjoining geographic areas are not published in sequence (e.g. vols 3 and 5)



Quantitative measure

Where would you expect to find an account of the church of St Giles in the Fields ?

Actual question

Please click on where you'd expect to find an account of the church of St Giles in the Fields

Initial click test result ('before')

July 2011, 200 responses.

'Before'



Development change

Insertion of new map facility with simple jQuery based zoom function at the head of the volume list (Survey of London on British History Online).



Follow-up click test result ('after')

September 2011, 154 responses.

'After'

Reflection

Difficult to make out any movement in clicks using the Survey of London. That may be because the question asked users to find an area for which an account existed - it may be that users who are researching areas not covered by the Survey are able to see more quickly that it won't be directly useful to their work.

It may be more use therefore in the structuring of a research project (i.e. the identification of relevant sources), rather than detailed research itself. Future usability studies should take account of this value.