Lessons learned

Integrating usability into your work patterns means being able to demonstrate that you are in touch with the full range of users in a non-jargonistic way that excludes no-one. If you want to bootstrap usability, you will find that it has to influence and inform every single step of the development process and the relationship the project has with the wider organisational/business context in which it exists.

The biggest cultural shift is to make the usability analysis usable, i.e. to make the projects outputs extensible, transparent and traceable. The project will now be able to demonstrate an existing framework of usability into which any new development can be placed, discussed and tracked. Informal relationships with key users can now become formalised, events such as workshops can be opened up appealing to different audiences. Perhaps these workshops could also be taken out 'on the road' to visit institutions which development teams would not ordinarily get the chance to go to.

One potential dividend would be to build up experiences on the development of generic pieces of functionality, e.g. a registration form, a search form. New projects could then build these items of the back of hundreds (thousands?) of hours of research and development into these functions within the HE sector - an enviable knowledge base of good practice into which their own process could be recorded.

Usability practice has been integrated into our approach with no consultancy input - whilst there may be a need to create resources to define what constitutes the usability toolkit, this is liable to change over even a short period of time. Therefore, for usability to self-propagate, development teams across the HE sector need ways of communicating and discussing these approaches through worked examples. From a personal standpoint, there's no shortage of advice on the matter - what would be more useful if how other HE teams, facing the same institutional pressures, have actually run projects.

The funding for this project has generated a large amount of raw data on usage patterns which could inform other projects (e.g.  library catalogues) at their outset. The opportunity is there to for this to be a two-way process - to give results back to the community upon project closure and so build up a corpus of material which could potentially be used in a longitudinal study of systems support in HE.

The usability process seeks to make clear specific obstacles and impediments which your users face; it would be ironic if the approach taken to more closely integrate usability practice into HE systems development relied on generalised case studies and theory.

How successful have we been?

Our project plan lays down two kinds of success metric: quantitative and qualitative. However, a secondary aim is looking at identifying a method whereby these practices could be built into existing projects without much additional cost and enhance the quality of the next generation of software being built in UK HE. This is essential for niche areas of scholarship for which supporting software needs a high degree of innovation and no other product currently meets the needs of researchers.

Quantitative ('what people do')

Each identified issue have been separately reported; a digest appears below.

• Issue 1: met
• Issue 2: partially met
• Issue 3: met
• Issue 4: partially met
• Issue 5: partially met
• Issue 6: no change
• Issue 7: unmet
• Issue 8: no change
• Issue 9: met

Overall, this list most closely reflects my personal view of the success of this project - that in articulating user needs, it has created a mandate for change which extends far beyond what is possible within this project's timescale.

Qualitative ('what people say')

It was not possible to schedule re-interviews following the modifications being made live but here are the differences in the results from the System Usability Scale (SUS), a link to which appears on every page in BHO.


SUS before development

• Best imaginable: 40
• Excellent: 33
• Good: 28
• OK: 22
• Poor: 16
• Awful: 14
• Worst imaginable: 14

SUS after modification

• Excellent: 40
• Good: 28
• Poor: 21

More time is needed to build the level of response as the project had to report within 3 months (extended to 4) and perhaps we tried to cram too much in. Also, the SUS is not promoted actively (the click tests above were, for instance, publicised through the site news and blog); as a result, a lower level of response is to be expected and so a greater period of time is required to build up an impression of any change in the pattern of satisfaction.

Approach

We have successfully implemented a usability-centric approach, without consultant input, which covers the needs of a temporary project revolving around one set of software updates as well as providing the means for an ongoing inclusive dialogue across all functional departments (technical, editorial, managerial, marketing etc). There have been virtually no direct costs; the process has been devised, developed, implemented and reported on by existing staff, with the intention of making the results and as much of the raw data available for re-use by the HE community.

In addition, it is extensible; given the resources, each issue could be revisited, redeveloped and retested; new issues in other areas could be added to the issue list document. By publishing the information openly using a blog platform, the entire process is opened up for discussion and analysis.

It lends itself to networking/discussion and gives development teams the opportunity to discuss approaches to improving software beyond institutional technical environments.

Bootstrapping usability

As more and more content was added to British History Online (BHO), the listings pages and search results became longer and users were confronted by growing amounts of information which they would need to sift through to help them decide on which sources were going to be relevant to their research.

To prevent users becoming overwhelmed by the volume of information (thus impairing their ability to use the site for research), we undertook a usability project as part of the JISC 01/11B funding round. Our intention was to modify our way of working, building in usability practices, to find a self-sustaining way by which usability could be built into our working pattern and persist after the funding round had ended, i.e. bootstrapping.

Many project teams will be wondering how to employ usability without a budget  - and that's the model we followed. Our only direct cost, a rolling subscription with VerifyApp.com, was USD10 per month but enabled the crucial quantitative aspect to our investigation.

Our plan envisioned us researching qualitative and quantitative feedback, altering the BHO interface according to recommendations, and then re-testing both sets afterwards to give a before and after style report. It wasn't all possible (we weren't able to schedule interviews after the development) but most of it is in place and we reported the results through this blog.

Here is an outline of the work:

• interviews with historians
• focus group with the Survey of London at English Heritage
• benchmarking of System Usability Scale (SUS) before developments
• production of the issue list
• Click testing for each issue
• Development and deployment of new and modified functions on BHO
• Click testing for each issue using new modified interfaces
• Review of SUS results after modified interface went live
• Publishing the findings

At its heart, the project produced an issue list - a jargon-free document containing a clear description of the issues, how to reproduce them, screen shots and suggested questions to ask users to test any remedial development - it was the most important managerial output from the exercise because it could be understood by anybody thus providing the means for support for the changes to come from a range of different departments.

BHO Issue 9: 'Users have no way of re-ordering a list'

Page

Listings and source

Heuristic

Subjective

Description

Listings pages are sequenced by an editorial weighting: if a user remembers a source by time period, or by name, they have no way of re-ordering the list. Although source pages are arranged alphabetically, they may be structured by time period but authored in a non-linear sequence; alternatively, later volumes may be more comprehensive than earlier ones.

Impact severity

High

Recommend

Add in-page options to change the sort order. Configure some sources to be sequenced differently by default.

Examples

The Survey of London is being added to but newer volumes currently appear at the bottom of the listings, well below the fold.

New volumes appear right at the end (large descriptions also don't help scrolling)



Quantitative measure

[Use the Lords Journals series]: if you wanted to see the most recent volumes first, click where you expect to be able to re-order this list

Actual question

Please click on where you'd expect to re-order this list to see the most recent volumes FIRST

Initial click test result ('before')

July 2011, 200 responses.

'Before'



Development change

Added toggling pair of links, Sort A-Z and Sort Z-A, right-aligned beneath search text box (Survey of London on British History Online).

Modified interface



Follow-up click test result ('after')

September 2011, 154 responses.


'After'



Reflection

Clear grouping on correct sort function but still a few looking for this feature on the left hand side. In hindsight, that is where the search sort functions were located (see earlier Issue 3) and it compounds the recurring theme that the site needs a clearly designed and consistent layout for a control panel, to be used wherever appropriate.

BHO Issue 8: 'It is not clear what the top sources are'

Page

Listings 

Heuristic

Learnability

Description

A page listing sources for a specific taxonomy dimension will list the most relevant sources at the top. However, they are visually indistinct from sources which contain a lower quantity of relevant content.

Impact severity

Medium

Recommend

Prominence to be given to sources which occupy the top n positions, possibly through a gallery device, after which sources have clearly lower relevance.

Examples

For Scotland, the list of sources moves from the Topographical Dictionary of Scotland to local records from Cumberland without any presentational cue. The underlying weighting figure drops by nearly 90% at that point.

Green indicates the most relevant for Scotland



Quantitative measure

Click on ALL of the records on this page which you feel are directly relevant to Scottish affairs

Actual question

Please click on the LOWEST record on this page which you feel is directly relevant to Scottish affairs

Initial click test result ('before')

July 2011, 200 responses.

'Before'



Development change

Descriptions of the five major source types were amended (Sources on British History Online).



Using a background colour (no doubt an accessibility issue)

Follow-up click test result ('after')

September 2011, 154 responses.


'After'



Reflection

Almost identical click patterns so the visual cue seems to be unnecessary here; plus, it would difficult to explain why sources appear in the highlighted panel or not without recourse to editorial guidelines (which themselves are often a matter of personal taste amongst users) and statistical calculation. It looks thankfully like this is less of an issue than we believed but we do need to trace this issue back to ensure we haven't missed something.

BHO Issue 7: 'Source pages are text heavy but little of that is "explanation"'

Page

Listings 

Heuristic

Learnability

Description

First time users, who might arrive through a search engine, have very little introductory text with which to 'ground' themselves. What explanation there is tells users where to find things, not what those things are. Returning users know what they are looking for and rarely use any descriptive signpost material. Screen real estate can be repurposed safely for first time users.

Impact severity

High

Recommend

Descriptive material suited to first time users to be created. Potential for a new verb-less strap line to go beneath the header.

Examples

The description for Primary sources confirms where you are, but doesn't explain what they are.

An issue of signposting



Quantitative measure

Click on the category where you would expect to find the Journal of the House of Commons

Actual question

Please click on where you'd expect to find the letters and papers of HENRY VIII

Initial click test result ('before')

July 2011, 200 responses.

'Before'



Development change

Descriptions of the five major source types were amended (Sources on British History Online).



Descriptions modified

Follow-up click test result ('after')

September 2011, 154 responses.




'After'

Reflection

Unfortunately, the third row contains the correct link and there's been a move away from it. In principle, this has been unsuccessful; however, the heterogeneity of sources on BHO defies generalisation. This list could easily be expanded to show the top 5 sources from each type to give a clearer idea of content without overwhelming the user.

It looks like we were too conservative on this issue - the follow-up click test has made that clear.

BHO Issue 6: 'Data structured geographically is not represented visually'

Page

Source

Heuristic

Memorability

Description

For a set of local history volumes, there is no visual way to assess coverage over what could be a substantial amount of data.

Impact severity

High

Recommend

If the source contains a map showing coverage, then provide that map at the source level with positioned links to the individual volumes or articles as appropriate.

Examples

The Survey of London volumes have been created over the course of 120 years and cover some of the most iconic locations in the capital. At the moment, the user would have to remember the descriptions of over 50 volumes to go straight to the one they want because the subtitles can be quite general.

Volumes which cover adjoining geographic areas are not published in sequence (e.g. vols 3 and 5)



Quantitative measure

Where would you expect to find an account of the church of St Giles in the Fields ?

Actual question

Please click on where you'd expect to find an account of the church of St Giles in the Fields

Initial click test result ('before')

July 2011, 200 responses.

'Before'



Development change

Insertion of new map facility with simple jQuery based zoom function at the head of the volume list (Survey of London on British History Online).



Follow-up click test result ('after')

September 2011, 154 responses.

'After'

Reflection

Difficult to make out any movement in clicks using the Survey of London. That may be because the question asked users to find an area for which an account existed - it may be that users who are researching areas not covered by the Survey are able to see more quickly that it won't be directly useful to their work.

It may be more use therefore in the structuring of a research project (i.e. the identification of relevant sources), rather than detailed research itself. Future usability studies should take account of this value.

BHO Issue 5: 'There are no shortcuts within a source other than to the volumes therein'

Page

Source

Heuristic

Learnability

Description

If a source is structured by theme, then the only way to find a specific theme is to go through each volume's table of contents looking for it. This is frustrating as the publication sequence of sources is often not the same sequence as its structure.

Impact severity

High

Recommend

Where a source has a structure which can be defined as a theme and where that structure is not followed by the publication sequence of the volumes, enable a list-based device which supports quicker navigation (i.e. one click).

Examples

The volumes from the Catalogue of Ancient Deeds list records from different departments in varying amounts, e.g. A.1 – A.1819, B.1 - B.1798.

The categories of deed are split inconsistently across publications



Quantitative measure

You have found the Catalogue of Ancient Deeds. Click on the volume which contains the deed C.3000

Actual question

Using the 'Catalogue of Ancient Deeds', please click on where you'd expect to find the deed C.3000

Initial click test result ('before')

July 2011, 200 responses.

'Before'

Development change

Insertion of new tagging structure at the head of the volume list; clicking a category filters the publication list accordingly (Catalogue of Ancient Deeds on British History Online).



Modified interface



Follow-up click test result ('after')

September 2011, 154 responses.

'After'

Reflection

Movement away from the search facility and onto the new tagging structure but perhaps signposting is not clear enough and again, the overall impression is that the control area of the page (i.e. search text box, tagging, sorting etc) is not meshed into one unit. The tagging section looks closer to the listings that the control section.

This process seems to be generating as many development points as it was supposed to solve!

BHO Issue 4: 'It is not clear which search filters are being employed'

Page

Search results

Heuristic

Learnability

Description

The textual part of the search query is repeated in the text box and result count line; but there is no description of what filters were applied (e.g. filter on the Calendar of State Papers).

Impact severity

Medium

Recommend

List each operating filter with an option to remove it, or add to it.

Examples

Search for Earl of Sussex using the filter Calendar of State Papers Domestic, James I

Not clear which filters are currently employed, and only option is to remove them all



Quantitative measure

You have searched for the Earl of Sussex and filtered for results only appearing in the Calendar of State Papers for James I. Click where you would expect to edit or remove that filter

Actual question

Please click on where you'd expect to EDIT the filter for 'Calendar of State Papers for James I'

Initial click test result ('before')

July 2011, 200 responses.

'Before'





Development change

Insertion of new 'breadbox' device between search text box and sorting options (search for Earl of Sussex on British History Online).

Modified interface





Follow-up click test result ('after')

September 2011, 154 responses.

'After'



Reflection

Clear grouping in the breadbox to remove the filter; though some users still drawn to the old switch in the statistical section, some to the search text box, and some further still drawn to the link back to the advanced search. Further segment of users seemingly thinking that the fourth search result, which explicitly has James I in its title, would contain the trigger to remove the filter.

Much more thought needs to go into the 'control' section at the head of the results page to make the purpose of the functions painstakingly clear. Typically, putting look up filters into free text boxes wouldn't be recommended due to misspelling, but some kind of auto completing drop down list might help here - research needed.

BHO Issue 3: 'Search result filtering can be time-consuming when working chronologically'

Page

Search results

Heuristic

Efficiency, memorability

Description

The search results sequence is controlled by a relevancy algorithm. However, with a range of resources, users find it intellectually easier to approach the filtering process by date. This leads to more navigation forward and back through the result set.

Impact severity

High

Recommend

That an option be made available on all search results to re-sort the result set chronologically. Any results without a specific date will come behind those that do. In addition, include a counter for each result as an aide memoire.

Examples

A search for lord sussex



Quantitative measure

Click where you would expect to be able to re-order these results by date

Actual question

Please click on where you'd expect to RE-ORDER these results chronologically by date

Initial click test result ('before')

July 2011, 200 responses.

'Before'





Development change

Added two toggle links: sort by date and sort by relevance beneath the search text field (search for lord sussex on British History Online).

Modified interface

Follow-up click test result ('after')

September 2011, 154 responses.

'After'

Reflection

Pretty clear agreement that the nomenclature and position of the toggle option is well understood. However, the statistical data of the query (result count, time taken) appears right-aligned and seems to have drawn a few people to it.

Looking at the small images of the page, it seems quite text heavy - maybe distinguishing the toggle option as a button and altering the format of the statistical data would create a clearer 'control' section to the search results which follow it, hopefully improving users' navigation.

BHO Issue 2: 'Several advanced search items are ignored'

Page

Search form

Heuristic

Efficiency

Description

Not a single user from our interviews even acknowledged the existence of the places, subjects and periods filters in the advanced search form. That was surprising as they all had to circumnavigate them to get to the source and publication filters which they did use.

Impact severity

Low

Recommend

Demote the unused options to allow faster operation

Examples

All users employed filters on searching –but not one of them was for place, subject or period.

Filtering options unused - try to improve signposting

Quantitative measure

You are searching for the place 'Bloomsbury' – click where you would expect to add a filter to only look in volumes from the series 'The Survey of London'.

Actual question

Please click on where you'd expect to add a filter to ONLY look in volumes from the series 'The Survey of London'

Initial click test result ('before')

July 2011, 200 responses.

'Before'

Development change

Appended extra descriptive text for filtering options (advanced search on British History Online).

Follow-up click test result ('after')

September 2011, 154 responses.

'After'

Reflection

Small but noticeable grouping under the new correct function (located in the second to last row) - however, still plenty of clicks for the first filtering row which was for all resources related to London rather than the specific source called 'The Survey of London'. I wonder if the wording of the question, which accentuated 'ONLY' led some users to click on the 'Only / Don't' drop down list at the head of each filter.

Given Issue 1, can't help feeling that the form is trying to do too much simultaneously. I need to research complex forms used elsewhere (e.g. popular e-commerce) and see if a new approach comes to mind.

BHO Issue 1: 'Support documents can clog up the search results filtering process'

Page

Search results

Heuristic

Efficiency

Description

Users often have to filter out certain types of document by eye, lengthening the amount of time it takes to complete the task

Impact severity

High

Recommend

Include option to exclude both indexes and prefatory material from results

Example

A search for longleat returns elements such as 'Notes on Abbreviations'; a search for richard ferris results in multiple 'index' components. In each case, the instances are mixed with narrative documents.

Supporting documents sometimes represent 'noise' for users

Quantitative measure

"You're searching for the person 'Richard Ferris' – click where you would expect to exclude certain types of document from your results, e.g. indexes, lists of abbreviations"

Actual question

Please click on where you'd expect to EXCLUDE certain types of document from your results, e.g. indexes, lists of abbreviations

Initial click test result ('before')

July 2011, 200 responses.

Development change

New row inserted under text fields, titled 'Include in results' with checkboxes for indexes and abbreviaton list ticked by default (advanced search on British History Online).

Follow-up click test result ('after')

September 2011, 154 responses.

Reflection

Clear movement from Before to After of clicks to the newly inserted function, less scatttered pattern to clicks overall. Still quite some work to do though; it appears that some users associate this function with the 'Filtering options' section underneath the main form.

Perhaps filtering options should be removed and its functionality built into the main form to remove this confusion. This work could simply be extended by incrementally changing functionality and stopping to check, provided we don't exhaust our users' patience with click tests!

Another idea would be to configure the form to have include and exclude boxes into which criteria could be placed - plenty of possibilities now that the mandate for change has been proven.

The issue list

As this rewarding project draws to a close, I thought I'd share the central plank along which it has run here at the Institute of Historical Research.

We used several qualitative tools to draw up an issue list, a document containing each problem which our analysis had thrown up in as clear a language as possible.

This document became the mandate for change here; it laid down our development objectives and gave us ideas for how to measure the effect of issues using qualitative analysis. Crucially, it attempts to steer clear of technical jargon so remaining transparent to end-users.

A document like this, handled in the wrong spirit, can become a political weapon; however, I feel that this risk is outweighed by the greater need to be honest with users about how a resource which they have come to rely on is developed.

I'll begin publishing all 9 of the issues we developed together with what we changed as a result, plus visuals of the before and after distructions of click tests and finally end up by offering an opinion (and no more) of whether we've been successful or not.