Lessons learned

Integrating usability into your work patterns means being able to demonstrate that you are in touch with the full range of users in a non-jargonistic way that excludes no-one. If you want to bootstrap usability, you will find that it has to influence and inform every single step of the development process and the relationship the project has with the wider organisational/business context in which it exists.

The biggest cultural shift is to make the usability analysis usable, i.e. to make the projects outputs extensible, transparent and traceable. The project will now be able to demonstrate an existing framework of usability into which any new development can be placed, discussed and tracked. Informal relationships with key users can now become formalised, events such as workshops can be opened up appealing to different audiences. Perhaps these workshops could also be taken out 'on the road' to visit institutions which development teams would not ordinarily get the chance to go to.

One potential dividend would be to build up experiences on the development of generic pieces of functionality, e.g. a registration form, a search form. New projects could then build these items of the back of hundreds (thousands?) of hours of research and development into these functions within the HE sector - an enviable knowledge base of good practice into which their own process could be recorded.

Usability practice has been integrated into our approach with no consultancy input - whilst there may be a need to create resources to define what constitutes the usability toolkit, this is liable to change over even a short period of time. Therefore, for usability to self-propagate, development teams across the HE sector need ways of communicating and discussing these approaches through worked examples. From a personal standpoint, there's no shortage of advice on the matter - what would be more useful if how other HE teams, facing the same institutional pressures, have actually run projects.

The funding for this project has generated a large amount of raw data on usage patterns which could inform other projects (e.g.  library catalogues) at their outset. The opportunity is there to for this to be a two-way process - to give results back to the community upon project closure and so build up a corpus of material which could potentially be used in a longitudinal study of systems support in HE.

The usability process seeks to make clear specific obstacles and impediments which your users face; it would be ironic if the approach taken to more closely integrate usability practice into HE systems development relied on generalised case studies and theory.

How successful have we been?

Our project plan lays down two kinds of success metric: quantitative and qualitative. However, a secondary aim is looking at identifying a method whereby these practices could be built into existing projects without much additional cost and enhance the quality of the next generation of software being built in UK HE. This is essential for niche areas of scholarship for which supporting software needs a high degree of innovation and no other product currently meets the needs of researchers.

Quantitative ('what people do')

Each identified issue have been separately reported; a digest appears below.

• Issue 1: met
• Issue 2: partially met
• Issue 3: met
• Issue 4: partially met
• Issue 5: partially met
• Issue 6: no change
• Issue 7: unmet
• Issue 8: no change
• Issue 9: met

Overall, this list most closely reflects my personal view of the success of this project - that in articulating user needs, it has created a mandate for change which extends far beyond what is possible within this project's timescale.

Qualitative ('what people say')

It was not possible to schedule re-interviews following the modifications being made live but here are the differences in the results from the System Usability Scale (SUS), a link to which appears on every page in BHO.


SUS before development

• Best imaginable: 40
• Excellent: 33
• Good: 28
• OK: 22
• Poor: 16
• Awful: 14
• Worst imaginable: 14

SUS after modification

• Excellent: 40
• Good: 28
• Poor: 21

More time is needed to build the level of response as the project had to report within 3 months (extended to 4) and perhaps we tried to cram too much in. Also, the SUS is not promoted actively (the click tests above were, for instance, publicised through the site news and blog); as a result, a lower level of response is to be expected and so a greater period of time is required to build up an impression of any change in the pattern of satisfaction.

Approach

We have successfully implemented a usability-centric approach, without consultant input, which covers the needs of a temporary project revolving around one set of software updates as well as providing the means for an ongoing inclusive dialogue across all functional departments (technical, editorial, managerial, marketing etc). There have been virtually no direct costs; the process has been devised, developed, implemented and reported on by existing staff, with the intention of making the results and as much of the raw data available for re-use by the HE community.

In addition, it is extensible; given the resources, each issue could be revisited, redeveloped and retested; new issues in other areas could be added to the issue list document. By publishing the information openly using a blog platform, the entire process is opened up for discussion and analysis.

It lends itself to networking/discussion and gives development teams the opportunity to discuss approaches to improving software beyond institutional technical environments.

Bootstrapping usability

As more and more content was added to British History Online (BHO), the listings pages and search results became longer and users were confronted by growing amounts of information which they would need to sift through to help them decide on which sources were going to be relevant to their research.

To prevent users becoming overwhelmed by the volume of information (thus impairing their ability to use the site for research), we undertook a usability project as part of the JISC 01/11B funding round. Our intention was to modify our way of working, building in usability practices, to find a self-sustaining way by which usability could be built into our working pattern and persist after the funding round had ended, i.e. bootstrapping.

Many project teams will be wondering how to employ usability without a budget  - and that's the model we followed. Our only direct cost, a rolling subscription with VerifyApp.com, was USD10 per month but enabled the crucial quantitative aspect to our investigation.

Our plan envisioned us researching qualitative and quantitative feedback, altering the BHO interface according to recommendations, and then re-testing both sets afterwards to give a before and after style report. It wasn't all possible (we weren't able to schedule interviews after the development) but most of it is in place and we reported the results through this blog.

Here is an outline of the work:

• interviews with historians
• focus group with the Survey of London at English Heritage
• benchmarking of System Usability Scale (SUS) before developments
• production of the issue list
• Click testing for each issue
• Development and deployment of new and modified functions on BHO
• Click testing for each issue using new modified interfaces
• Review of SUS results after modified interface went live
• Publishing the findings

At its heart, the project produced an issue list - a jargon-free document containing a clear description of the issues, how to reproduce them, screen shots and suggested questions to ask users to test any remedial development - it was the most important managerial output from the exercise because it could be understood by anybody thus providing the means for support for the changes to come from a range of different departments.

BHO Issue 9: 'Users have no way of re-ordering a list'

Page

Listings and source

Heuristic

Subjective

Description

Listings pages are sequenced by an editorial weighting: if a user remembers a source by time period, or by name, they have no way of re-ordering the list. Although source pages are arranged alphabetically, they may be structured by time period but authored in a non-linear sequence; alternatively, later volumes may be more comprehensive than earlier ones.

Impact severity

High

Recommend

Add in-page options to change the sort order. Configure some sources to be sequenced differently by default.

Examples

The Survey of London is being added to but newer volumes currently appear at the bottom of the listings, well below the fold.

New volumes appear right at the end (large descriptions also don't help scrolling)



Quantitative measure

[Use the Lords Journals series]: if you wanted to see the most recent volumes first, click where you expect to be able to re-order this list

Actual question

Please click on where you'd expect to re-order this list to see the most recent volumes FIRST

Initial click test result ('before')

July 2011, 200 responses.

'Before'



Development change

Added toggling pair of links, Sort A-Z and Sort Z-A, right-aligned beneath search text box (Survey of London on British History Online).

Modified interface



Follow-up click test result ('after')

September 2011, 154 responses.


'After'



Reflection

Clear grouping on correct sort function but still a few looking for this feature on the left hand side. In hindsight, that is where the search sort functions were located (see earlier Issue 3) and it compounds the recurring theme that the site needs a clearly designed and consistent layout for a control panel, to be used wherever appropriate.

BHO Issue 8: 'It is not clear what the top sources are'

Page

Listings 

Heuristic

Learnability

Description

A page listing sources for a specific taxonomy dimension will list the most relevant sources at the top. However, they are visually indistinct from sources which contain a lower quantity of relevant content.

Impact severity

Medium

Recommend

Prominence to be given to sources which occupy the top n positions, possibly through a gallery device, after which sources have clearly lower relevance.

Examples

For Scotland, the list of sources moves from the Topographical Dictionary of Scotland to local records from Cumberland without any presentational cue. The underlying weighting figure drops by nearly 90% at that point.

Green indicates the most relevant for Scotland



Quantitative measure

Click on ALL of the records on this page which you feel are directly relevant to Scottish affairs

Actual question

Please click on the LOWEST record on this page which you feel is directly relevant to Scottish affairs

Initial click test result ('before')

July 2011, 200 responses.

'Before'



Development change

Descriptions of the five major source types were amended (Sources on British History Online).



Using a background colour (no doubt an accessibility issue)

Follow-up click test result ('after')

September 2011, 154 responses.


'After'



Reflection

Almost identical click patterns so the visual cue seems to be unnecessary here; plus, it would difficult to explain why sources appear in the highlighted panel or not without recourse to editorial guidelines (which themselves are often a matter of personal taste amongst users) and statistical calculation. It looks thankfully like this is less of an issue than we believed but we do need to trace this issue back to ensure we haven't missed something.

BHO Issue 7: 'Source pages are text heavy but little of that is "explanation"'

Page

Listings 

Heuristic

Learnability

Description

First time users, who might arrive through a search engine, have very little introductory text with which to 'ground' themselves. What explanation there is tells users where to find things, not what those things are. Returning users know what they are looking for and rarely use any descriptive signpost material. Screen real estate can be repurposed safely for first time users.

Impact severity

High

Recommend

Descriptive material suited to first time users to be created. Potential for a new verb-less strap line to go beneath the header.

Examples

The description for Primary sources confirms where you are, but doesn't explain what they are.

An issue of signposting



Quantitative measure

Click on the category where you would expect to find the Journal of the House of Commons

Actual question

Please click on where you'd expect to find the letters and papers of HENRY VIII

Initial click test result ('before')

July 2011, 200 responses.

'Before'



Development change

Descriptions of the five major source types were amended (Sources on British History Online).



Descriptions modified

Follow-up click test result ('after')

September 2011, 154 responses.




'After'

Reflection

Unfortunately, the third row contains the correct link and there's been a move away from it. In principle, this has been unsuccessful; however, the heterogeneity of sources on BHO defies generalisation. This list could easily be expanded to show the top 5 sources from each type to give a clearer idea of content without overwhelming the user.

It looks like we were too conservative on this issue - the follow-up click test has made that clear.

BHO Issue 6: 'Data structured geographically is not represented visually'

Page

Source

Heuristic

Memorability

Description

For a set of local history volumes, there is no visual way to assess coverage over what could be a substantial amount of data.

Impact severity

High

Recommend

If the source contains a map showing coverage, then provide that map at the source level with positioned links to the individual volumes or articles as appropriate.

Examples

The Survey of London volumes have been created over the course of 120 years and cover some of the most iconic locations in the capital. At the moment, the user would have to remember the descriptions of over 50 volumes to go straight to the one they want because the subtitles can be quite general.

Volumes which cover adjoining geographic areas are not published in sequence (e.g. vols 3 and 5)



Quantitative measure

Where would you expect to find an account of the church of St Giles in the Fields ?

Actual question

Please click on where you'd expect to find an account of the church of St Giles in the Fields

Initial click test result ('before')

July 2011, 200 responses.

'Before'



Development change

Insertion of new map facility with simple jQuery based zoom function at the head of the volume list (Survey of London on British History Online).



Follow-up click test result ('after')

September 2011, 154 responses.

'After'

Reflection

Difficult to make out any movement in clicks using the Survey of London. That may be because the question asked users to find an area for which an account existed - it may be that users who are researching areas not covered by the Survey are able to see more quickly that it won't be directly useful to their work.

It may be more use therefore in the structuring of a research project (i.e. the identification of relevant sources), rather than detailed research itself. Future usability studies should take account of this value.

BHO Issue 5: 'There are no shortcuts within a source other than to the volumes therein'

Page

Source

Heuristic

Learnability

Description

If a source is structured by theme, then the only way to find a specific theme is to go through each volume's table of contents looking for it. This is frustrating as the publication sequence of sources is often not the same sequence as its structure.

Impact severity

High

Recommend

Where a source has a structure which can be defined as a theme and where that structure is not followed by the publication sequence of the volumes, enable a list-based device which supports quicker navigation (i.e. one click).

Examples

The volumes from the Catalogue of Ancient Deeds list records from different departments in varying amounts, e.g. A.1 – A.1819, B.1 - B.1798.

The categories of deed are split inconsistently across publications



Quantitative measure

You have found the Catalogue of Ancient Deeds. Click on the volume which contains the deed C.3000

Actual question

Using the 'Catalogue of Ancient Deeds', please click on where you'd expect to find the deed C.3000

Initial click test result ('before')

July 2011, 200 responses.

'Before'

Development change

Insertion of new tagging structure at the head of the volume list; clicking a category filters the publication list accordingly (Catalogue of Ancient Deeds on British History Online).



Modified interface



Follow-up click test result ('after')

September 2011, 154 responses.

'After'

Reflection

Movement away from the search facility and onto the new tagging structure but perhaps signposting is not clear enough and again, the overall impression is that the control area of the page (i.e. search text box, tagging, sorting etc) is not meshed into one unit. The tagging section looks closer to the listings that the control section.

This process seems to be generating as many development points as it was supposed to solve!

BHO Issue 4: 'It is not clear which search filters are being employed'

Page

Search results

Heuristic

Learnability

Description

The textual part of the search query is repeated in the text box and result count line; but there is no description of what filters were applied (e.g. filter on the Calendar of State Papers).

Impact severity

Medium

Recommend

List each operating filter with an option to remove it, or add to it.

Examples

Search for Earl of Sussex using the filter Calendar of State Papers Domestic, James I

Not clear which filters are currently employed, and only option is to remove them all



Quantitative measure

You have searched for the Earl of Sussex and filtered for results only appearing in the Calendar of State Papers for James I. Click where you would expect to edit or remove that filter

Actual question

Please click on where you'd expect to EDIT the filter for 'Calendar of State Papers for James I'

Initial click test result ('before')

July 2011, 200 responses.

'Before'





Development change

Insertion of new 'breadbox' device between search text box and sorting options (search for Earl of Sussex on British History Online).

Modified interface





Follow-up click test result ('after')

September 2011, 154 responses.

'After'



Reflection

Clear grouping in the breadbox to remove the filter; though some users still drawn to the old switch in the statistical section, some to the search text box, and some further still drawn to the link back to the advanced search. Further segment of users seemingly thinking that the fourth search result, which explicitly has James I in its title, would contain the trigger to remove the filter.

Much more thought needs to go into the 'control' section at the head of the results page to make the purpose of the functions painstakingly clear. Typically, putting look up filters into free text boxes wouldn't be recommended due to misspelling, but some kind of auto completing drop down list might help here - research needed.

BHO Issue 3: 'Search result filtering can be time-consuming when working chronologically'

Page

Search results

Heuristic

Efficiency, memorability

Description

The search results sequence is controlled by a relevancy algorithm. However, with a range of resources, users find it intellectually easier to approach the filtering process by date. This leads to more navigation forward and back through the result set.

Impact severity

High

Recommend

That an option be made available on all search results to re-sort the result set chronologically. Any results without a specific date will come behind those that do. In addition, include a counter for each result as an aide memoire.

Examples

A search for lord sussex



Quantitative measure

Click where you would expect to be able to re-order these results by date

Actual question

Please click on where you'd expect to RE-ORDER these results chronologically by date

Initial click test result ('before')

July 2011, 200 responses.

'Before'





Development change

Added two toggle links: sort by date and sort by relevance beneath the search text field (search for lord sussex on British History Online).

Modified interface

Follow-up click test result ('after')

September 2011, 154 responses.

'After'

Reflection

Pretty clear agreement that the nomenclature and position of the toggle option is well understood. However, the statistical data of the query (result count, time taken) appears right-aligned and seems to have drawn a few people to it.

Looking at the small images of the page, it seems quite text heavy - maybe distinguishing the toggle option as a button and altering the format of the statistical data would create a clearer 'control' section to the search results which follow it, hopefully improving users' navigation.

BHO Issue 2: 'Several advanced search items are ignored'

Page

Search form

Heuristic

Efficiency

Description

Not a single user from our interviews even acknowledged the existence of the places, subjects and periods filters in the advanced search form. That was surprising as they all had to circumnavigate them to get to the source and publication filters which they did use.

Impact severity

Low

Recommend

Demote the unused options to allow faster operation

Examples

All users employed filters on searching –but not one of them was for place, subject or period.

Filtering options unused - try to improve signposting

Quantitative measure

You are searching for the place 'Bloomsbury' – click where you would expect to add a filter to only look in volumes from the series 'The Survey of London'.

Actual question

Please click on where you'd expect to add a filter to ONLY look in volumes from the series 'The Survey of London'

Initial click test result ('before')

July 2011, 200 responses.

'Before'

Development change

Appended extra descriptive text for filtering options (advanced search on British History Online).

Follow-up click test result ('after')

September 2011, 154 responses.

'After'

Reflection

Small but noticeable grouping under the new correct function (located in the second to last row) - however, still plenty of clicks for the first filtering row which was for all resources related to London rather than the specific source called 'The Survey of London'. I wonder if the wording of the question, which accentuated 'ONLY' led some users to click on the 'Only / Don't' drop down list at the head of each filter.

Given Issue 1, can't help feeling that the form is trying to do too much simultaneously. I need to research complex forms used elsewhere (e.g. popular e-commerce) and see if a new approach comes to mind.

BHO Issue 1: 'Support documents can clog up the search results filtering process'

Page

Search results

Heuristic

Efficiency

Description

Users often have to filter out certain types of document by eye, lengthening the amount of time it takes to complete the task

Impact severity

High

Recommend

Include option to exclude both indexes and prefatory material from results

Example

A search for longleat returns elements such as 'Notes on Abbreviations'; a search for richard ferris results in multiple 'index' components. In each case, the instances are mixed with narrative documents.

Supporting documents sometimes represent 'noise' for users

Quantitative measure

"You're searching for the person 'Richard Ferris' – click where you would expect to exclude certain types of document from your results, e.g. indexes, lists of abbreviations"

Actual question

Please click on where you'd expect to EXCLUDE certain types of document from your results, e.g. indexes, lists of abbreviations

Initial click test result ('before')

July 2011, 200 responses.

Development change

New row inserted under text fields, titled 'Include in results' with checkboxes for indexes and abbreviaton list ticked by default (advanced search on British History Online).

Follow-up click test result ('after')

September 2011, 154 responses.

Reflection

Clear movement from Before to After of clicks to the newly inserted function, less scatttered pattern to clicks overall. Still quite some work to do though; it appears that some users associate this function with the 'Filtering options' section underneath the main form.

Perhaps filtering options should be removed and its functionality built into the main form to remove this confusion. This work could simply be extended by incrementally changing functionality and stopping to check, provided we don't exhaust our users' patience with click tests!

Another idea would be to configure the form to have include and exclude boxes into which criteria could be placed - plenty of possibilities now that the mandate for change has been proven.

The issue list

As this rewarding project draws to a close, I thought I'd share the central plank along which it has run here at the Institute of Historical Research.

We used several qualitative tools to draw up an issue list, a document containing each problem which our analysis had thrown up in as clear a language as possible.

This document became the mandate for change here; it laid down our development objectives and gave us ideas for how to measure the effect of issues using qualitative analysis. Crucially, it attempts to steer clear of technical jargon so remaining transparent to end-users.

A document like this, handled in the wrong spirit, can become a political weapon; however, I feel that this risk is outweighed by the greater need to be honest with users about how a resource which they have come to rely on is developed.

I'll begin publishing all 9 of the issues we developed together with what we changed as a result, plus visuals of the before and after distructions of click tests and finally end up by offering an opinion (and no more) of whether we've been successful or not.

Enhancements #1

The first set of enhancements designed to improve usability on British History Online specifically produced as part of the JISC 01/11B funding went live this morning. I'll round them up below but first I want to say a little about how these changes came about.

Our analysis was led by qualitative interactions with a set of academics from different locations around the country and different specialised subject areas using interviews and a focus group. Each interaction came up with a qualified wish list, the aggregation of which enabled us to look across the field and identify common bugbears and frustrations.

This led us to create an issue list – a plain English account of the problem including screen shot, a suggested recommendation and, crucially, a test question which embodied the issue. All nine test questions were then spliced into an online click test and put out through our public communications channels (site news, Facebook, Twitter etc) to the BHO audience.

This first consultation has enabled us to benchmark the current state of the site, and as you can see from the sample below, for some test questions, there has been a worryingly large distribution of clicks from users (the red circles indicate user clicks, and the arrow points to the correct click location).

I hope that the power of this approach is now becoming clear. After redevelopment, we are able to ask exactly the same questions of our users - a before and after situation - which will enable us to report the success or otherwise of each amendment. The key to enabling this is creating these easy to understand test questions.

So here are the enhancements which have just gone live, together with the issues to which they relate:

Issue: Support documents can clog up the search results filtering process
Enhancement: New checkboxes added to the advanced search form allowing users to exclude indices and abbreviation lists (see how to lose the noise from indices and abbreviations lists in search results)

Issue: Search result filtering can be time-consuming when working chronologically
Enhancement: Documents with clear event dates have been re-indexed, and a toggle option added to search results page enables users to switch between the two sequences (see how documents without specific event dates appear after those that do)

Issue: Issue: It is not clear which search filters are being employed
Enhancement: 'Breadbox' added to search results page enables users to examine their query and instantly remove certain parts (see how the construction of the query is clearly exposed)

Issue: Data structured geographically is not represented visually
Enhancement: Device to include a map at the head of the sources to which they refer, with simple zoom function (see how we brought an overview to the Survey of London)

Issue: Users have no way of re-ordering a list
Enhancement: Added 'A-Z' and 'Z-A' sort options to sources, most useful where the volumes are ordered chronologically, e.g. parliamentary/state papers (see the sequencing options in action for the Calendar of Close Rolls)

On Gideon and experimentation

In reaching the final stages of our usability project, I've started to think about the kind of change which this process forces upon the system manager/developer. It's not just about working your way through a list of software modifications; it's about identifying which ones are the most important and prioritising them.

Knowing that they are going to be re-tested by the same users as before binds the organisation to the user. If some of this testing is anonymous (for instance, through remote click tests), then the pressure to deliver a great service builds because users can feed back on the product as they find it, and there is no opportunity for political interference of these results.

But that pressure requires a specific ability on the part of the system developer: experimentation. Functions need to be tested over and over again before release, often using other team members from within the organisation. To function effectively, all the team members need to appreciate the value of this iterative but ultimately repetitive behaviour. For a certain period of time, they have to reject what is customary or an accepted standard for an information service and place themselves in a research context (much like how the usability standard persona tool works).

This isn't an easy piece of intellectual gymnastics to perform and some might say it's too exposed to the mundane organisational risks of political interference. However, the levelling factor will be the end user's response – no amount of influence will tell a humanities researcher what to think. And if that response is well documented and capable of being compared across time, as the usability process gives us the framework to do, then that voice will be heard.

So it looks as if running experiments such as surveys and click tests, standard tools within the social sciences, is finally here to stay for humanities products, as the results can easily communicate across all levels of an organisation how specific investments can provide a return. That's why the term usability applies as much to the methodology of this process (i.e. interview & focus group transcriptions/notes, survey results, click records) as it does to the system under investigation. Approaching the process in this way allows it to form not only the basis for modification of the system under review, but also for the very processes used to examine the system.

PS: you may be wondering why Gideon was in the title to this post. Gideon, as you may know, was unsure of whether he was actually speaking to God when he prayed; so he asked him to put water on his fleece but keep the ground dry. This was duly done, but being a sceptical fellow, the next day Gideon asked God to make the ground wet but keep the fleece dry. Now, instead, he could have simply asked someone if God existed and accepted their answer...but he didn't. Not only did he experiment, but crucially, he made that experiment easy to understand.

Budget

Project team

Bruce Tate has been Project Manager at the Institute of Historical Research working on British History Online since 2002 and has overseen the site since its launch in June 2003 through to the present day. He is accountable for all aspects of planning, budget, timeframe and quality, and is PRINCE2 (Practitioner) certified. He also developed the front end of the JISC-funded Connected Histories website. Previously, he worked for the Audit Commission and commercial publishers including Macmillan and Wilmington.

Bruce is responsible for project management, the initial analysis, the development work required to incorporate user feedback, and evaluation.

Jonathan Blaney joined the Institute of Historical Research in 2007 as Project Editor for an AHRC-funded project to complete the digitisation of the National Archives’ Calendars of State Papers via British History Online (BHO). He now continues to work for part of the time on BHO, as well as contributing to a range of other IHR Digital projects. Jonathan has worked as lexicographer for Oxford University Press and as an editor on the Oxford Dictionary of National Biography. He subsequently worked for the Oxford Digital Library, where he was a Text Encoding Reviewer on the Early English Books Online Text Creation Partnership and Eighteenth Century Collections Online Text Creation Patnership, and also advised on a number of digitisation projects.

Jonathan will be involved with reviewing the results of the analysis phase of the project. Like all members of the team he will also have some responsibility for dissemination of results.

Peter Webster is Editorial Controller of British History Online and Manager of SAS-Space, the digital repository for the School of Advanced Study. He also contributes to a range of other IHR Digital projects, and has particular responsibility for the analysis and benchmarking of user behaviour. Before joining the IHR, Peter completed his PhD at the University of Sheffield, and was Technical Assistant to the AHRB Russian Visual Arts Project in the Sheffield Humanities Research Institute. He also taught in the University's Department of History.

Peter will also be involved with reviewing the results of the analysis phase of the project and will also contribute to dissemination activities.

Colin Thom is a Senior Historian with the Survey of London, the closest thing to an 'official' history of the capital, now part of English Heritage's Department of Heritage Protection. He has contributed to the Survey's ongoing series of detailed studies of the capital's architecture and topography for more than 20 years. He is also the author of Researching London's Houses (Historical Publications, 2005), an archives users’ guide to the subject, and lectures and publishes regularly on all aspects of the capital's history. He acted as Project Officer for English Heritage (EH) during the joint EH / IHR project to make the Survey’s volumes freely available in digital form via BHO.

Colin will identify and help convene the user focus group with colleagues at English Heritage, and help with the evaluation of the prototype revisions to the service.

Risk analysis

Action to prevent / manage risk

1. Survey response rate may disappoint

BHO has run two multi-page surveys in the last six months both of which pooled roughly 1,000 responses. Some of the consultation proposed will be much shorter in length than those so response rates can be expected to remain high. In addition the IHR, which is centrally funded to facilitate research at the national level, is ideally placed to communicate with the academic history community in the UK. It hosts the History Lab, the national postgraduate organisation for historians, and hosts numerous seminars, workshops and conferences which are attended by history teachers and researchers across the UK.

2. We may not prioritise the right needs for the academic sector
Each piece of user consultation will be identifiable by the audience categories as previously laid out by the System Usability Scale.

3. Difficulties recruiting and retaining staff
The proposed staff are already in post, and are enthusiastic about the project. In addition there is a large pool of digital project staff on whom to draw within the Institute, should anyone leave or be otherwise unavailable.

Timeline and work packages

Timeline

Work package 1: project management
Ensure the timely delivery of all work packages to the required standard, and inform stakeholders about the project.

Outputs

• Project plan


• Project blog


• Input for the support project


• Dissemination of outputs across IHR communication channels


• Final budget and completion report

Work package 2: analysis

Understand the major issues for users through face-to-face techniques, convert those issues wherever possible into remote tests, and record them quantitatively.

Outputs

• Individual user interviews and task observation


• Focus group with the Survey of London, English Heritage


• System usability scale (SUS)


• Remote testing using VerifyApp


• Case study report of usability issues identified

Work package 3: innovation

Take advice and generate ideas for the reconfiguration of services to improve the user experience.

Outputs

• Build prototype solutions to usability issues, including:
  
  
  
  • Listings
  
  
  • Source
  
  
  • Search form
  
  
  • Search results

Work package 4: evaluation

Undertake follow-on research to uncover satisfaction with proposed revisions to services.

Outputs

• (Shorter) individual user interviews and task observation


• Focus group with selected user from Survey of London


• Continued monitoring of SUS


• Repeat remote testing using VerifyApp


• Report drawing together issues and feedback for remedial solutions

Aims and objectives: BHO usability case study

Goal
Identify mitigating strategies for the problem of information overload across three generic types of site facility: category listings, product listings, and search (form and results). This will enable the project to show an improved return on investment to all of its funding sources by leading to greater and more informed use of the site as a whole, strengthening its case for sustainability.

The choice of goal maximises the impact of the project by increasing return on investment for BHO (essential for its own funding arrangements), as well as giving insight to other resource owners across the field as the functions are generic and implemented widely across the field (part of the Institute of Historical Research's own broader remit to encourage innovation in research). Our primary outcome for this project is the noticeable improvement of click-through ratios for each function; and secondly, to produce recommendations for how the identification of issues could be built into the ongoing managerial process behind British History Online (i.e. adopting lessons learned).

Success measures
Produce evidence of improved quantitative ratings and qualitative feedback on revised designs in each of the areas under review. Secondly, reflect on the specific conditions under which the tools and techniques used generate the most value. Success will be measured by evaluating the difference in successful click rates, and also looking at qualitative measures such as annotation tests and the System Usability Scale (SUS).

All the usability components outlined in the Approach section are used to baseline performance during the initial analysis phase. After prototyping, remote testing will be used, but will include both quantitative and qualitative strands with the intention of comparing the two sets of results.

The measures are clear enough to be understood by different roles within the organisation, i.e. they can be used to justify change with business managers as much as indicate development areas to the information architect or developer. Doing the research within the project means the ambition of the changes proposed is realistically linked to the amount of resources which the project has its disposal, leading to recommendations that are practicable to implement.

The project could have included looking at a set of websites rather than just one; however, where user outcomes cannot be compared, it becomes impossible to judge where resources should be assigned to the maximum effect (are two medieval historians better than one early modern?).

The project could also have focussed wholly on canvassing either qualitative or quantitative feedback and extended the depth of consultation. However, that would be to assume that what people say and what people do is materially equivalent, which would not necessarily be true.

Approach
The following techniques will be used throughout the project: Interviews, remote testing (e.g. click, annotation, labelling of system designs), user groups, and the SUS).

	What people say	What people do
Initial analysis	Individual Interviews User group SUS	Click tests
Post-prototyping	Annotation tests SUS	Click tests A/B tests

The initial analysis phase will result in a number of identified usability issues which will be presented as report cards. The report card device is easily understood and lends itself not just for use in the same way in other projects, but as a starting point for discussion into usability issues. This may be critical to the widespread recognition of usability as a core component of academic information service provision.