Enhancements #1

The first set of enhancements designed to improve usability on British History Online specifically produced as part of the JISC 01/11B funding went live this morning. I'll round them up below but first I want to say a little about how these changes came about.

Our analysis was led by qualitative interactions with a set of academics from different locations around the country and different specialised subject areas using interviews and a focus group. Each interaction came up with a qualified wish list, the aggregation of which enabled us to look across the field and identify common bugbears and frustrations.

This led us to create an issue list – a plain English account of the problem including screen shot, a suggested recommendation and, crucially, a test question which embodied the issue. All nine test questions were then spliced into an online click test and put out through our public communications channels (site news, Facebook, Twitter etc) to the BHO audience.

This first consultation has enabled us to benchmark the current state of the site, and as you can see from the sample below, for some test questions, there has been a worryingly large distribution of clicks from users (the red circles indicate user clicks, and the arrow points to the correct click location).

I hope that the power of this approach is now becoming clear. After redevelopment, we are able to ask exactly the same questions of our users - a before and after situation - which will enable us to report the success or otherwise of each amendment. The key to enabling this is creating these easy to understand test questions.

So here are the enhancements which have just gone live, together with the issues to which they relate:

Issue: Support documents can clog up the search results filtering process
Enhancement: New checkboxes added to the advanced search form allowing users to exclude indices and abbreviation lists (see how to lose the noise from indices and abbreviations lists in search results)

Issue: Search result filtering can be time-consuming when working chronologically
Enhancement: Documents with clear event dates have been re-indexed, and a toggle option added to search results page enables users to switch between the two sequences (see how documents without specific event dates appear after those that do)

Issue: Issue: It is not clear which search filters are being employed
Enhancement: 'Breadbox' added to search results page enables users to examine their query and instantly remove certain parts (see how the construction of the query is clearly exposed)

Issue: Data structured geographically is not represented visually
Enhancement: Device to include a map at the head of the sources to which they refer, with simple zoom function (see how we brought an overview to the Survey of London)

Issue: Users have no way of re-ordering a list
Enhancement: Added 'A-Z' and 'Z-A' sort options to sources, most useful where the volumes are ordered chronologically, e.g. parliamentary/state papers (see the sequencing options in action for the Calendar of Close Rolls)

On Gideon and experimentation

In reaching the final stages of our usability project, I've started to think about the kind of change which this process forces upon the system manager/developer. It's not just about working your way through a list of software modifications; it's about identifying which ones are the most important and prioritising them.

Knowing that they are going to be re-tested by the same users as before binds the organisation to the user. If some of this testing is anonymous (for instance, through remote click tests), then the pressure to deliver a great service builds because users can feed back on the product as they find it, and there is no opportunity for political interference of these results.

But that pressure requires a specific ability on the part of the system developer: experimentation. Functions need to be tested over and over again before release, often using other team members from within the organisation. To function effectively, all the team members need to appreciate the value of this iterative but ultimately repetitive behaviour. For a certain period of time, they have to reject what is customary or an accepted standard for an information service and place themselves in a research context (much like how the usability standard persona tool works).

This isn't an easy piece of intellectual gymnastics to perform and some might say it's too exposed to the mundane organisational risks of political interference. However, the levelling factor will be the end user's response – no amount of influence will tell a humanities researcher what to think. And if that response is well documented and capable of being compared across time, as the usability process gives us the framework to do, then that voice will be heard.

So it looks as if running experiments such as surveys and click tests, standard tools within the social sciences, is finally here to stay for humanities products, as the results can easily communicate across all levels of an organisation how specific investments can provide a return. That's why the term usability applies as much to the methodology of this process (i.e. interview & focus group transcriptions/notes, survey results, click records) as it does to the system under investigation. Approaching the process in this way allows it to form not only the basis for modification of the system under review, but also for the very processes used to examine the system.

PS: you may be wondering why Gideon was in the title to this post. Gideon, as you may know, was unsure of whether he was actually speaking to God when he prayed; so he asked him to put water on his fleece but keep the ground dry. This was duly done, but being a sceptical fellow, the next day Gideon asked God to make the ground wet but keep the fleece dry. Now, instead, he could have simply asked someone if God existed and accepted their answer...but he didn't. Not only did he experiment, but crucially, he made that experiment easy to understand.

Budget

Project team

Bruce Tate has been Project Manager at the Institute of Historical Research working on British History Online since 2002 and has overseen the site since its launch in June 2003 through to the present day. He is accountable for all aspects of planning, budget, timeframe and quality, and is PRINCE2 (Practitioner) certified. He also developed the front end of the JISC-funded Connected Histories website. Previously, he worked for the Audit Commission and commercial publishers including Macmillan and Wilmington.

Bruce is responsible for project management, the initial analysis, the development work required to incorporate user feedback, and evaluation.

Jonathan Blaney joined the Institute of Historical Research in 2007 as Project Editor for an AHRC-funded project to complete the digitisation of the National Archives’ Calendars of State Papers via British History Online (BHO). He now continues to work for part of the time on BHO, as well as contributing to a range of other IHR Digital projects. Jonathan has worked as lexicographer for Oxford University Press and as an editor on the Oxford Dictionary of National Biography. He subsequently worked for the Oxford Digital Library, where he was a Text Encoding Reviewer on the Early English Books Online Text Creation Partnership and Eighteenth Century Collections Online Text Creation Patnership, and also advised on a number of digitisation projects.

Jonathan will be involved with reviewing the results of the analysis phase of the project. Like all members of the team he will also have some responsibility for dissemination of results.

Peter Webster is Editorial Controller of British History Online and Manager of SAS-Space, the digital repository for the School of Advanced Study. He also contributes to a range of other IHR Digital projects, and has particular responsibility for the analysis and benchmarking of user behaviour. Before joining the IHR, Peter completed his PhD at the University of Sheffield, and was Technical Assistant to the AHRB Russian Visual Arts Project in the Sheffield Humanities Research Institute. He also taught in the University's Department of History.

Peter will also be involved with reviewing the results of the analysis phase of the project and will also contribute to dissemination activities.

Colin Thom is a Senior Historian with the Survey of London, the closest thing to an 'official' history of the capital, now part of English Heritage's Department of Heritage Protection. He has contributed to the Survey's ongoing series of detailed studies of the capital's architecture and topography for more than 20 years. He is also the author of Researching London's Houses (Historical Publications, 2005), an archives users’ guide to the subject, and lectures and publishes regularly on all aspects of the capital's history. He acted as Project Officer for English Heritage (EH) during the joint EH / IHR project to make the Survey’s volumes freely available in digital form via BHO.

Colin will identify and help convene the user focus group with colleagues at English Heritage, and help with the evaluation of the prototype revisions to the service.

Risk analysis

Action to prevent / manage risk

1. Survey response rate may disappoint

BHO has run two multi-page surveys in the last six months both of which pooled roughly 1,000 responses. Some of the consultation proposed will be much shorter in length than those so response rates can be expected to remain high. In addition the IHR, which is centrally funded to facilitate research at the national level, is ideally placed to communicate with the academic history community in the UK. It hosts the History Lab, the national postgraduate organisation for historians, and hosts numerous seminars, workshops and conferences which are attended by history teachers and researchers across the UK.

2. We may not prioritise the right needs for the academic sector
Each piece of user consultation will be identifiable by the audience categories as previously laid out by the System Usability Scale.

3. Difficulties recruiting and retaining staff
The proposed staff are already in post, and are enthusiastic about the project. In addition there is a large pool of digital project staff on whom to draw within the Institute, should anyone leave or be otherwise unavailable.

Timeline and work packages

Timeline

Work package 1: project management
Ensure the timely delivery of all work packages to the required standard, and inform stakeholders about the project.

Outputs

• Project plan


• Project blog


• Input for the support project


• Dissemination of outputs across IHR communication channels


• Final budget and completion report

Work package 2: analysis

Understand the major issues for users through face-to-face techniques, convert those issues wherever possible into remote tests, and record them quantitatively.

Outputs

• Individual user interviews and task observation


• Focus group with the Survey of London, English Heritage


• System usability scale (SUS)


• Remote testing using VerifyApp


• Case study report of usability issues identified

Work package 3: innovation

Take advice and generate ideas for the reconfiguration of services to improve the user experience.

Outputs

• Build prototype solutions to usability issues, including:
  
  
  
  • Listings
  
  
  • Source
  
  
  • Search form
  
  
  • Search results

Work package 4: evaluation

Undertake follow-on research to uncover satisfaction with proposed revisions to services.

Outputs

• (Shorter) individual user interviews and task observation


• Focus group with selected user from Survey of London


• Continued monitoring of SUS


• Repeat remote testing using VerifyApp


• Report drawing together issues and feedback for remedial solutions

Aims and objectives: BHO usability case study

Goal
Identify mitigating strategies for the problem of information overload across three generic types of site facility: category listings, product listings, and search (form and results). This will enable the project to show an improved return on investment to all of its funding sources by leading to greater and more informed use of the site as a whole, strengthening its case for sustainability.

The choice of goal maximises the impact of the project by increasing return on investment for BHO (essential for its own funding arrangements), as well as giving insight to other resource owners across the field as the functions are generic and implemented widely across the field (part of the Institute of Historical Research's own broader remit to encourage innovation in research). Our primary outcome for this project is the noticeable improvement of click-through ratios for each function; and secondly, to produce recommendations for how the identification of issues could be built into the ongoing managerial process behind British History Online (i.e. adopting lessons learned).

Success measures
Produce evidence of improved quantitative ratings and qualitative feedback on revised designs in each of the areas under review. Secondly, reflect on the specific conditions under which the tools and techniques used generate the most value. Success will be measured by evaluating the difference in successful click rates, and also looking at qualitative measures such as annotation tests and the System Usability Scale (SUS).

All the usability components outlined in the Approach section are used to baseline performance during the initial analysis phase. After prototyping, remote testing will be used, but will include both quantitative and qualitative strands with the intention of comparing the two sets of results.

The measures are clear enough to be understood by different roles within the organisation, i.e. they can be used to justify change with business managers as much as indicate development areas to the information architect or developer. Doing the research within the project means the ambition of the changes proposed is realistically linked to the amount of resources which the project has its disposal, leading to recommendations that are practicable to implement.

The project could have included looking at a set of websites rather than just one; however, where user outcomes cannot be compared, it becomes impossible to judge where resources should be assigned to the maximum effect (are two medieval historians better than one early modern?).

The project could also have focussed wholly on canvassing either qualitative or quantitative feedback and extended the depth of consultation. However, that would be to assume that what people say and what people do is materially equivalent, which would not necessarily be true.

Approach
The following techniques will be used throughout the project: Interviews, remote testing (e.g. click, annotation, labelling of system designs), user groups, and the SUS).

	What people say	What people do
Initial analysis	Individual Interviews User group SUS	Click tests
Post-prototyping	Annotation tests SUS	Click tests A/B tests

The initial analysis phase will result in a number of identified usability issues which will be presented as report cards. The report card device is easily understood and lends itself not just for use in the same way in other projects, but as a starting point for discussion into usability issues. This may be critical to the widespread recognition of usability as a core component of academic information service provision.