BHO Issue 4: 'It is not clear which search filters are being employed'

Page

Search results

Heuristic

Learnability

Description

The textual part of the search query is repeated in the text box and result count line; but there is no description of what filters were applied (e.g. filter on the Calendar of State Papers).

Impact severity

Medium

Recommend

List each operating filter with an option to remove it, or add to it.

Examples

Search for Earl of Sussex using the filter Calendar of State Papers Domestic, James I

Not clear which filters are currently employed, and only option is to remove them all



Quantitative measure

You have searched for the Earl of Sussex and filtered for results only appearing in the Calendar of State Papers for James I. Click where you would expect to edit or remove that filter

Actual question

Please click on where you'd expect to EDIT the filter for 'Calendar of State Papers for James I'

Initial click test result ('before')

July 2011, 200 responses.

'Before'





Development change

Insertion of new 'breadbox' device between search text box and sorting options (search for Earl of Sussex on British History Online).

Modified interface





Follow-up click test result ('after')

September 2011, 154 responses.

'After'



Reflection

Clear grouping in the breadbox to remove the filter; though some users still drawn to the old switch in the statistical section, some to the search text box, and some further still drawn to the link back to the advanced search. Further segment of users seemingly thinking that the fourth search result, which explicitly has James I in its title, would contain the trigger to remove the filter.

Much more thought needs to go into the 'control' section at the head of the results page to make the purpose of the functions painstakingly clear. Typically, putting look up filters into free text boxes wouldn't be recommended due to misspelling, but some kind of auto completing drop down list might help here - research needed.

BHO Issue 3: 'Search result filtering can be time-consuming when working chronologically'

Page

Search results

Heuristic

Efficiency, memorability

Description

The search results sequence is controlled by a relevancy algorithm. However, with a range of resources, users find it intellectually easier to approach the filtering process by date. This leads to more navigation forward and back through the result set.

Impact severity

High

Recommend

That an option be made available on all search results to re-sort the result set chronologically. Any results without a specific date will come behind those that do. In addition, include a counter for each result as an aide memoire.

Examples

A search for lord sussex



Quantitative measure

Click where you would expect to be able to re-order these results by date

Actual question

Please click on where you'd expect to RE-ORDER these results chronologically by date

Initial click test result ('before')

July 2011, 200 responses.

'Before'





Development change

Added two toggle links: sort by date and sort by relevance beneath the search text field (search for lord sussex on British History Online).

Modified interface

Follow-up click test result ('after')

September 2011, 154 responses.

'After'

Reflection

Pretty clear agreement that the nomenclature and position of the toggle option is well understood. However, the statistical data of the query (result count, time taken) appears right-aligned and seems to have drawn a few people to it.

Looking at the small images of the page, it seems quite text heavy - maybe distinguishing the toggle option as a button and altering the format of the statistical data would create a clearer 'control' section to the search results which follow it, hopefully improving users' navigation.

BHO Issue 2: 'Several advanced search items are ignored'

Page

Search form

Heuristic

Efficiency

Description

Not a single user from our interviews even acknowledged the existence of the places, subjects and periods filters in the advanced search form. That was surprising as they all had to circumnavigate them to get to the source and publication filters which they did use.

Impact severity

Low

Recommend

Demote the unused options to allow faster operation

Examples

All users employed filters on searching –but not one of them was for place, subject or period.

Filtering options unused - try to improve signposting

Quantitative measure

You are searching for the place 'Bloomsbury' – click where you would expect to add a filter to only look in volumes from the series 'The Survey of London'.

Actual question

Please click on where you'd expect to add a filter to ONLY look in volumes from the series 'The Survey of London'

Initial click test result ('before')

July 2011, 200 responses.

'Before'

Development change

Appended extra descriptive text for filtering options (advanced search on British History Online).

Follow-up click test result ('after')

September 2011, 154 responses.

'After'

Reflection

Small but noticeable grouping under the new correct function (located in the second to last row) - however, still plenty of clicks for the first filtering row which was for all resources related to London rather than the specific source called 'The Survey of London'. I wonder if the wording of the question, which accentuated 'ONLY' led some users to click on the 'Only / Don't' drop down list at the head of each filter.

Given Issue 1, can't help feeling that the form is trying to do too much simultaneously. I need to research complex forms used elsewhere (e.g. popular e-commerce) and see if a new approach comes to mind.

BHO Issue 1: 'Support documents can clog up the search results filtering process'

Page

Search results

Heuristic

Efficiency

Description

Users often have to filter out certain types of document by eye, lengthening the amount of time it takes to complete the task

Impact severity

High

Recommend

Include option to exclude both indexes and prefatory material from results

Example

A search for longleat returns elements such as 'Notes on Abbreviations'; a search for richard ferris results in multiple 'index' components. In each case, the instances are mixed with narrative documents.

Supporting documents sometimes represent 'noise' for users

Quantitative measure

"You're searching for the person 'Richard Ferris' – click where you would expect to exclude certain types of document from your results, e.g. indexes, lists of abbreviations"

Actual question

Please click on where you'd expect to EXCLUDE certain types of document from your results, e.g. indexes, lists of abbreviations

Initial click test result ('before')

July 2011, 200 responses.

Development change

New row inserted under text fields, titled 'Include in results' with checkboxes for indexes and abbreviaton list ticked by default (advanced search on British History Online).

Follow-up click test result ('after')

September 2011, 154 responses.

Reflection

Clear movement from Before to After of clicks to the newly inserted function, less scatttered pattern to clicks overall. Still quite some work to do though; it appears that some users associate this function with the 'Filtering options' section underneath the main form.

Perhaps filtering options should be removed and its functionality built into the main form to remove this confusion. This work could simply be extended by incrementally changing functionality and stopping to check, provided we don't exhaust our users' patience with click tests!

Another idea would be to configure the form to have include and exclude boxes into which criteria could be placed - plenty of possibilities now that the mandate for change has been proven.

The issue list

As this rewarding project draws to a close, I thought I'd share the central plank along which it has run here at the Institute of Historical Research.

We used several qualitative tools to draw up an issue list, a document containing each problem which our analysis had thrown up in as clear a language as possible.

This document became the mandate for change here; it laid down our development objectives and gave us ideas for how to measure the effect of issues using qualitative analysis. Crucially, it attempts to steer clear of technical jargon so remaining transparent to end-users.

A document like this, handled in the wrong spirit, can become a political weapon; however, I feel that this risk is outweighed by the greater need to be honest with users about how a resource which they have come to rely on is developed.

I'll begin publishing all 9 of the issues we developed together with what we changed as a result, plus visuals of the before and after distructions of click tests and finally end up by offering an opinion (and no more) of whether we've been successful or not.

Enhancements #1

The first set of enhancements designed to improve usability on British History Online specifically produced as part of the JISC 01/11B funding went live this morning. I'll round them up below but first I want to say a little about how these changes came about.

Our analysis was led by qualitative interactions with a set of academics from different locations around the country and different specialised subject areas using interviews and a focus group. Each interaction came up with a qualified wish list, the aggregation of which enabled us to look across the field and identify common bugbears and frustrations.

This led us to create an issue list – a plain English account of the problem including screen shot, a suggested recommendation and, crucially, a test question which embodied the issue. All nine test questions were then spliced into an online click test and put out through our public communications channels (site news, Facebook, Twitter etc) to the BHO audience.

This first consultation has enabled us to benchmark the current state of the site, and as you can see from the sample below, for some test questions, there has been a worryingly large distribution of clicks from users (the red circles indicate user clicks, and the arrow points to the correct click location).

I hope that the power of this approach is now becoming clear. After redevelopment, we are able to ask exactly the same questions of our users - a before and after situation - which will enable us to report the success or otherwise of each amendment. The key to enabling this is creating these easy to understand test questions.

So here are the enhancements which have just gone live, together with the issues to which they relate:

Issue: Support documents can clog up the search results filtering process
Enhancement: New checkboxes added to the advanced search form allowing users to exclude indices and abbreviation lists (see how to lose the noise from indices and abbreviations lists in search results)

Issue: Search result filtering can be time-consuming when working chronologically
Enhancement: Documents with clear event dates have been re-indexed, and a toggle option added to search results page enables users to switch between the two sequences (see how documents without specific event dates appear after those that do)

Issue: Issue: It is not clear which search filters are being employed
Enhancement: 'Breadbox' added to search results page enables users to examine their query and instantly remove certain parts (see how the construction of the query is clearly exposed)

Issue: Data structured geographically is not represented visually
Enhancement: Device to include a map at the head of the sources to which they refer, with simple zoom function (see how we brought an overview to the Survey of London)

Issue: Users have no way of re-ordering a list
Enhancement: Added 'A-Z' and 'Z-A' sort options to sources, most useful where the volumes are ordered chronologically, e.g. parliamentary/state papers (see the sequencing options in action for the Calendar of Close Rolls)

On Gideon and experimentation

In reaching the final stages of our usability project, I've started to think about the kind of change which this process forces upon the system manager/developer. It's not just about working your way through a list of software modifications; it's about identifying which ones are the most important and prioritising them.

Knowing that they are going to be re-tested by the same users as before binds the organisation to the user. If some of this testing is anonymous (for instance, through remote click tests), then the pressure to deliver a great service builds because users can feed back on the product as they find it, and there is no opportunity for political interference of these results.

But that pressure requires a specific ability on the part of the system developer: experimentation. Functions need to be tested over and over again before release, often using other team members from within the organisation. To function effectively, all the team members need to appreciate the value of this iterative but ultimately repetitive behaviour. For a certain period of time, they have to reject what is customary or an accepted standard for an information service and place themselves in a research context (much like how the usability standard persona tool works).

This isn't an easy piece of intellectual gymnastics to perform and some might say it's too exposed to the mundane organisational risks of political interference. However, the levelling factor will be the end user's response – no amount of influence will tell a humanities researcher what to think. And if that response is well documented and capable of being compared across time, as the usability process gives us the framework to do, then that voice will be heard.

So it looks as if running experiments such as surveys and click tests, standard tools within the social sciences, is finally here to stay for humanities products, as the results can easily communicate across all levels of an organisation how specific investments can provide a return. That's why the term usability applies as much to the methodology of this process (i.e. interview & focus group transcriptions/notes, survey results, click records) as it does to the system under investigation. Approaching the process in this way allows it to form not only the basis for modification of the system under review, but also for the very processes used to examine the system.

PS: you may be wondering why Gideon was in the title to this post. Gideon, as you may know, was unsure of whether he was actually speaking to God when he prayed; so he asked him to put water on his fleece but keep the ground dry. This was duly done, but being a sceptical fellow, the next day Gideon asked God to make the ground wet but keep the fleece dry. Now, instead, he could have simply asked someone if God existed and accepted their answer...but he didn't. Not only did he experiment, but crucially, he made that experiment easy to understand.