[The contents of this page have been updated and relocated as UX Walkthrough Preparation and Execution.
This version is deprecated.] – JPZ, November 5, 2007

Fluid UX Walkthrough Protocol


This document provides background information on performing a UX Walkthrough. Detailed directions for performing an evaluation on a particular system can be found on the appropriate UX Walkthrough Report Template.


We want to develop a kind of combination of heuristic evaluation and walkthrough. Heuristic evaluation is done by examining the interface to a system as whole, and hence isn't specific to any particular user task. Walkthroughs are done by tracing the user actions, and associated cues and feedback, for one or more particular tasks, and as such, don't attempt to cover the entire interface for a nontrivial system. On the other hand, a walkthrough can pick up issues in working through a task that can be hard to detect when examining the interface as a whole, relating especially to the adequacy of cues and feedback in context.

Our aim for Fluid, "Software that works - for everyone", takes in accessibility as well as usability. Rather than having two separate inspections, we want to have a unified inspection that addresses both areas, if we can.

Choosing what to examine

Within a product, we want to examine tasks and subtasks that use parts of the product that have some functional affinity, that is, parts that in some abstract sense "do the same kind of thing". We believe this will help us identify especially useful components, ones that can be used in more than one place in a product. For example, the Lightbox component can be used to do lots of different things. Abstractly, it's a component for "rearranging the layout of things".

By coordinating our choices across products, we can promote the same goal on a wider scale. By pooling our findings about how related functionality is used in different products, as well as in different places within the same product, we can design components that can be reused as widely as possible.

Since uPortal, Moodle and Sakai are all large and complex products in themselves, we will do the UI walkthroughs in stages, focusing on subsets of activities the products support in each phase.  For example, the first phase may focus on content management related activities.  By focusing the product evaluations on similar user activities, we will be able to understand user needs and the interactions that fit those needs across products.  The goal is that UI components can be designed and developed to fit the specifics of each product.   For example we may find that organizing content is cumbersome across the products.  A generic version of the "Lightbox" component, the "Organizer" could meet a broad set of needs.  As always, the context of an interaction shapes the specific implementation of a interaction so "Organize" could be the umbrella component with several specific implementations that meet the needs of different kinds of content and/or contexts.  Content management is being loosely defined as authoring, maintaining, organizing, viewing and sharing content - in this case digital content.  One can imagine the many cross-cutting interactions that are "componentizable" around managing content

Note that this is not a completely "top down" approach to our priorities, in that we are not starting directly from a determination of what tasks are most important for users (though we an presume that functions that crop up again and again in our products are important to users). After we are well launched in discovering components in this way, we may want to allocate effort to look at the user experience for the products more holistically, to assess what we may be missing in taking the functional view.


1. Preparation

Obtain working software, a prototype, for the system you intend to inspect. We will probably also want to do inspections from design documents from systems not yet implemented, but that case is not addressed (yet) in these notes.

  • For the walkthrough portion of the inspection, choose one or more representative tasks to inspect.
  • If you are also doing an accessibility inspection, btain an appropriate tool to use in inspecting the user experience for screen reader users. (Query: what tools have people used for this?)
    • JAWS is a widely used screen reader - a free time-limited demo is available.
    • Fangs is a Mozilla Firefox extension that displays a text representation of a web page in a manner intended to be similar to how a screen reader would read it.

2. Process

Plan to inspect the system from four viewpoints: typical users, screenreader users, keyboard-only users, and users who have difficulty processing text.

(Query: Should we also inspect for low vision users who are not screenreader users? Other cognitive disabilities not captured by difficulty with text?)

You can decide how you want to organize these inspections, for example by doing them sequentially or by examining a portion of the system from each point of view before moving on to another portion. Be sure to share your experience with these different approaches.

There are also different ways to organize the heuristic analysis and walkthrough aspects of the inspection. Here is one approach that combines some of the prep for the walkthrough with part of the heuristic analysis.

The walkthrough method requires that the analyst know what the right way to perform a task is before doing the walkthrough itself for that task. So, as the first part of your inspection, examine the interface with the dual aim of checking the points listed in the Heuristics Checklist below AND working out a good approach to your task or tasks. (You can decide whether you want to pick one of your tasks and complete all work on that before beginning another, or do the prep for all tasks before doing the walkthroughs on them.) Make notes of any issues you encounter in figuring out how to do the tasks, as well as any other problems you identify in this phase using the heuristics.

Once you have worked out how to perform one or more tasks for which you will do a walkthrough, do the walkthrough. This means working through an appropriate method for the task, step by step, checking the points in the Walkthrough section of the Heuristics checklist below. Keep carefully in mind that the walkthrough is NOT a user test, with you being the user. That's why you START the walkthrough already knowing a good way to perform the task. At each step, you are checking whether SOMEONE ELSE, without benefit of your prior knowledge, will be able to figure out what to do, do it, and respond appropriately to the result of the step.

Remember that you want to inspect not only for typical users, but also for screenreader users and others. This means that you need to consider the Walkthrough points for ALL of these users:  an interface may provide adequate cues for some step for typical users, but the cues may simply not be available to screen reader users, or may be hard to interpret and process for users for whom text processing is difficult. Similarly, required actions may be easy for mouse users, but difficult or impossible for people who do not use a mouse.

Once you have completed all of the walkthroughs you plan to do, keep in mind that Heuristic Evaluation includes surveying, as well as you can, ALL aspects of the interface, not just those that figure in particular tasks. Do what you can to explore the interface broadly, getting (and recording) a feel for any issues triggered by the Heuristics checklist.

Inspecting widgets. While our inspections are intended to provide useful feedback on the overall user experience provided by the systems we work on, Fluid has a particular focus on developing a usable and accessible widget set. This means we need to use special care in inspecting widgets. Because widgets are intended to be widely reused, special treatment for widgets in our inspections can also save us work: if someone inspection system A has already inspected a widget, someone inspecting some system B that uses the same widget doesn't need to repeat the detail work.

To realize this potential work savings, while at the same time making sure we provide careful inspections of all widgets, we should (a) make a list of all the widgets we encounter in an inspection, (b) check to see if an inspection for this widget is already available, (c) if it is, don't take the time for an in-depth inspection of the operation of the widget, and (d) if it is not, do a careful inspection, including accessibility aspects, and record the results in a shared list of widget inspections (Note: we need to create this shared list, which includes deciding how we are going to identify and describe them.)

An example of the application of this idea would be encountering a multiple selection widget while examining an interface overall, or doing a walkthrough for a task. If the shared list shows that the widget has already been inspected, we would pass over it, assuming that the user can make it work (or, more accurately, assuming that any problems have already been found and recorded). If it has not been inspected, we would do the inspection, carefully. For example, we would determine whether the multiple selection can be operated without the mouse; in fact this is not always possible with existing multiple selection widgets.

3. Recording results

While doing the inspection, you should note any issues you find, which heuristic checklist items are violated, and describe where in the system the problem occurs. We'll need to develop a system for giving these descriptions, and sharing the results.

Related Links


  1. This looks great!  Thanks for putting this together Clayton.  A couple thoughts on areas we'll want to explore further:

    - We may want to define "typical users".  uPortal has well fleshed out personas.  There are also some for Sakai although none for users with disablitities.  We may need to create new personas or perhaps we can leverage uPortals.  As we do the cognitive walkthrough, it will be much easier to "be the user" if we have personas to focus our attention and bring us back to their context, needs, goals, etc.

    - Tasks for the cognitive walkthroughs - this is an area we'll want to spend some time in thinking about the most common scenarios our users complete in the system.  We'll also want to include some more complicated tasks.  Depending on the size of the system, I like to have anywhere between 5 and 20 tasks to walkthrough the system with.  Scenarios we focus on should be realistic activities a particular persona would complete.  So we may have scenarios for both a faculty member and student for instance.  In addtion, each of those scenarios will need to be evaluated for accessibility for a screen reader user, someone who only uses the keyboard and a person that has difficulty processing text.

    - We will want to be fairly systematic in the way we go through the evaluations and document the findings.  Since we'll have several evaluators looking at each application, we can do some work up front identifying the report format to make  our synthesis easier.  I've seen spreadsheet like checklists where the heuristics are listed down the first column and each evaluator has a column for their comments.  I found this a bit to limiting.  The format I prefer describes the problem, the priniciple, priority and direction for solution.  Each individual evaluator goes off and documents the problems they find and related principle.  Then the group of evaluators come back together to collaboratively determine priority and solutions. 

  2. We should modify 2: Support consistent and correctly tagged navigation to be in line with ARIA to increase usability

    • Always provide a home page link
    • XHTML navigation landmarks from XHTML2 http://www.w3.org/TR/2006/WD-xhtml-role-20061113/#s_role_module_attributes (don't use skip to main content) as these roles are much more comprehensive
    • Consider a navigation section on each page (role="navigation")
    • Correctly tag headings
    • Make sure all navigation landmarks have a title including customer ARIA regions
    • Correctly tag table headers
    • mark tables which should not behave has tables as presentational (role=:"wairole:presentation")
    • Correctly tag table headers
    • Ensure Rich Internet Application widgets properly support ARIA and follow specified navigation rules following the tab to container and arrow within navigation rules
    • Define consistent keyboard mnemonic conventions for context menus and a limited set of UI widgets
    • Ensure keyboard navigation is properly coordinated with screen reader keyboard assignments
    • Properly define application and document areas in our web pages to assist the assistive technology with keyboard navigation. (A screen reader should not be in browse mode in a web application)

    Expand on Help and documentation:

    Help and documentation
        Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large. UI Componentry should provide rich hint, help, or tooltip text by establishing a  relationships between the UI widget (could be a form control) and the added help information. (ARIA provides a describedBy and Tooltip relationship for these situations)

    Ensure that all forms support extended ARIA form information such as: required and invalid

    Where possible synchronize ARIA states and Properties with CSS style properties.

    UI components supporting multiple selections or options should not have repeated option text.

  3. Hello--

    (I'm new to this project and just peeking in: I met Clayton at a conference last week and I'm very interested in this project! I work at Landmark College, in Putney, VT USA--a two-year college for students with learning disabilities. -Julie) 

    Regarding screen reader choices, it would be good to test with one designed for people who are blind (e.g. JAWS) and one for sighted people with cognitive difficulties (e.g. Wynn or Kurzweil 3000) and Mac OSX's VoiceOver. Users (especially for educational purposes) of Wynn/Kurzweil do a lot of text manipulation (highlighting, extracting, getting definitions, adding text notes), so tasks will be different than with JAWS.

    I do think it will be important to test for users with low vision who do not use screenreaders. This group includes people for whom decoding is difficult as well as users with macular degeneration.

    What about testing for digital divide types of accessibility, such as low bandwidth and low technical ability?

    Is there group understanding of what it means to have "difficulty processing text?" People with such difficulties often have difficulty with complex vocabulary, sentences embedded within sentences, and passive sentence construction. Other issues include difficulty remembering multiple-step instructions and literal interpretation of expressions. It's helpful to break out individual steps, and keep instructions visible (addressed by the 'recognition rather than recall' heuristic).

    Regarding Paddison and Englefield's accessibility heuristic #6: "Ensure contrast is sufficient in images," how about something broader, such as: "Ensure color contrast in hue and brightness are sufficient in all content, and can be adjusted by the user."

  4. To see what accessibility evaluators return in the way of results, try some out:

    WAVE 3.5: http://dev.wave.webaim.org/index.jsp

    WebXACT (Bobby): http://webxact.watchfire.com/

    aChecker (U Toronto): http://www.atutor.ca/achecker/demo.php

    Copy the url from the page you want to evaluate, then go to the checker(s), paste it in, select the evaluation criteria (WCAG 1.0 or 2.0 Priority One and Two), and hit the appropriate key.  That's it!

    The checker(s) will point out where the page fails to meet the guidelines. At this point, aChecker is the only one that evaluates against WCAG 2.0.

  5. I should check that status of AChecker's support for the latest WCAG 2.0 drafts now that its primary author, Chris Ridpath, has left the ATRC. It is still being support and will be moved forward by a new development team, but things may be a bit slow as they get accustomed to the code. Nonetheless it works great.

  6. Here's a 2005 HFI article that looks at several different heuristic checklists, including one actually based on research (imagine that!). A bit dated, but I think it's still useful. Here are the items proposed by Gerhardt-Powals (1996):

    1. Automate unwanted workload

    2. Reduce uncertainty

    3. Fuse datea

    4. Present new information with meaningful aids to interpretation

    5. Use names that are conceptually related to function

    6. Group data in consistently meaningful ways to decrease search time

    7. Limit data-drive tasks

    8. Include in the displays only that information needed by the user at a given time

    9. Provide multiple coding of data when appropriate

    10. Practice judicious redundancy 

    More detail is in the attachment. 

  7. I've attached the two evaluation forms used to review accessibility in Sakai: one that's kind of a cognitive walkthrough, and the other for reviews using adaptive technology, like JAWS.

  8. I've also found a research paper ("Analysis of Strategies for Improving and Estimating the Effectiveness of Heuristic  Evaluation" by Effie Lai-Chong Law and Ebba Thora Hvannberg) comparing Nielsen's approach with Gerhardt-Powals' to usability testing. Nielsen's approach found more issues that coincided with the testing, but, in a nutshell, neither heuristic approach was very effective (see abstract below). Not sure it changes our plans, but I thought you might find it interesting.

    Since posting on servers or redistributing to a list requires special permission, and/or a fee, members of the project team should contact me for a copy or get their own using this citation:

    ACM International Conference Proceeding Series; Vol. 82
    Proceedings of the third Nordic conference on Human-computer interaction

    Tampere, Finland
    Pages: 241 - 250  
    Year of Publication: 2004
    Effectiveness of Heuristic Evaluation (HE) has severely been challenged. The present study investigated two strategies for improving the effectiveness of HE: Selection of usability guidelines (Gerhardt-Powals' cognitive engineering principles vs. Nielsen's heuristics); Provision of support and training. We also examined various strategies for estimating the effectiveness of HE. An empirical study with a balanced within- and between subject 2x2 factorial design involving 18 novice evaluators was conducted. 290 <i>non-consolidated</i> usability problems (UPs) identified in 35 HE test sessions were evaluated against 88 <i>actual</i> UPs found in the user tests (n = 17) performed earlier. Nielsen's heuristics were proved to be more effective in enabling the evaluators to uncover significantly more UPs, though in general the effectiveness of HE was low. Implications for future work are drawn.