Information for designers

What is a cognitive walkthrough?

The cognitive walkthrough is a method for evaluating the learnability of a product or service. In this method, a small group of evaluators from a product team walk through a set of tasks and ask questions at each step in the task that get at whether the user would know what to do at that step and if the user does the right thing, will that user know if she has made progress toward the task goal. If the answer is “no” to the walkthrough questions, a usability defect is recorded.

source: https://doi.org/10 .1016/B978-0-12-410391-7.00004-X

When should the cognitive walkthough be used?

The cognitive walkthrough can be used during any phase of the development process but is most suited to the design stage where a functional specification and other design documents that provide enough information for the creation of action sequences (user inputs and system responses) for a set of tasks

Resources

Personnel

The cognitive walkthrough can be conducted by an individual or group. In a group evaluation, these are the important roles:

1. Facilitator. The facilitator is generally the organizer and maestro of the walkthrough process. The facilitator is responsible for making sure that the walkthrough team is prepared for the session and follows the rules of good meeting management.
2. Notetaker. The notetaker records the output of the cognitive walkthrough.
3. Product expert. Because the cognitive walkthrough can be conducted early in the design stage (e.g., after requirements and a functional specification), a product expert is desired to answer questions that members of the walkthrough team may have about the system’s features or feedback.
4. Evaluators. Evaluators are representatives from the product team. These representatives can be usability practitioners, requirements engineers, business analysts, developers, writers, and trainers.

Documents and material

1. User profile. Description of the primary users who perform the tasks that will be evaluated in a cognitive walkthrough.
2. Task list. The task list should describe the task in a realistic and concrete manner.
3. Action sequence for each task in the task list. includes the information such as date, analysts, users, interface, task, action sequence, comments etc...
4. Problem reporting form. This is the form for recording the problems that emerge from the walkthrough.
5. Representation of the UI. Flip charts, electronic display of outputs from cognitive walkthrough, prototypes, or working features.

Planning procedure of a cognitive walkthrough

1. Define the users of the product. Different personas might take different paths through a system. If you have multiple personas, you can walk through the system with a focus on the attributes of each persona that could have an impact on how easily they can learn how to perform tasks with the target system.
2. Determine what tasks and task variants (different ways to do the same task) are most appropriate for the walkthrough. The choice of tasks must balance complexity, realism, and the time allocated to the walkthrough. One strong recommendation is that the first task in the walkthrough be relatively simple so the team can learn the method before moving on to longer and more complex tasks.
3. Develop the ground rules for the walkthrough. Ours are available here.
4. Generate the action sequences for each task. An action sequence is a detailed description of the actions the target users must take to complete a task and the associated system responses at each step.
5. Provide a representation of the interface. This representation can be a detailed text scenario, an operating procedure, a set of sketches, a storyboard, a paper prototype, a partially working prototype, or a fully working product.
6. Assemble a group of evaluators for the cognitive walkthrough. Candidates include usability practitioners, writers, trainers, product managers, quality engineers, and developers.

Conducting a cognitive walkthrough

The cognitive walkthrough involves the following main steps:

1. Walk through the action sequences for each task from the perspective of the “typical” users of the product.
2. Record success stories, failure stories, design suggestions, problems that were not the direct output of the walkthrough, assumptions about users, comments about the tasks, and other information that may be useful in design.

The four-question approach

Wharton et al. (1994, p. 106) propose an approach based on four questions to tackle each of the items in the action sequences for each task (step 1 of the walkthrough protocol):

1. Will the user try to achieve the right action?
2. Will the user notice that the correct action is available?
3. Will the user associate the correct action with the effect that the user is trying to achieve?
4. If the correct action is performed, will the user see that progress is being made toward solution of the task?

After the Cognitive Walkthrough

After the walkthrough, follow these steps:

1. Bring the appropriate stakeholders together to review the results of the cognitive walkthrough.
2. Discuss potential solutions to the UI problems that were found in the walkthrough.
3. Determine which solutions you will apply to the product.
4. Evaluate the cognitive walkthrough process, and determine if there are improvements you could make for subsequent walkthroughs.

Informal user testing

The cognitive walkthrough will have uncovered some issues with the prototype, which must be addressed (possibly scraping the prototype and restarting). Based on the fixed prototype, you want to perform user testing. This procedure aims at evaluating both the low-fi prototype and the conceptual model, as well as validating the task examples.

When running the participants through the testing protocol, it is best to have at least two team members at each session: one for managing the session and the other for taking detailed notes. With the user consent, the user testing sessions are video recorded and transcribed in order to replay them with the entire team for more detailed analysis.