“Defining Goals and Concerns” and “Deciding Who Should Be Participants” – Joseph Dumas & Janice Redish

Before you begin testing it is important to set your primary evaluation goals and concerns. Once those have been defined, it will be possible to plan a usability test accordingly. Dumas & Redish raise the following processes: making choices among goals and concerns; moving from general concerns to specific ones; and understanding sources of goals and concerns.

Making Choices Among Goals and Concerns
Depending on the stage at which a product has been designed and prepared for testing, goals and concerns will differ. In any case, they should be defined in order to uncover the appropriate problems.

Moving From General Concerns to Specific Ones
The authors suggest both looking at general and specific concerns, those that help frame a type of user and those that may suggest specific tasks for evaluation.

Understanding Sources of Goals and Concerns
– Concerns from task analysis and quantitative usability goals: Quantitative performance data may, for instance, refer to how much time participants will spend on a given task.
– Concerns from a heuristic analysis or expert review: Local problems raised by heuristic analyses or expert reviews help guide some of the questions you may want answered during an evaluation.
– Concerns from previous tests: In the context of iterative testing, questions may concern the problems revealed in previous tests.

Selecting who are the most appropriate participants for testing can be challenging. At this point you should seek representative users. The authors suggest: developing user profiles, selecting subgroups for a test, defining and quantifying characteristics for each subgroup, and finally deciding how many people to include in a test.

Developing user profiles

User profiling is assessed collectively within the team, and involves marketers, engineers, and designers. Profiles will differ depending on what type of product is to be tested, and whether it is a novel or a upgraded version of an existing product.
– Relevant characteristics are found in two categories: those that are found to be common to all users and those that may be specific to some users. – Grouping the target audience into different subgroups helps you decide on which factors matter most. Those subgroups may concern background information such as: users’ specializations, their general and specific computer skills (if any), their knowledge of the product at test, and their experience with similar products. – It is also important when testing the usability of a product to think broadly about users and to seek non-representative users as well. These type of participants will give an insight as to how efficiently this product might correspond to non-expert users.

Selecting subgroups for a test
Dumas & Redish define subgroups as “people who share specific characteristics that are important in the user profile.” (p.123) They propose beginning with a pretest questionnaire or survey for defining those different categories or subgroup users. A detailed understanding of user characteristics can be found through collecting quantitative responses. This helps dividing users into subgroups with limited and measurable requirements which include novice, experienced, and intermediate users.

Deciding how many people to include in a test
Finally, it is important to limit the number of subgroups and the amount of participants within each subgroup before you begin testing. As suggested in Nielsen and Molich 1990’s study, between 4 to 10 participants would suffice to uncover 80-90% of usability problems. Dumas & Redish propose to test 6 to 12 users to make up 2 to 3 subgroups of at least 3 users per subgroup to avoid collecting idiosyncratic data. While testing you will find that similar problems within subgroups as well as across subgroups will emerge, which help determine a number of global problems to consider when redesigning.