How to Plan, Design, and Conduct Effective Tests – Jeffrey Rubin & Dana Chisnell

“What Is Usability Testing?”

Rubin & Chisnell define ‘usability testing’ as “a process that employs people as testing participants who are representative of the target audience to evaluate the degree to which a product meets specific usability criteria.” (p.21) For them, the overall methodology is the same, but each test will have different results depending on the stage at which a product is tested and the type of goals and concerns a development team has for testing. They propose 4 types of informal tests that can be employed iteratively and that correspond to the different stages of the product at test and respond to issues of: exploration, assessment, validation, and comparison.

Why Test? Goals of Testing

Testing has two main objectives: on the marketing level, it aims at improving sales; on a user-centered level, it aims at minimizing user frustrations and maximizing a product’s usability. As the authors point out, testing goals inform the design of a product; those work in terms of the usefulness or relevance, learnability, efficiency, effectiveness, and satisfaction factors.

Basics of the Methodology

The authors emphasize the need for ‘tight controls’ when screening participants. This means making sure you recruit participants with similar backgrounds, such as: novice computer users. Assuming that you’ve picked the right participants for each test, testing will help you collect quantitative data that determine usability problems, causes and potential solutions.

Each test should involve: a purpose (goals and concerns), a participant (or two), an environment (i.e. a lab or an office room), observation method (a camera and observer who take notes), a methodology (interview or survey questions, and key tasks), data collection (qualitative and quantitative performance measures), results-debriefing (determining solutions for further implementations of the product).

Rubin & Chisnell make it a point that testing is not the ultimate answer, but is always better than not testing at all. In this sense, testing should be thought of as a practical tool for eliminating major problems and finding appropriate solutions for them. However, “Testing does not guarantee success or even prove that a product will be usable.” (p.26) The environment is always somewhat artificial and never the ideal place or circumstance of eventual use. Also, results of a test will depend greatly on the manner in which a test is conducted. This is why choosing an experienced moderator is crucial and making sure you have the right participants will also affect the validity of your data. Testing participants is one way of revealing usability problems, but different products call for different evaluation methods, some empirical and some non-empirical.

“When Should You Test?”

Based on the stage of development of your product, the authors suggest 4 corresponding types of tests, each focused on revealing different types of data in terms of your specific goals and concerns: exploratory (early stage), assessment (early and middle stages), validation (near-final stage), and comparison (all stages). When used as a complement to an iterative design process, those tests help ‘ “shape” the product to fit the end users’ abilities, expectations, and aptitude.’ (p.28)

The Exploratory or Formative Study is used at the early stage of the development process of your product. Here, testing will allow a high-level evaluation (horizontal functions) of the core concepts and ideas by revealing qualitative data. This stage also helps you reconsider your assumptions of your end users and understand them better in terms of their abilities to comprehend the visual language of an interface or product, their expectations of what a product does and how it functions, and their aptitude to learn and perform.

The Assessment or Summative Test is used early on and/or midway in the development process of your product. This test is a low-level evaluation (vertical functions) of the key tasks involved in using the product. How successful are the participants at completing at task? is typically your testing purpose and results in a quantitative performance data collection.

The Validation or Verification Test is used late in the process. The objective of this study is to define a appropriate ‘usability standard’ for future products and verify the efficiency and effectiveness of your product based on your two previous test results. Here, the test looks at various quantitative measurements such as: How well do users perform? How much time does it take them to complete a task? How consistent is the overall architecture of your design?

Finally, the Comparison Test usually comes in two forms: evaluating similar conceptual designs with slight variations or evaluating disparate alternatives of the same content. This method can be employed at any stage of the development process: to compare and explore different design layouts and visual (graphical or textual) languages used, and to determine the stance of your product against the competition.