Chapter 14: Introducing Evaluation
Chapter Introduction | Web Resources | In-Depth Activity Comments | Teaching Materials
Objectives
The main goals of this chapter are to accomplish the following:
- Explain the key concepts and terms used in evaluation.
- Introduce a range of different types of evaluation methods.
- Show how different evaluation methods are used for different purposes at different stages of the design process and in different contexts of use.
- Show how evaluation methods are mixed and modified to meet the demands of evaluating novel systems.
- Discuss some of the practical challenges of doing evaluation.
- Illustrate through short case studies how methods discussed in more depth in Chapters 8, 9, and 10 are used in evaluation and describe some methods that are specific to evaluation.
- Provide an overview of methods that are discussed in detail in the next two chapters.
Introduction
Imagine that you designed an app for teenagers to share music, gossip, and photos. You prototyped your first design and implemented the core functionality. How would you find out whether it would appeal to them and whether they will use it? You would need to evaluate it—but how? This chapter presents an introduction to the main types of evaluation and the methods that you can use to evaluate design prototypes and design concepts.
Evaluation is integral to the design process. It involves collecting and analyzing data about users’ or potential users’ experiences when interacting with a design artifact such as a screen sketch, prototype, app, computer system, or component of a computer system. A central goal of evaluation is to improve the artifact’s design. Evaluation focuses on both the usability of the system (that is, how easy it is to learn and to use) and on the users’ experiences when interacting with it (for example, how satisfying, enjoyable, or motivating the interaction is).
Devices such as smartphones, iPads, and e-readers, together with the pervasiveness of mobile apps and the emergence of IoT devices, have heightened awareness about usability and interaction design. However, many designers still assume that if they and their colleagues can use a product and find it attractive, others will too. The problem with this assumption is that designers may then design only for themselves. Evaluation enables them to check that their design is appropriate and acceptable for the target user population.
There are many different evaluation methods. Which to use depends on the goals of the evaluation. Evaluations can occur in a range of places such as in labs, people’s homes, outdoors, and work settings. Evaluations usually involve observing participants and measuring their performance during usability testing, experiments, or field studies in order to evaluate the design or design concept. There are other methods, however, that do not involve participants directly, such as modeling users’ behavior and analytics. Modeling users’ behavior provides an approximation of what users might do when interacting with an interface; these models are often done as a quick way of assessing the potential of different interface configurations. Analytics provide a way of examining the performance of an already existing product, such as a website, so that it can be improved. The level of control on what is evaluated varies; sometimes there is none, such as for studies in the wild, and in others there is considerable control over which tasks are performed and the context, such as in experiments.
In this chapter, we discuss why evaluation is important, what needs to be evaluated, where evaluation should take place, and when in the product lifecycle evaluation is needed. Some examples of different types of evaluation studies are then illustrated by short case studies.