Chapter 13: An Evaluation Framework
Chapter Introduction
| Web Resources
| Assignment Comments
| Teaching Materials
Objectives
The main aims of this chapter are to:
- Introduce and explain the DECIDE framework.
- Discuss the conceptual, practical, and ethical issues involved in evaluation.
Introduction
As products evolve from initial ideas through conceptual design and prototypes, iterative cycles of design and evaluation help to ensure that they meet users' needs. However, deciding when and how to evaluate a product requires careful consideration and may be different for different kinds of products. This can be challenging given the rapid development of a wide variety of mobile and distributed systems.
The design process starts with the designers working to develop a product that meets users' requirements, but understanding requirements tends to happen by a process of negotiation between designers and users over a period of time. As designers understand users' needs better, their designs reflect this understanding. Similarly, as users see and experience design ideas, they are able to give better feedback that enables the designers to improve their designs. The process is cyclical, with evaluation facilitating understanding between designers and users.
Evaluation is driven by questions about how well the design or particular aspects of it satisfy requirements and offer appropriate user experiences. Some of these questions provide high-level goals to guide the evaluation. For example, does this product excite users so that they will buy and use it? Others are much more specific. Can users find a particular menu item? Do they interpret a particular graphic as the designers intended and do they fi nd it attractive? Practical constraints play a big role in shaping how evaluation is done: tight schedules, low budgets, or limited access to users constrain what evaluators can do and may prompt them to seek new ways of doing evaluations, e.g. using crowdsourcing as discussed in Chapter 12. There are ethical considerations too: medical records are confi dential, certain areas of people's homes are private, and so is some of the information that users put on their Facebook and other social networking sites.
There is a broad repertoire of evaluation methods that can be tailored for specifi c circumstances, and experienced designers get to know what works and what doesn't. When planning evaluations, evaluators must consider the nature of each product, the target user population, where the product will be used and the contexts of use, as well as logistical issues. Planning evaluation studies involves asking questions about the process and anticipating potential problems. Within interaction design there are many books and websites that list different techniques and guidelines for conducting an evaluation, but there is very little overarching guidance for how to plan an evaluation. To help you, we propose the DECIDE framework, which provides a structure for planning evaluation studies.