Chapter 14: Evaluation Studies: From Controlled to Natural Settings
Chapter Introduction
| Web Resources
| Assignment Comments
| Teaching Materials
This assignment provides useful experience in using these approaches. As you do the usability testing remember that you need to control the testing environment so that each participant has a similar experience. A preliminary step in doing this is to work on developing a typical task, or two tasks that you expect anyone using the system to need to do. An obvious example is to book a ticket. Some tasks can only be achieved by following a set order of steps, others can be achieved in a variety of different ways. Which applies to your task. Then write the task down being careful to make sure that you provide a clear description of the task. In larger evaluation studies you would be wise to carry out a pilot test with participants who are not going to participate in the main study, so that you can test out the task, your description of it and the test set-up and environment – for example, if you were going to video record the person and log their keystrokes, you need to make sure that the equipment that you will use to capture this data is working properly. Another thing to think about is whether you need to give the participants some activities to get them used to using the system before you conduct the usability test. In addition to working out a task you need to decide who your participants will be. Who is a “typical user” of this system. Sometimes there are clearly a set of people with similar expertise, experience and demographic make up who will use the system – for example an accounting system that is designed for trained accountants. Often, as in the case of this system, the potential user group is quite broad and variable, so how will you select typical users? Is there such a thing as a typical user?
Almost always you will need to develop an informed consent form. You will also need to pay attention to how you introduce each participant to the task and how you control the environment in which they will be working. Many of us do our usability testing in lab-like conditions that we create by using a small room; few of us have access to a fully-fledged usability laboratory.
The activity suggests that you should observe each participant, noting any errors that they make, and the paths that they take, etc., and, if possible, timing them. This is going to be more difficult than you may expect, so try out the method you will use to do this before you work with one of your participants. Having collected your data think about how you will analyze it. There will be too few participants to do much analysis but think about how you will present your findings and what you would do if you were performing a larger study
When thinking about the field study that you would do, focus on the differences between usability testing and field studies, including “in the wild studies”. What could you learn from a field study that you don’t know from usability testing.