Chapter 13: An Evaluation Framework
Chapter Introduction
| Web Resources
| Assignment Comments
| Teaching Materials
For this activity we recommend that you look for a substantial paper that is 4-8 pages long and not a short 1-2 page paper. The ACM Digital Library is a good source for finding papers. Many universities pay for membership so that students and faculty have access to these papers, which come from several sources including the annual SIGCHI Conference - CHI - Computer Human-Interaction. Papers from TOCHI - Transactions on Computer Human-Interaction, and several other journals and conferences are also available. These longer papers are more likely to provide more information about how the evaluation studies were done which will enable you to perform a much fuller DECIDE analysis. Having said this, you should not be surprised if you do not find answers to all of the questions listed in the activity. Read the paper carefully, and answer as many of the questions as you can with the information given. For those questions that you cannot answer directly using the information in the paper, make your own suggestions about what would be appropriate within the context of the evaluation study but make sure that you signal that these are your suggestions.
Commenting upon the reliability, validity, ecological validity, biases, and scope of the study are likely to be more difficult than questions concerned with identifying the evaluation goals and methods used, because these aspects are often not reported and commented upon in papers, though they are important aspects of a good evaluation study. Similarly, some papers do not report on their pilot studies. In this case, you should suggest what you would do for a pilot study if you were running the evaluation study.
The ACM Digital Library has papers that cover all the evaluation approaches and methods discussed in this book. You should not be surprised if two or even three approaches and several methods are used to answer different questions or to gain different perspectives on the same question. It is quite usual to use different approaches at different stages of design. For example, field studies may be used when evaluating early design ideas, followed by usability testing when there is a working prototype and or analytical evaluation such as heuristic evaluation, and possibly more field studies to gain a deeper understanding of how a product is used in a natural setting.