Chapter 14: Evaluation Studies: From Controlled to Natural Settings
Chapter Introduction
| Web Resources
| Assignment Comments
| Teaching Materials
The websites in this section present examples of usability testing and field studies. Some of them also contain information about how to perform the methods used in usability testing and field studies. For example, there are templates for designing questionnaires. These websites are also relevant to chapters 7 and 8.
http://usableweb.com/ One of the first links on Keith Instone's usable web site is to a page of links about user testing http://usableweb.com/topics/000878-0-0.html that provides practical advice to help novices. For example, it discusses the basics of usability testing, how to adapt basic techniques for different situations, whether testing needs to be done in a laboratory and many other important issues. There are also links to articles and to guidelines and there are links to logging tools and articles about logging analysis.
http://www.asktog.com/columns/042ButterflyBallot.html Take this link on Bruce Tognazzini's website to read about the usability fiasco of the Butterfly Ballot in Florida in the 2000 US Presidential elections and the design of more recent voting systems.
http://oldwww.acm.org/perlman/question.html This site provides useful information about questionnaire design. In addition to descriptions and references the site also contains templates for you to try out or to build your own questionnaires. The results can be mailed either to yourself or someone else. The questionnaire templates provide questions with Likert or Semantic scales. There are also fields for open-ended text comments. Clicking on a small icon next to each Likert scale question causes a open-ended comment area are to be produced. http://www.lap.umd.edu/quis These questionnaires, developed at the University of Maryland Human-Computer Interaction Laboratory, have been used for many years. They are tried and tested but unfortunately there is a licensing fee for use, although this fee is reduced for students. http://www.fas.harvard.edu/~stats/survey-soft/survey-soft.html This site provides many useful links to survey software analysis tools. Though now rather old this site - http://hfrg.ucc.ie/resources/qfaq1.html - offers a list of important questions about questionnaire design in usability testing and usability engineering. For example, Dr. Jurek Kirakowski asks: what is a questionnaire; how do you analyze open-ended questionnaires; how can I tell if a question should use a Likert scale or not; can you tell if a respondent is lying; and many other interesting and important questions. The site was last updated in 2000 but much of the information that it contains is still relevant.
Two well-known sites that offer online questionnaire templates are Survey Monkey (Surveymonkey.com) which can be used free for small population surveys but a license must be purchased for for use on large surveys. A Youtube video provides training to help you get started with Survey Monkey - https://www.youtube.com/user/SurveyMonkey
The other survey package that is often used is produced by Qualtrics - http://www.qualtrics.com. Many universities and corporations hold licenses so that their community can use Qualtrics. The Qualtrics site contains a
http://www.pewinternet.org The Pew Internet and American Life survey reports regularly on a wide array of topics featured on the Internet; these include health, use by different demographic groups, e-government, education, many topical issues and more. As well as reading about the interesting content it is useful to examine large-scale survey design.
http://culturematters.wordpress.com is a blog about issues in anthropology. http://culturematters.wordpress.com/2007/11/27/ethnography-in-human-computer-interaction/ discusses ethnography in HCI. Though old now, this blog raises some interesting issues. http://transground.blogspot.com/2006/06/ethnography-in-hci-comments-on-dourish.html This blog also discusses Paul Dourish’s 2006 CHI paper on ethnography in HCI.
http://www.the2the.com/adi/publications_presentations/supporting_rapid_ethnography_hci_field_research_pair_writing.pdf This paper suggests a technique for doing rapid ethnography in HCI – it is good material for a debate on the pros and cons of the technique.