Chapter 14: Evaluation Studies: From Controlled to Natural Settings
Chapter Introduction
| Web Resources
| Assignment Comments
| Teaching Materials
The websites in this section present examples of usability testing and field studies. Some of them also contain information about how to perform the methods used in usability testing and field studies. For example, there are templates for designing questionnaires. These websites are also relevant to chapters 7 and 8.
http://usableweb.com/ One of the first links on Keith Instone’s usable web site is to a page of links about user testing http://usableweb.com/topics/000878-0-0.html that provides practical advice to help novices. For example, it discusses the basics of usability testing, how to adapt basic techniques for different situations, whether testing needs to be done in a laboratory and many other important issues. There are also links to articles and to guidelines and there are links to logging tools and articles about logging analysis.
http://www.consult-me.co.uk/csc-case-studies.htm is a site that contains brief accounts of case studies that show how designers evaluated usability during the design process.
http://www.asktog.com/columns/042ButterflyBallot.html Take this link on Bruce Tognazzini’s website to read about the usability fiasco of the Butterfly Ballot in Florida in the 2000 US Presidential elections.
http://www.otal.umd.edu/hci-rm/index.html This site was developed by students at the University of Maryland and it offers a classification and descriptions of some key techniques that are used in field studies. It also provides a useful overview of different types of data logging and links to a variety of logging tools.
http://www.acm.org/%7Eperlman/question.html This site provides useful information about questionnaire design. In addition to descriptions and references the site also contains templates for you to try out or to build your own questionnaires. The results can be mailed either to yourself or someone else. The questionnaire templates provide questions with Likert or Semantic scales. There are also fields for open-ended text comments. Clicking on a small icon next to each Likert scale question causes a open-ended comment area are to be produced. http://www.lap.umd.edu/quis These questionnaires, developed at the University of Maryland Human-Computer Interaction Laboratory, have been used for many years. They are tried and tested but unfortunately there is a licensing fee for use, although this fee is reduced for students.http://www.fas.harvard.edu/~stats/survey-soft/survey-soft.html This site provides many useful links to survey software analysis tools.
http://hfrg.ucc.ie/resources/qfaq1.html This site offers a list of important questions about questionnaire design in usability testing and usability engineering. For example, Dr. Jurek Kirakowski asks: what is a questionnaire; how do you analyze open-ended questionnaires; how can I tell if a question should use a Likert scale or not; can you tell if a respondent is lying; and many other interesting and important questions. The site was last updated in 2000 but much of the information that it contains is still relevant.
http://www.gvu.gatech.edu/user_surveys/ This site contains examples of national surveys that you may wish to examine. They contain results of annual surveys of Internet usage. One problem with these surveys is that is they use convenience sampling which is not scientific. By sending the survey to as many Internet users as possible these researchers are trying to obtain as random a sample as possible.The last survey results posted were for 1998.
http://www.gvu.gatech.edu/user_surveys/others/ Provides useful links to other sources of information about web surveys.
http://www.pewinternet.org The Pew Internet and American Life survey reports regularly on a wide array of topics featured on the Internet; these include health, use by different demographic groups, e-government, education and more. As well as reading about the interesting content it is useful to examine the survey design.
http://culturematters.wordpress.com is a blog about issues in anthropology. http://culturematters.wordpress.com/2007/11/27/ethnography-in-human-computer-interaction/ discusses ethnography in HCI. Though old now, this blog raises some interesting issues. http://transground.blogspot.com/2006/06/ethnography-in-hci-comments-on-dourish.html This blog also discusses Paul Dourish's 2006 CHI paper on ethnography in HCI.
http://www.the2the.com/adi/publications_presentations/supporting_rapid_ethnography_hci_field_research_pair_writing.pdf This paper suggests a technique for doing rapid ethnography in HCI - it is good material for a debate on the pros and cons of the technique.