One of the main ways of finding out what users like and don't like about a product is by asking them, which can be done using interviews or questionnaires as we discussed in Chapter 13. One of the benefits of using questionnaires is that the same questions can be given to many people at the same time. Questionnaires are also relatively inexpensive to distribute and analyze, particularly web-based questionnaires, which are becoming increasingly common. Web-based questionnaires also have the advantage that data can be stored directly into a database for analysis.

Gary Perlman's website (www.acm.org/~perlman/question.html) provides eight online questionnaires that you can use. These questionnaires are based on the work of experts in human-computer interaction.

There are two ways of using the questionnaires:

1. The most straightforward way is to use the questionnaires as they are without changing any code. You simply select the questionnaire that best suits your needs and either ask the participants to complete it online and email their data to you or print the questionnaire and distribute it on paper.
2. You can customize the questionnaires by changing the cgi code and there are instructions for doing this on the website. The advantage of this approach is that you can decide what the questions should be and you can distribute the questionnaire from your own university server, which gives you more control over the process. To do this you'll need some technical expertise.
   

Activity 1

This activity encourages you to review the questionnaires before using them for an evaluation. Start by going to the website (www.acm.org/~perlman/question.html) and explore and compare the eight questionnaires. Try completing some of them and send the data to yourself. How do they differ? Here are some things that we noticed, you may find others:

• The questionnaires are derived from different sources but they all use various forms of Likert scales.
• The number of questions varies from 3 to 100. When might you want only 3 questions and when might you want more? If you are planning an evaluation, which are the most appropriate questions for your evaluation and what is the ideal number?
• Notice that you can add comment boxes by clicking on the little icons or by requesting that comment boxes are added to all the questions. The QUIS questionnaire also has 6 blank spaces for questions.
• There are differences between the scales; some are 7-point and some are 9-point scales. Most scales start from 1 but the QUIS scale starts from zero. Most provide a NA (not applicable) option too. The higher the number on the scale the more positive the rating. Some of the questionnaires also have semantic labels such as 'bad' and 'good' to show the direction of the scale. QUIS uses pairs of adjectives and places one word at each end of the scale. All the questionnaires repeat the scale for each group of questions so that users don't get confused.
• Some of the questionnaires invite users to write additional comments in a text box at the end.

Also notice that there are useful links to papers or abstracts written by the questionnaire authors and to additional information about questionnaire design. For example, take a look at the link to 'Frequently Asked Questions about Questionnaires in Usability Engineering' provided by Jurek Kirakowski.

When you've finished examining the questionnaires scroll to the bottom of the page and read about how to modify the questionnaire and look at the example of the data format that is emailed for one of the questionnaires

Activity 2

In this activity you will compare two different questionnaires. Select two questionnaires of your choice to compare and then ask 3 participants to complete each questionnaire for an evaluation of one of the following items: a VCR, a computerized toy, a clock, the home page of a website, or something else of your choice. In order to make sure that the participants experience the product, you should suggest a few tasks for them to do with it before completing the questionnaires. For example, insert and play a video, rewind it, set the VCR to record a program. When the questionnaires have been completed compare the data that each provides. Which questionnaire provides the most useful data about the product's usability and why do you think this? Which questionnaire did the participants prefer, and why?

Activity 3

In this activity you will compare the efficacy of the Nielsen Heuristic Evaluation (NHE) questionnaire with standard heuristic evaluation. To do this, start by selecting a product, such as those mentioned in activity 2, and ask at least 3 typical users to evaluate it using the questionnaire. As in activity 2, you should suggest tasks that the participants should do with the product so that they experience it. Then ask one or two expert evaluators to use the same heuristics to evaluate the same product but this time give them just the list heuristics (i.e., not in questionnaire form) and ask them to follow the process for heuristic evaluation described in Chapter 13. You can use peers from your class as experts but if you do this try to get people outside of your class to participate in the questionnaire part of the activity. Do the two groups (users and experts) report on the same issues or are their responses quite different? What are the advantages and limitations of heuristic evaluation verses collecting data with the NHE questionnaire?