Difference between revisions of "Usability test results"

Revision as of 18:59, 13 January 2008

Using this test protocol, it was possible to compare the innovative map-based e-consultation technique with the conventional paper questionnaire used in the Probation Board consultation.

[[Image:|thumb|Chart]]

The chart (See Figure 1) shows the difference between the post-test scores given to the web map and the paper consultation questionnaire. The scores were 5 for strongly agree down to 1 for strongly disagree. The difference can range from +4 to -4. Where the question was a negative one, the difference was taken away from 0, so that in all cases +4 is better for the web map, and -4 is worse (i.e. the paper questionnaire is better).

Now for two of the testers (R and S), the web map was better than the paper questionnaire on all but one criterion. The exception was the organization of information on the page. For those with less Internet experience, the questionnaire was better. But on every other criterion of ease of use, the map-based interface was better. The third tester (P) was over 60, and had had less experience of computers and the Internet. On some criteria he found the paper questionnaire easier (based on his experience of filling in forms in the past). Nevertheless, he was more satisfied overall with the new web map than with the traditional questionnaire.

During the tests, there were frequent complaints about the paper questionnaire. Q5 was particularly hard to grasp. It reads like a university or A-level examination question.48 But even when answering the simpler questions, the testers discussed what the questions mean before attempting to answer. In contrast, there were few problems when using the on-line map. They found the probation office locations, recognised local features, and managed to manipulate and move around the map display. Writing in several words as a comment on a particular site was not a problem for two of them: the third got the research assistant to help him. In the discussion afterwards, they expressed their satisfaction with the map interface, and how easy it was to use. Appendix 7 shows some sample comments added to locations on the map. They relate directly to the issues of the consultation, focused on specific sites. By starting from the particular, rather than requiring consultees to give general comments, it is both easier for consultees to express their views, and consulters to understand and analyse them.

48 The question started: “5. PBNI Equality Scheme has given a commitment that in carrying out its functions it will have due regard to the need to promote equality of opportunity between people who fall under the following groups as stated in s75 of the Northern Ireland Act 1998. To help us consider potential inequalities please identify the people in the following groups who you believe will be most affected by the proposed changes (for staff and service users):- Religious belief: Positive [ ] Negative [ ] ...”