User testing

7.2 User testing

On 26 May, this research project was given the opportunity to test an experience prototype at a WREMO volunteer evening.

Of the approximately 20 volunteers in attendance, most had their own smartphone and were able to access this project’s supplied prototype.
In a controlled test, the user is normally led through the experience.

This allows the test host to enquire after the reason that prompted the user to make so that user considers and explains their decisions in response to how the prototype encourages them to behave.

On this occasion, the testing environment did not allow for a lead situation, so the users were given test scripts to follow independently within the experience prototype and to write down their responses. This process amounted to 18 (documented) responses to five user journeys each with a specific task.

7.2.1 Feedback from user testing process

Practically none of the testers had used a ‘crisis management’ tool, or similar application related to civil defence on a mobile phone.

Most users were comfortable with the idea of maps on their smartphone as most of them had used a mapping service on their phone before. Ultimately this is the first overarching lesson learnt: the ubiquity of maps on a smartphone device dictated how the users approached it.

The testers aired concerns regarding whether they would have control over the map (as opposed to a static image-only map), and whether the map would show their location “…like on Google Maps.” A screenshot of a Google Map was used to simulate live data, but despite this, the appearance of the map gave users pause to consider how they thought the map might respond in terms of their prior experiences.

The second key finding from the test regarded the use of language. Many people disagreed, or disliked the terminology of the prototype. In some ways, this could be brought down to the limitations of linear test in a static environment, but it also suggested that the ‘civil defence jargon’ language used by emergency management organisations sounded too  much “like government speak” (User 18), or simply not friendly enough.

The user test was, by a matter of circumstance a blind test. Not only did the user tester have to enter the URL of the prototype themselves, they had to parse a test script that described each assigned task. The wide array of backgrounds ages and technical competence made for an unexpectedly varied and diverse test.

All users had the option of choosing their own task(s). They were:

1: Scenario: Business-As-Usual (No Emergency), Setting Up

• Locate the ‘learn about the maps’ function. Do you think this
  would be useful?
• Where would you look for information about the app from the
  home screen?
• What kind of information would you expect to find here?

2: Scenario: Business-As-Usual (No Emergency), Day-To-Day Use

• Locate the event map – what do you think this map would be for?
• Add a ‘potential community asset’. What is one of these? How would you add it?
• Add a privacy setting for your ‘potential community asset’

3: Scenario: Business-As-Usual (No Emergency),Day-To-Day Use

• Locate the map – what do you think this map would be for?
• How would you add an event?

4: Scenario: Post-Event (Emergency) Tasks:

• Add a report of some damage (a fire)
• Locate a report on the map and confirm or add to it
  
  

In addition to the above, please see Appendix 3 for the questionnaire that guided the volunteer users through the different journeys.

The following verbatims are included to highlight some of the more notable feedback encountered:

In conclusion, there is one more scenario that was unexpectedly brought to light. An elderly gentlemen (possibly in his 70s) was quite familiar with a smartphone but was not necessarily comfortable with the idea of the web (accessed in a browser), or maps in general (he stated he didn’t need them as he was a long time resident).

When asked how he used his phone, he said that he only used it for email and short form messaging. This observation was informative for understanding how the elderly might benefit from, or otherwise ignore, a tool like this. This one lesson also suggests that in the next few years, organisations like WREMO will have to maintain targeted messaging for certain sectors of the society.