7. Observation of Search Behaviour

7.1 Walk Throughs

Six participants in the series of observed walk throughs searched for information on a set of pre-selected topics on government web sites, and indicated how they would go about answering the question. The tasks involved finding information on:

1. The justification and risks of spraying areas in Auckland for the painted apple moth (this information was available on the MAF site);

2. Information and necessary forms that must be completed by landlords and tenants as part of setting up a rental agreement (this information was available on the Tenancy Services web site, which is linked from the Ministry of Housing;

3. Information for the participant's [fictional] recently widowed mother about how much she will receive on the single pension, with a living alone allowance, and how much additional income she can have without penalty (this information is on the Work and Income web site, under the heading Get financial assistance/Main benefits/ New Zealand Superannuation);

4. Information about facilities, and whether you can make a booking for the DOC camp site at Elaine Bay, in Pelorus Sounds (information is available on the DOC web site, but the camp site is not bookable);

5. Information behind a recent news announcement that Māori life expectancy had increased slightly, and recent figures on Māori health status (a number of sources are need to compile this information).

None of the six participants, who ranged from a young male computer expert to a retired female civil servant, were able to complete all of the tasks fully and completely although some got much further than others. Some only managed to tackle three tasks in the allotted time for the walk-through sessions (1 hour). Participants used a range of search strategies, but the approach of each individual to each of the set of tasks was relatively consistent.

Most participants in the walk-throughs started their search in one of two ways: if they knew or could guess the URL of the web site they would enter that directly into the address box. Once they reached the web site of the agency they would then search either by navigating through the site, using headings, menus or textual links. Although some later used a search box on a government web site, this was later in the search. Alternatively, they would employ a search engine (either Google, or MSN (because MSN search was the more obvious option on the PC being used, although some participants were less familiar with it). Once they had opened a search engine, participants did one of two things: they would either enter the department name, or what they thought was part of its URL, or the topic they wished to search for (a small number of searches combined the two—name and topic). In total ten of the searches started with MSN, six using the department name and four focused on topic; five started with Google (two using the department name, and three the topic), and eleven started with the URL (ten of which then proceeded to navigation through headings and links, and one employed a search engine). Table 8 shows how effective participants were in their search strategies.

Table 8. Number of Clicks Taken to Reach Target Information in Walk-through Observations (Compared with the Number Needed)

Task (Number of clicks needed to fully complete the assignment, depending on which final version of information is considered to answer the question)

NA = Task not attempted

NC = Task not completed

C = Task essentially completed, although participant has not identified information exactly

FC = Task fully completed—participant can identify relevant information

E = Task ended for another reasons

P = Participant would phone at this point

On many occasions participants had reached the right part of a web site to retrieve the information they were searching for, but did not fully recognise this. On 10 out of 26 occasions participants indicated they would phone the department in question, whether or not they had retrieved the required information. Their reasons for this were largely uncertainty that they could now act on the information retrieved, and a reluctance to trust an online booking. Two out of the six participants had heard of the government portal, but they would not use it for this kind of search. Some would arrive at it as part of their search, but showed no recognition of what it was.

7. 2 Problems Participants Encountered

The sorts of problems participants encountered as they attempted to complete the tasks were that they were:

• Not sure which agency web site would contain the information, and, if they knew which one it might be, what its URL was;
• Not sure which agencies were parts of which department, or the structure of a department, and "who does what";
• Not sure whether a particular heading would take them where they wanted to go on the site, or how information on the site was categorised;
• Lacking skill in being able to interpret the results of a search by Google or MSN, or to read the keywords, or URLs given in annotations on search results;
• Lacking some basic knowledge of web navigation, and protocols (e.g. links changing colour once activated);
• Unskilled in using search engines (e.g. in choice of search terms, or restriction to New Zealand sources);
• Failing to read all the information on the page that would answer a question, especially if it was a lengthy pdf file (as in the DOC campsite information brochure);
• Not employing the same language as agencies (e.g. many people continued to search for the commonly used term pension, although the information they sought required them to recognise that it was found under the heading New Zealand superannuation);

• Complaining about the size of fonts (especially on the portal if they ended up there);
• Not always seeing links or important headings that required scrolling vertically or horizontally;
• Stymied by a lack of live links to critical information within web sites, and especially from parts of the portal (an exception was the Tenancy site, where forms are well linked and labelled);
• Wanting more links to have explanatory notes of what they linked to, and terms and headings to be more clearly defined;
• Failing to link to critical information on other government web sites (e.g. the failure of the Ministry of Health site to suggest users also check the Statistics NZ site for life tables).

Searching was often a question of hit and miss, going backwards and forwards, sometimes over the same territory, often with a rising sense of frustration, and resulting in "at this point I would call the department." The problems listed above can be attributed to two main causes. The first is lack of user skills. Agencies need to take this into account and provide more support for unskilled users of their web sites. The second cause is poor design, and lack of user-friendly web site organisation and navigation, and the fact that the language is not focused on the needs of those approaching the site. In some cases brochures put on the web as pdf files do not permit links out to useful information, and internal web site search engines do not search contents of pdf files very well. Many online brochures in pdf format need to be redesigned for web use, and the habits of web users, in order to relinquish their information more easily—an example is the difficulty for web users in putting two separate pieces of information together (as in the case of the Department of Conservation booklet on South Island conservation camp sites, which explained in one place that standard camp sites are not bookable, and then several pages further on, that the Elaine Bay site is a standard camp site. What users quickly see in print, they do not see on a computer screen.) By contrast, users were able to get tenancy information in pdf format or printable files, and link from information pages to forms. This was the task most successfully completed by participants.