The second survey was web based, and anyone who wished to could go to the website and take part. Results for that survey will be released later.
As with the EQC customer satisfaction survey results blogged yesterday (here), the way in which the wellbeing survey was conducted means it is not possible to associate responses with locations. This also means there is no way of knowing if the various areas most affected by the earthquakes (Red Zones, TC3 land and the hills) have been accurately represented in the survey sample. As the response rate was just 52% it is quite possible that a large part of the earthquake affected population are adversely under-represented in the survey.
A C Nielson, who conducted the survey, have this to say about the survey and it’s results:
Opinion StatementI struggle with the validity of using market research methods for something as important as a post-disaster wellbeing analysis. This is especially so when secret “proprietary methodologies” are used. It may well be that the second half of the statement is correct: the report represents a fair, accurate and comprehensive analysis of the data gathered.
Nielsen certifies that the information contained in this report has been compiled in accordance with sound market research methods and principles, as well as proprietary methodologies developed by, or for, Nielsen. Nielsen believes that this report represents a fair, accurate and comprehensive analysis of the information collected, with all sampled information subject to normal statistical variance.
Where the survey fails, along with EQC’s reports, is in the quality of the information gathered in the first place. Location of respondents is key to identifying concentrations of problems, where wellbeing issues can be expected to be greatest and most life affecting.
In Q21 of the survey respondents were asked if they “live day to day in a damaged home”. Of those who lived in Christchurch 7% said they experienced a major negative impact and another 15% said they experienced a moderate negative impact. Where were these people – Belfast or Bromley, Avonhead or Avonside, TC1 or TC3? How else are they faring?
The presentation of the data is also inadequate for identifying how groups are affected. Q12 asked respondents for their age, and Q13 the household's annual income before tax, however there is no analysis of how people in each of the categories responded to each question in the survey. Also missing is analysis on the wellbeing of women, those who live along, and minority ethnic groups. Are there groups who did better or worse? What can we learn and where is effort needed? Occasional headline snippets are inserted into parts of the report, but there is no analysis of the extent to which such groups are represented in other areas of negative impact.
None of this would be of great importance if we could be confident the survey would go the same way as the report of the Royal Commission on Social Policy. It would also be easy to write it off as just another marketing survey in support of brand “National”. Unfortunately the one page summary that accompanies the survey contains the following heading.
CERA and its partner agencies undertook the Wellbeing Survey to measure earthquake recovery progress across greater Christchurch. It provides timely feedback to social and other agencies as trends in community wellbeing emerge.Unless more analysis is done on the figures, supplemented by social science research, social and other agencies won’t have a clear picture of what needs have to be addressed now (or in the future) and time, money and opportunities will be lost. Along with a number of people.