Survey design and analysis

A survey is a measurement instrument and a sampling procedure at the same time, and either can undo the other. Work at the design stage covers the frame and how it is constructed, the sampling scheme, the number of respondents needed for the precision the question requires, and the questionnaire itself — its ordering, its response scales, and whether the items behave the way they are intended to.

Analysis respects the design that produced the data. Clustering, stratification and unequal selection probabilities all affect the standard errors, and estimates computed as though the sample were simple random will be too precise, sometimes by a wide margin. We apply design weights, calibrate to known population totals where they exist, and estimate variance in a way that reflects how the sample was drawn.

Non-response is treated as a question rather than as a setting. Response propensity is modelled, the respondents are compared with the frame on everything the frame knows about them, and the sensitivity of the estimates to the weighting and imputation choices is reported rather than absorbed. Where a service wants to compare the people it reaches with the population it is funded to serve, the same machinery places both on one basis.

The work extends to secondary analysis of existing surveys, including the large national studies whose complex designs and derived weights are frequently applied incorrectly, and to the construction and validation of scales through factor analysis and item response models.