Programme and service evaluation

Most services generate enough data to answer the question they are asked about themselves. Attendance registers, assessment records, referral logs and routine outcome measures accumulate as a by-product of delivery, and an evaluation built from them describes the programme as it runs in practice, across every site and every participant, rather than the version of it that a trial protocol would have produced.

The first question in any such evaluation is who is being counted. A figure computed on the participants who completed answers a narrower question than most readers assume, because people who leave a programme early do not leave at random. We report the effect on everyone enrolled, on everyone who started and on completers, say what each describes, and compare complete-case, multiply-imputed and carried-forward treatments of the missing outcomes so that the assumption behind each is visible.

Delivery structure is carried into the models rather than ignored. Participants sit in groups, groups sit with a deliverer and deliverers sit in venues or localities, and treating those observations as independent understates the uncertainty and hides real variation between sites. Multilevel and variance-component models separate the variation that belongs to the participant from the variation that belongs to the place, which is usually the more actionable of the two.

Subgroup differences are estimated with interaction terms rather than by splitting the sample and comparing significance, and reported with their intervals so that a difference the data cannot resolve is visible as such. Where the design will not support a causal reading, the report says so at the point the claim would have been made.

Evaluation of an organisation’s own service is treated by the Health Research Authority as usual professional practice rather than as research, so the work does not require research ethics committee review and can begin as soon as the data is ready. The organisation’s information governance and any data-sharing or processing agreement apply throughout.