What we know
Six families of method. Most engagements draw on more than one, and the method follows from the question rather than the other way round.
Most services already hold the data that answers the question they are asked about themselves. Attendance, referrals, assessments and outcome measures accumulate as a by-product of delivery.
The first question in any evaluation is who is being counted, because people who leave early do not leave at random. We report the effect on everyone enrolled and on those who completed, and say what each of those describes.
A survey is a measurement instrument and a sampling procedure at once, and either can undo the other.
Design work covers the frame, the sampling scheme and the number of responses the question needs. Analysis carries that design through to the standard errors, because estimates computed as though the sample were simple random are too precise, sometimes by a wide margin.
Most quantitative questions reduce to a regression of some kind, and most of the difficulty is choosing which kind and defending it.
Data is rarely a flat table of independent observations: patients sit within practices, participants within groups, areas within regions. Multilevel models represent that directly and report how much variation belongs at each level, which is often the most useful output of the exercise.
Repeated measurement is the most under-used asset in routine data. A single cross-section can only compare people who differ in every respect.
A second observation lets each person act as their own comparison, removing everything stable about them that would otherwise confound the estimate. Growth models then describe trajectories rather than endpoints.
Many questions are about timing. When members stop attending, how long people wait, how quickly they return, how long a change lasts.
Reducing any of those to a yes or no at a fixed follow-up throws away most of the information. Time-to-event methods keep everyone in the analysis for exactly as long as they were observed.
Most of the problems that appear in an analysis were created at the design stage and cannot be repaired afterwards. A missing baseline, an outcome recorded inconsistently between sites, or a comparison group that was never assembled will each defeat any amount of subsequent modelling.
The output is a data specification and an analysis plan, dated and agreed in writing before the data exists.
Where a piece of work needs a method outside that range, that is said at the outset and it is carried out alongside specialists in it. Where a question cannot be answered with the data available, we say so before the work starts and set out what would be needed instead.
Describe the question and the data, and we will say what is feasible.