Study and experimental design

Most of the problems that appear in a later analysis are created at the design stage and cannot be repaired afterwards. A missing baseline measurement, an outcome recorded inconsistently between sites, or a comparison group that was never assembled will each defeat any amount of subsequent modelling. This is the smallest engagement offered and the one with the largest effect on what the data can eventually answer.

Where a change can be introduced deliberately, it can be evaluated by design. Randomisation is often more feasible in a service setting than it first appears: a new element introduced across sites in a staggered order, an offer made to a random half of a waiting list, or a variation in how a reminder is worded all produce a credible comparison at no cost to delivery. The unit of randomisation, the smallest difference worth detecting and the number of clusters needed to detect it are settled before anything starts.

Where randomisation is genuinely not possible, the design work is to build the comparison into the roll-out instead — a staggered introduction that supports a difference-in-differences comparison, an eligibility threshold that supports a regression discontinuity, or a matched comparison group assembled before the intervention rather than afterwards.

The output is a data specification and an analysis plan, dated and agreed in writing before the data exists. An analysis plan written afterwards is a description of what was found, and a reader has no way to distinguish the two. It is also written to be followed by the people who will be collecting the data, because a specification that delivery staff cannot apply produces the missing data it was meant to prevent.