We were privileged to have Jack Colford of UC Berkeley deliver a 4 hour (!) session on impact evaluation in the Neuroscience building yesterday. All in all, Dr Colford’s insight was even more impressive than the list of letters next to his name. He ran through applied examples of evaluating the impact of global health interventions in California, Mexico, Kenya, Bangladesh and India, to name a few (‘where hasn’t he done/run a study?’ was one of my unanswered questions).
If you missed it (or forgot a pen), some highlights of his insight included:
1. Stepped Wedge randomized controlled trials: In this design, communities are randomized to receive the intervention at each time interval, x. At x0, no community has it, at x1, some number of communities get ‘treated’, at x2, some more communities get it, and so on. In this way, all communities eventually get the intervention. Further, all communities get to exist as both controls and treated groups at different time points (familiar to a case crossover?!), enabling some particularly interesting analyses (such as separating the effect in treatment compliers from communities who would have seen the effect regardless of treatment!)
2. Different options for randomization when you know baseline characteristics are dissimilar:
-Match similar pairs, then randomize 1 partner to receive the intervention, and 1 to be control
-‘Big stick’ randomization: let a computer create 10000’s of randomization sequences, and flag the sequences that meet your pre-specified ‘acceptable’ criterion for similar distributions of characteristics. Of the resulting possible randomizations, you randomly choose 1.
3. Retrospective impact evaluation: Sometimes (most of the time!), the program is already in place so it is not possible to implement it randomly or using stepped-wedge. In this case, communities can be assigned propensity scores, i.e. the probability of having received the intervention (which is based on a set of covariates you derive from various sources). These communities are compared against controls with similar propensity scores who happened to miss out on the intervention. So you’re essentially ‘mimicking’ randomization in hindsight.
4. Collaboration is essential but flawed: Partners (NGOs, funders, governments and otherwise) are highly necessary, but it seems that differing reward systems hinder these collaborations’ potential:
-People at the World Bank are more highly rewarded for getting something done then actually doing something.
-People in NGOs are more highly rewarded for doing something then evaluating whether it was done right
-We, researchers, are more highly rewarded for publishing whether something was done right than disseminating this vital information to stakeholders.
In other words, the system is a series of contra-indications (let me guess – as if the whole word wasn’t enough, it’s on us to fix this one too).
For more on Jack Colford’s work and methods:
–Paper illustrating stepped wedge RCT design
–Methodological paper on estimating Complier Average Causal Effect in stepped wedge design
-Paper illustrating matching when intervention already exists
–Methodological paper on matching when intervention already exists