The material on Beyond Budgets is the responsibility of individual authors alone. It does not represent the views of any project, organisation or government.


Randomized control trials (RCTs): interesting, but a marginal tool for governments 

Author: ODI Researcher Philipp Krause

Randomized control trials (RCTs) have had a great decade. The stunning line-up of speakers who celebrated J-PAL’s tenth anniversary in Boston last December gives some indication of just how great. They are the shiny new tool of development policy, and a lot of them are pretty cool. Browsing through J-PAL’s library of projects, it’s easy to see how so many of them end up in top-notch academic journals.  

So far, so good. But the ambition of RCTs is not just to provide a gold-standard measurement of impact. They aim to actually have an impact on the real world themselves. The scenario goes something like this: researchers investigate the effect of an intervention and use the findings to either get out of that mess quickly (if the intervention doesn’t work) or scale it up quickly (if it does). In the pursuit of this impact-seeker’s Nirvana, it’s easy to conflate a couple of things, notably that an RCT is not the only way to evaluate impact; and evaluating impact is not the only way to use evidence for policy. Unfortunately, it is now surprisingly common to hear RCTs conflated with evidence-use, and evidence-use equated with the key ingredient for better public services in developing countries. The reality of evidence use is different.

Today’s rich countries didn’t get rich by using evidence systematically. This is a point that we recently discussed at a big World Bank – ODI conference on the (coincidental?) tenth anniversary of the WDR 2004. Lant Pritchett made it best when describing Randomistas as engaging in faith-based activity: nobody could accuse the likes of Germany, Switzerland, Sweden or the US of achieving human development by systematically scaling up what works.

What these countries do have in spades is people noisily demanding stuff, and governments giving it to them. In fact, some of the greatest innovations in providing health, unemployment benefits and pensions to poor people (and taking them to scale) happened because citizens seemed to want them, and giving them stuff seemed like a good way to shut them up. Ask Otto Bismarck. It’s not too much of a stretch to call this the history of public spending in a nutshell.

The big assumption of scaling up the evidence from RCTs is that politicians and officials actually care. The findings may convince international agencies (who may well have commissioned the research in the first place), but aid doesn’t drive government decisions, governments themselves do. And, except for the most aid-dependent countries, domestic taxes fund the public spending that the evidence is supposed to influence. Governments need a reason to care about public services and ways to improve their delivery. Not all of them do. It is politically naïve to think that better information is the binding constraint to better services in countries where better services aren’t a government priority. More likely, RCTs are like trees falling in a forest with nobody around to hear them fall. It’s like they never happened.

Quite a few of these arguments could be put under empirical scrutiny. For instance, we do not have a counterfactual for the experience of today’s rich countries. Perhaps services would be better still if governments had used better evidence in the past (or could be improved by doing so today). It would be worth taking a closer look at use of impact evidence and how that tracks across countries. To my knowledge, not much comparative work has been done on this. In his recent blog post, Lant points out that RCTs have been known as a policy tool in the US since the 70s but never caught on.

And what about development? At the WDR 2004+10 conference, Nora Lustig referred to what must be the most frequently cited case for impact and scaling up of an RCT: Mexico’s Progresa/Oportunidades. The story in a nutshell is that incoming presidential administrations in Mexico traditionally canned their predecessors’ flagship initiatives. In 2000, Progresa, an innovative conditional cash transfer (CCT) program, was one such a flagship that seemed to be in acute danger of elimination. Evidence from an RCT-based evaluation helped persuade the incoming government to keep it (though under a different name). For CCT programs, the rest is history.

So have we seen a wave of policy-shaping RCTs being conducted in Mexico since 2000?

Well, not quite. The government did develop an institutional interest in evidence to inform social policy. A semi-independent national evaluation council (CONEVAL) was established to provide evidence to the government. A legal provision that every social program be annually evaluated for its impact was also passed. In practice, however, officials quickly found that RCTs are time consuming and very expensive. They found lessons to be learned from Chile, which had developed a sophisticated portfolio of evaluation tools.

Similarly to Chile, Mexico now uses a menu of options for evaluations. The bulk of these are systematic desk-based evaluations that look at issues of consistency and management, outputs against targets, and so forth. In Chile, one such ‘program evaluation’ cost about $12,000 (in 2003, inflation will have pushed it up accordingly). Compare that to RCTs, which are easily ten times more expensive, if not considerably more. Despite the relatively low cost, they are having a big impact on the efficiency of public spending, resulting in often profound reorganisations and budgetary re-allocations. Mexican and Chilean officials have found enough incremental work to be done to improve program delivery (‘how can we make this program a bit better, or cheaper, or less bureaucratic?’) , so that big, potentially scalable innovation (‘does this work?’) is pretty marginal.

The bottom line is governments s that care about impact have plenty of cheaper, timelier and more appropriate tools and options available to them than RCTs. That doesn’t mean RCTs shouldn’t be done, of course. And the evaluation of aid is a different matter altogether, where donors are free to be as inefficient about evidence-basing as they wish without burdening poor countries.

But for governments the choice of how to go about using systematic evidence is theirs to make. And it’s a tough capability to pick up. Many governments choose not to do it, and there’s no evidence that they suffer for it. It would be wrong for donors to suggest to low-income countries that RCTs are in any way critical for their public service capability. Better call them what they are: interesting, but marginal.

PrintView Printer Friendly Version

EmailEmail Article to Friend

References (7)

References allow you to track sources for this article, as well as articles that were written in response to this article.
  • Response
    beyondbudgets - Beyond Budgets - Randomized control trials (RCTs): interesting, but a marginal tool for governments
  • Response
    beyondbudgets - Beyond Budgets - Randomized control trials (RCTs): interesting, but a marginal tool for governments
  • Response
    Response: Long island SEO
    beyondbudgets - Beyond Budgets - Randomized control trials (RCTs): interesting, but a marginal tool for governments
  • Response
    Response: Click here.
    beyondbudgets - Beyond Budgets - Randomized control trials (RCTs): interesting, but a marginal tool for governments
  • Response
  • Response
    beyondbudgets - Beyond Budgets - Randomized control trials (RCTs): interesting, but a marginal tool for governments
  • Response
    Response: best golf packages
    beyondbudgets - Beyond Budgets - Randomized control trials (RCTs): interesting, but a marginal tool for governments

Reader Comments (2)

Here, here. Sure some papers will get published but overall, marginal.

I love the asymmetric application of critical thought to evidence.

I have heard people from the RCT camp repeat the PROGRESA story over and over as evidence that RCT has a causal impact on policy outcomes. yet this could not be more "post hoc ergo propter hoc" hocy.

I was just reading today a paper in which the author was looking at outcomes for children who enrolled in kindergarten just before and after an arbitrary cut-off, but felt compelled to worry that may be couples were timing their sex to produce children near that cut-off.

Its like people have a PhD in criticizing causality claims of others but are still in causality kindergarten for telling "just so" stories that support their narrative. Which applies to all of us I suppose.

March 24, 2014 | Unregistered CommenterLant Pritchett

Lant, many thanks! You're right of course about the "just so" story, but then it's true that everyone does that. It's policy development, after all, and entrepreneurs with ideas will be selective, whether deliberately or not. You don't advance your project by starting out with 15 caveats.

What's more important (to me) is that there is a really important implications from the work around Progresa and the work of the gov'ts of Mexico, Chile and surely others. It just isn't about RCTs, it's about what prompts some parts of some governments to care about the efficiency and effectiveness of spending when so many others don't. It's just about putting the horse back before the cart.

March 25, 2014 | Registered CommenterPhilipp Krause

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>
« Podcast: Helder Da Costa from the g7+ talks on post-2015 & the New Deal | Main | Giving institutional reforms a chance to succeed »