Limitations of RCTs (part 2): How politics and context confound measurement

(This is the second in a three-part series on the limitations of randomized controlled trials (RCTs) in development. As mentioned in my previous post, a recent set of essays in the Boston Review looks at the role of behavioral economics in development. Rachel Glennerster and Michael Kremer provided the opening and anchoring essays.)

The discourse around RCTs fundamentally misses some aspects of what politics and context mean for development. Glennerster and Kremer, along with the other contributors to the Boston Review, ignore the fact that all development has impacts on the local politics and power relationships within a community. This is true even if an intervention is not explicitly political or governmental. Below, I’ll describe how I believe this limits our ability to measure an intervention’s impacts.

Lateral consequences and the political nature of all development

I want to start with the claim that all development activities impact politics and power relationships within a community. To see why this is, let’s play a game I like to call: things-really-are-that-complicated. Here’s how you play.

  1. First, choose a relatively straightforward activity. You can play this game with any kind of development activity: health, education, economic, environmental, etc. For the sake of example, let’s go with building a new school. Suppose the community in question currently has a school that’s too small and is falling apart.
  2. Next, imagine the intended impacts of that activity. You might expect that a new and bigger school will encourage more students to attend, which will raise educational levels among the community’s youth. For the sake of the game, we can assume the activity is successful in these regards.
  3. Finally, think about the unintended impacts the activity might have. These include unintended direct consequences, as well as the indirect knock-on effects of the intended impacts. Be imaginative. For example, a nice new school building could be used for many purposes, and its presence might create conflict between community leaders or others who want to use it for something else. It might provoke jealousy if nearby communities didn’t receive a new school. The building might serve as a community meeting space that improves social capital. The building might also provide organizing space for people who agitate against the local status quo. Education might induce more local youth to move in search of better jobs, which could reduce social cohesion. The possibilities are endless.
  4. Bonus round: Consider the potential impacts of how the activity is conducted. For example, if the school is constructed by foreign volunteers, their presence might have economic and cultural impacts on the community. Similarly, if there are multiple NGOs involved in the area, whether they’re coordinating or not, there are likely to be cumulative impacts. (However, I should note that this game works regardless of whether the intervention is being conducted by local residents or by outsiders to the community.)

You can play this game with any well-intentioned development activity. It also works with apparently simple aid projects like orphanages. The round 3 and 4 possibilities are even more fun if the activity’s intended impacts are at the community or national levels. We might call that the Structural Adjustment edition: start with some broad economic reforms (privatization, deregulation, etc.) and then play out scenarios for what these reforms might mean for the social safety net, public services, the environment, political stability, and more.

These potential unintended consequences don’t necessarily mean that pursuing the activity is a bad idea. Unintended consequences are inevitable given that aid and development efforts often push beyond the realm of merely complicated, into the complex or even chaotic. The point of the exercise is to think laterally. Rather than moving from one cause to its obvious effects, lateral thinking requires us to move sideways to semi-related issues. In general, we’re not very good at doing that. The hardest lateral effects to consider are the political ones, because they manifest at the community level (or higher) and hinge on so many subtleties of cultural norms, power structures, personal relationships — in a word: context.

To summarize: because of the subtleties of context, interventions often have unintended (and possibly unexpected) impacts on politics and other issues.

What unintended impacts mean for RCTs and measurement

Data collection in an RCT (as with any research) is limited by a number of constraints. One of these is resources: measurement of a program’s impact can only cover a limited number of indicators. Staff time is limited, and survey respondents eventually grow weary of the questions. So measurement can only cover the intended outcomes (or proxies for them) and perhaps a few other potential consequences. If context and politics result in unintended lateral consequences from an intervention, then our RCTs will often fail to capture those.

Even for a well-run RCT on a relatively simple intervention, we never fully understand the impacts.

As noted, many of these unexpected consequences will depend greatly on the context. This raises complications for the already-complicated question of external validity. The normal thinking (as put forth by Posner, Glennerster and Kremer, and others) is that testing an intervention in a number of different areas will allow us to understand how context influences the impact that the intervention has. But if we never fully understand the impact in a single context, then how could we ever understand the impact across a variety of contexts? Demonstrating a similar program effect in multiple different contexts would still leave the question of whether differences exist, given that our measures are limited in scope.

False precision

This isn’t really an indictment against RCTs alone. They’re not the only method that fails to capture the political dimensions of development. (Duncan Green is very good at pointing out when politics is missing from otherwise smart analysis: see here and also here.) Even their most stalwart proponents don’t claim that RCTs will answer all of our questions. That said, IPA has ventured into peacebuilding and governance.

I think the danger here comes from a false level of precision. We talk about RCTs as having a scientific rigor that distinguishes them from pseudo-experimental approaches. There is some truth to this. However, if the calculated average effect of a program is stripped of all the caveats and nuance about the things we were unable to measure and calculate, then we risk being overconfident in our knowledge. Science brings a potentially inflated sense of our own expertise. RCTs, and the development industry as a whole, would benefit from less certainty and greater humility.

(A huge hat tip to my friend Sarah Bardack for pre-comments on this post.)