Evidence gap maps, ambient accountability, and more things I learned this week

I’ve had an interesting week of new things. More on those in a second.

First, a shameless plug: I have a new post about the intersection of design and behavioral economics over at Reboot’s blog. It’s mainly a response to the 2015 World Development Report, with some praise, some critiques, and some new ideas thrown in. (Stay tuned for a sequel, focused on politics and power, in the next few days.)

Okay, now on to the new things. I was able to spend some time this week at interesting events and reading, mostly while on trains to or from those events. I came across several new ideas along the way, so I thought I’d share a few.

1. Evidence gap maps: Knowing what you don’t know

On Thursday, I got to sit in on a discussion of evidence for peacebuilding initiatives. It’s a topic I’ve always found interesting because peacebuilding efforts are incredibly difficult to evaluate.

Annette Brown from 3ie presented a tool called an evidence gap map. This is basically a matrix that visualizes where we have evidence and where we don’t, charting intervention types against outcome measures. Here’s an excerpt from 3ie’s education evidence gap map:


(Exported from 3ie.)

Each bubble represents a certain number of studies that test the intervention in the row along the outcome in the column. A quick glance shows that there’s a decent amount of evidence for how school feeding impacts enrollment, but not child labor. Different colored bubbles represent different confidence levels in the evidence.

The peacebuilding gap map is still in an early form and not publicly available yet. Brown previewed a version with just a count of the number of studies and no assessment of the quality of evidence. There’s also no analysis yet of the meaning of the findings; this is especially critical for peacebuilding and conflict research, as studies from different contexts may lead to contradictory generalizations.

There are some implicit assumptions baked into this. The very nature of the visualization suggests that any gaps should be filled, so your selection of categories matters. In his comments at the event, Chris Blattman pointed out that impact evaluations and the gap map both focus on interventions in a way that seems very practical, but is ultimately less useful than a focus on theory that could help us understand what the problem is.

2. Ambient accountability: Making the space do the work

I took a train down to Princeton on Friday to join the “Public Norms & Government Performance” conference, hosted by Innovation for Successful Societies. It was a great mix of ideas and perspectives, in a pretty small group with lots of discussion.

One of the ideas that grabbed me is called ambient accountability. Coined by Dieter Zinnbauer from Transparency International, who presented at the conference, the term refers to the way that the physical environment and built space can enhance accountability. Favorite example: windows that let restaurant chefs and patrons see one another increase reported food quality by 17% and service speed by 13%. More interestingly, even a one-way window that lets chefs see patrons (but not vice versa) has a positive effect on food quality—perhaps a result of empathy as much as accountability.

Another good example comes from airport security. You may have seen these at Heathrow or other European airports:


(Photo credit: Dieter Zinnbauer)

These small kiosks greet travelers just after they have pass through a security checkpoint. It’s a great example of gathering immediate feedback in an intuitive way. The only question is where the data goes. Zinnbauer suggested that it should somehow feed back into the space immediately (e.g. I could imagine the kiosk lighting up in the color of that day’s average report) but it doesn’t sound like the airports are keen on that.

3. Methodology reflexivity: Turning your framework back on itself

Turning a methodology or framework back on itself is a clever (sometimes too clever) rhetorical trick. How does something measure up by its own standards? I saw two interesting examples of that this week.

First, Duncan Green discussed the influencing strategy of the “thinking and working politically” (TWP) agenda. In other words: how can TWP start to TWP? This is necessary for the TWP crowd to start changing thinking and practice throughout the sector, and related efforts (like Doing Development Differently and the Global Delivery Initiative) also face challenges in their influencing strategies.

Green’s post led me to a sobering thought that it might be inherently impossible to institutionalize TWP. After all, politics is the opposite of formal processes and rules. It’s what happens when authority is unclear, information is imperfect, and even values are contested. We call a space “political” precisely when power, decisions, influence, and personal interests are traded freely. That might be inherently in conflict with large institutions, for whom systematization and scale are central. It suggests that a cultural change in the sector, while no small feat itself, might be the easier route.

Quite separately, at the aforementioned Princeton event, the Elizabeth Linos from the Behavioral Insight Team pushed the group to think about what behavioral insights tell us about how to get behavioral insights incorporated into government policies and services. Clearly, just showing evidence is not enough, because policy decisions involve more than simply weighing the evidence.

4. Autocratic athletic events: To the dictator, go the spoils

Last week’s Economist had two separate pieces (here and here) on how the economics of the Olympics and the World Cup are driving them into the arms of autocrats.

The argument in short: The selection criteria for host cities/countries is so skewed toward big new stadiums, excessive pageantry, new services for visitors, and other costly investments that it ends up being a very bad economic deal for the hosts. Places where authorities are elected by voters are shying away, while authoritarians who have less accountability are lining up. Hence, the upcoming World Cups will be in Russia in 2018 and Qatar in 2022; similarly, the 2022 Winter Olympics will be in either Beijing or Almaty, Kazakhstan—the only two bidders. (Winter 2018 will be in Seoul and Summer 2020 in Tokyo, apparently the last democratic hosts for a while.)

We all knew this a bit, but the evidence is stacking up more starkly. I’m finding this distressing because I love the Olympics and the World Cup (and also, democracy). The international gatherings are pretty much the only sports of any kind that I follow.

So Dear FIFA and IOC: Get your act together.

5. And on a lighter note…

The genie and the nonprofit

(From Rhymes with Orange.) 

Though I’m not sure if this is making fun of the restrictions that donors put on grants, or on grant recipients’ expectations that funding should come with no constraints at all…