Aid transparency and internal evaluations

There’s been lots of interesting movement and commentary on aid transparency recently, including the following:

  • Owen Abroad blogged today on why donors and others should just put the data out there, rather than focusing on interpreting (or spinning) it for users.
  • Easterly’s AidWatch blog recently lauded the World Bank for making its data available and user-friendly.
  • Texas in Africa has a guest post from AidData describing how their system could be useful to project managers (as opposed to just researchers/academics).

I agree whole-heartedly with Owen. The data needs to be free — and also easily searchable. So I applaud AidData for bringing the information together from multiple sources. The important point is that the organization holding the data will never be able to predict all the useful things someone might do with it. Better to make it accessible and let others decide.

Of course, numerical data is not the only thing that needs to be public. Project documents and evaluations should be as well. Accessibility and search-ability become harder with these. I bumped up against this a few weeks ago when I was researching the impact of multilaterals on decentralization in developing countries. (There are some interesting questions about sovereignty and agency, but I’ll save those issues for another day.)

What I discovered was a stark contrast between the World Bank and UNDP when it comes to providing project information. The Bank makes documentation on all their projects (relatively) searchable by country, region, sector, theme, etc. UNDP provides nothing of the sort. Most UNDP country websites have project lists, but the agency is decentralized enough that there’s no way to search across countries. If you’ve ever worked in a large bureaucracy, you can imagine that such a central system would place a massive burden on the country staff. Entering project information in a standardized format takes time. And it wouldn’t be very accurate for creative interventions that fall outside the “project” paradigm of aid delivery.

So my attempt to understand how UNDP engages with decentralization efforts took a lot of time, as I waded through each individual country site. I was hopeful that someone in UNDP’s evaluation office had already done the legwork. “Thematic Evaluations” looked especially promising — until I discovered that the most recent document on decentralization is a decade old. Dead end.

But all was not lost. As long as I was on the site, I perused the “Independent Review of UNDP Evaluation Policy” from earlier this year. I haven’t read it thoroughly, but here are some choice quotes from the “Overarching Conclusions” section. For starters, the reviewers found:

“There is little need to revise the present [evaluation] policy document, as the correct principles are identified.”

Translation: Some smart person wrote a good policy. So far so good. However:

“Specifying changes in operational procedures for budget preparation and human resources is required.”

And furthermore:

“Independent evaluation…has not been sufficiently institutionalized in systems and procedures.”

As usual, the challenge is implementation. And a final interesting tidbit:

“Increase in the quality of independent evaluations hinges on strengthening the basic ‘evaluability’ of UNDP interventions.”

In other words: evaluation starts with program design. You can design a program in a way that lends itself to evaluation, and start the evaluation early enough to be meaningful, without compromising the program legitimacy.

The report seemed to acknowledge that UNDP’s decentralized structure provides a hurdle to implementing the evaluation policy and making interventions more “evaluable” (evaluatable?), just as it provides a hurdle to creating a central database of projects. The report offered recommendations. The first one stated:

“…the most significant challenge lies with the pace and commitment of senior management to drive improvements in the decentralized evaluation system, as envisaged in the evaluation policy.”

Which is to say, they’re not really trying. But it reminds me of the common complaint that insufficient “political will” is responsible for the failure of various developing country reforms. In my opinion, this is an analytical punt insofar as it ignores the (lack of) incentives facing key actors.

I justified this detour in my decentralization research with the question: How can UNDP have a beneficial impact on decentralization in developing countries, if it can’t manage decentralization within itself? Really, I was just procrastinating. But this issue does have implications for the aid transparency movement: sometimes there are internal organizational constraints compounding the external (dis)incentives to making information available. Such constraints may be difficult to overcome.