What does it all add up to? Measurement at the agency level, part 1

You’ve already heard my thoughts on how any effort to rank NGOs will descend into absurdity. The major difficulty with such an endeavor stems from the remarkable range of organizations that fall in the “NGO” bucket. Let’s call that the “Wikimedia v. Oxfam” problem. How would you ever make a meaningful statement about which of those organizations is “better”?

A slightly smaller hurdle is presented by the range of activities within a given organization. Many large NGOs conduct a variety of activities (direct relief, long-term development, policy advocacy, etc.) across a range of issues (education, health, WASH, etc.) in a variety of countries. Even if we acknowledge the intractability of the “Wikimedia v. Oxfam” problem, we may still want to get a handle on how a given organization is doing on its own terms. Can the M&E of those disparate activities be added up to something more comprehensive?

The tentative answer is: “Yes, but it’s tricky.” Let’s start with the “why” and then move into some of the complications. I’ll draw a few lessons from the corporate world, and then a later post will have a case study from a real live NGO.

1. Why pursue measurement at the agency level?

There are two possible reasons to do this, and they may lead to different decisions on the “how” questions. Measurement at the agency level could serve…

As a management and strategy tool: Picture yourself as a board member of a large international NGO. The CEO stands in front of you, telling you that the organization has had a great year and that she has a plan for making the next year even better. Do you believe her? To make a proper judgment, you need more than just a financial report. You need to understand the whole scope and impact of the NGO’s work. But you’re a busy person and you don’t have time to analyze detailed narratives of every program (nor does some poor staffer have the time to assemble them). You need a relatively small set of numbers and visual aids to convey the information, so you can digest it and draw appropriate conclusions. So what should the CEO put in front of you to convince you?

As a public relations/fundraising tool: Picture yourself as a donor, scanning NGO websites. Most of them provide you with a few numbers on the size and scope of their work: number of countries they operate in, number of staff, annual expenditures, etc. These numbers can be useful for establishing legitimacy, but as GiveWell has observed, they don’t tell us much about what the agency actually does or what impact it has.

Depending on which audience you choose to serve, you might make different decisions in the measurement process. I’m more interested in the first use: to inform management and strategy decisions. The rest of this series will focus on that. Even at levels lower than the board, such data can be useful for troubleshooting problems, spotting opportunities, and generally helping managers focus their attention more efficiently.

2. What complications arise?

As I hinted above, there are a number of challenges to agency-wide measurement. Here are just a few:

  • Type of programming: Some issues lend themselves more easily to quantitative indicators. Compare health interventions to local peacebuilding work: there’s far more consensus around the appropriate indicators for the former than for the latter.
  • Variety of programming: NGOs that work on only one or a few issues will find it easier to maintain the same indicators across multiple programs, which naturally makes aggregation easier.
  • Variety of context: Even similar programs in wildly different regions may be hard to capture with the same set of indicators.
  • Long-term impacts: These are notoriously hard to measure in any circumstance. For certain agencies and activities, it may be particularly tricky to make useful measurements. For example, in a humanitarian relief setting, the decision loop is much too short to observe the long-term impacts of any given action. The tendency will be to focus on activities and outputs instead.
  • Organization-specific factors: There are as many types of NGOs as there are NGOs. Organizational culture matters a great deal. So do factors like whether an NGO is highly decentralized, currently has strong M&E systems, or gets funding from a variety of sources. So as always, individual results may vary.

These are just a few of the hurdles. I’m sure you can think of more, like simple organizational inertia. Speaking of which…

3. How do you even get this system off the ground?

Forget the bullet points above. The biggest challenge? Simply managing the whole process of implementing such a system. You have to create an overall framework, define the specifics, design something to manage the information, and eventually implement it all. If done right, this process will allow you to sort through the problems mentioned above.

You’ll need input and involvement from all corners of the organization. If you’ve ever tried to manage a project across multiple departments/countries, you understand what a slog it can be. It requires support at the executive level, a fair amount of cohesion across the organization, and persistence.

4. An aside: How does the corporate world manage it?

I’m a bit skeptical of the “run NGOs like businesses” attitude that pervades many parts of our sector. I don’t believe that the attitude is wrong, so much as incomplete. I would suggest a fuller version: run NGOs like good businesses, and run businesses like good NGOs, and generally learn best practices regardless of where they come from.

That said: I think this is a topic where business best practice is well ahead of almost every NGO. Although profit is any company’s final metric, most will rigorously track and manage to a number of daily metrics: size of average purchase, number of customers served, delivery time, staff efficiency, repeat customer rate, etc. These metrics are the drivers of success for which mid- and lower-level employees are held accountable. They’re different for every company.

Companies face the same set of challenges as NGOs when selecting metrics, implementing systems to track them, and making decisions based on them. We can learn from the corporate experience. The first lesson is that it takes real investment in infrastructure (both human and technological) to overcome these challenges. The second lesson is that the payoff to doing it right (in terms of success against mission) can be huge.

5. Still seems a bit abstract, huh?

I’m working on a case study on how one large NGO tackles this issue. That should give you a clearer picture of how this works in practice. I might do a few case studies on other organizations too, depending on what information is available and how much time I have. If you think your organization handles agency-level measurement well, or has a unique situation, or just has some interesting lessons to share, please get in touch.

Stay tuned…


Related posts: