What Humanitarian and Development Implementers Say about the Path to Geospatial Impact Evaluation

Last year, the Mercy Corps team within the GeoField initiative worked to conduct and report on a landscape assessment on the status of geospatial data collection and use at Mercy Corps, framed as our readiness to engage in geospatial impact evaluations (GIEs) like those seen conducted across the GeoField Community of Practice. The assessment found that of 258 active programs, around 12% (33 programs) were collecting spatial data at a level sufficient to basically enable a GIE, defined simply as GPS points for households or implementation sites. Of those programs, very few (only 1 or 2) had engaged in GIEs, leaving a lot of opportunity around data that is there. As we’ve seen throughout GeoField’s use cases and dozens of other evaluations from the GeoField community, accessible and usable ground data is key to the ability to make the geospatial connection for program evaluation. 

This year, we wanted to extend the understanding of Mercy Corps’ assessment to our peer organizations, so we spoke with technology practitioners at four peer INGOs — Save the Children International, Save the Children Norway, World Vision International, and Catholic Relief Services — to hear their perspectives on the barriers, enablers, and resource gaps around spatial data collection and use. Our conversations touched on GIE readiness specifically, and ranged into how organizations collect, manage, and actually use spatial data for program learning and monitoring, foundational to GIE readiness.

Those interviewed were usually in the technology or Monitoring, Evaluation, Accountability, and Learning (MEAL) space and offered clear insight into their organizational structure and function, but certainly didn’t claim to speak for use and collection of geospatial data across their whole organization. We don’t present these conversations as a systematic survey, but rather a candid, grounded set of practitioner perspectives that we feel is worth sharing with the wider community.

Barriers: What’s Getting in the Way

• Demand vacuum at the organizational level: Without a clear signal from donors or headquarters that spatial data is expected, it tends to get deprioritized in data collection planning. As one practitioner put it, "donors aren't demanding it... HQ isn't really demanding it... and local MEAL isn't demanding it either." This absence of top-down demand creates a structural problem that goes beyond individual capacity — as the same respondent observed, the demand for geospatial data "feels really hard to emerge naturally" in project-based organizations where the value of spatial data is rarely visible within the scope of a single program.
  
  
• Capacity gaps - it’s relatively easy to collect a GPS coordinate, but being able to use it effectively is tough: Most organizations we spoke with noted that adding GPS collection to an existing survey isn't the primary challenge, but being able to do something with that data is another story. One respondent noted the high barrier to entry for geospatial analysis, an “incredibly hard tipping point with GIS to get people over…” as well as a lack of understanding of what else might be missing…  “we’re collecting GPS data, but we don’t know if that’s the only thing that’s useful, what else do we need to be collecting.”
  
  
•  “Champion” dependency making progress fragile: Due to the highly technical nature of spatial data utilization and analysis, “champions,” or those with strong geospatial capacity or interest often drive forward the collection, adoption, and use of this data and technology at organizations surveyed. The risk becomes the moment when a program ends and this person moves on, or budgets are cut, or they simply find a new opportunity, that capacity is lost.  In our discussions, we heard that losing those individuals can often impact trajectory of how an individual team connects with spatial data, underscoring the need to formalize and institutionalize the collection of this data.
  
  
• Challenges with data use: Evaluators and implementors alike will be familiar with the struggle of getting data from the collection phase to be processed in a way that offers insight. Multiple respondents mentioned the challenge of decision makers and programs knowing what can be done with spatial data, but much less regarding how to use it for a GIE.
  
  
• Funding disruptions: For most organizations we spoke with, there was an undercurrent of ripple effects from the shutdown of USAID last year and a general ramp down of foreign assistance budgets globally. Areas like technology, and especially geospatial, are impacted during these times, and loss of programs and staff as a result of the cuts have led to losses of champions and diminished time and interest in things like communities of practice. This is where structuralization (see next section) of spatial data within an organization can help to mitigate some of this impact on budgets and provide clear demand for building back up capacities when programs are cut and staff time directed elsewhere.

Enablers: What Has Actually Worked and Why

• Making spatial data structural… and tied to reporting: As discussed in the previous section, success in scaled collection of spatial data needs to have a demand signal.  One organization we spoke with found success in tying collection to their reporting structure saying “It's part of the reporting cycle now... To report on these indicators, [programs] have to collect the spatial data at the same time... it's not a nice-to-have. It's part of the reporting process." This approach helps bridge the demand gap that is often so difficult to close, while also making scale more practical by building a structural component.

•  Cultivation of champions and communities of practice: We’ve noted the risk around the champion model, but they are also the most durable enablers of adoption as well, especially when they are actively cultivated. One organization we spoke with noted their success in seeing growth around collection and use of spatial data came from their ability to generate buy-in and cultivate champions within country teams. In one example, an entire monitoring system around WASH data was championed and built for all countries at the organization, allowing individual countries to see data across their portfolio as well as own and manage their own data. While the success of each country’s utilization of the system hinged on this, the existence of a system and an accompanying community of practice made it much easier for them to identify and cultivate these champions where gaps existed.

• Team ownership and visibility of their own data: Success has also been found in putting data in the hands of the programs who are generating it. Democratization of data has long been a trope and aspiration of the humanitarian sector, but it turns out it can also have a meaningful contribution to enabling better systematic collection of spatial data, allowing teams to “contribute to [something] in a visual way that ties what they’re doing to the overall picture.”  This obviously needs to be tied with proper champion cultivation and capacity building where it doesn’t exist, but it’s a heartening reminder that spatial data can and should be a participatory process.

• Structured process, tools and support around spatial data: Where organizations had seen meaningful progress, the right combination of tool selection, process guidance, and dedicated human support was usually behind it. Data can be collected in a one-off fashion for a GIE or another purpose, but for there to be a significant shift toward organization “readiness” for GIEs and broader geospatial analysis, having the right tools is key. Organizations mentioned end-to-end workflows like Esri ArcGIS tools and mWater/Solstice as convenient solutions that make several different elements of data collection, processing, analysis and visualization relatively easy.    

What Implementers Say They Need

• Planning and process guidance for GIEs: A key piece of feedback was that understanding where to start with planning for GIEs was an important need. While good data is one of the most important contributions from an implementing organization, one organization mentioned the need to step back, really think about how to budget for something like this, plan the right collection protocol, include examples of successful GIEs, and offer support to ensure that what is being collected is what is needed to do an evaluation. The GeoField use case papers and forthcoming textbook begin to address some of these points, but it also underscores the need to establish partnerships around evaluation to support planning and to start the planning itself as early as it makes sense within a program cycle.

• A digestible entry point to geospatial data and analysis: Folks from multiple organizations spoke to making resources available that are digestible, and designed with minimal functionality (simple visualization, layering of data) - to start to make obvious to users why it matters to them and their program. While the step-by-step analysis for preprocessing data for analysis, and methodological considerations for impact evaluation is absolutely going to be useful for whomever is doing the work on that front, mainstreaming geospatial technology is often tough because it has that conceptual barrier to entry previously mentioned.  For mapping in general, respondents (including Mercy Corps) have found that simple visualizations can go a very long way in communicating the value of having plottable, program information readily available. And for more complex topics like Earth observation and GIE, a 101 document or visual that touches on the key concepts while linking to the detailed resources behind it was mentioned as a potentially very valuable resource for program managers and decision-makers who need to make the case for a GIE internally.

• Partnerships: Partnerships are essential when it comes to many data science and geospatial problems, especially in the current funding environment. The GeoField program kickstarted this process, and it needs to continue. Implementers and researchers, evaluators and other sectoral experts need to continue to make linkages, find opportunities to collaborate, enter proposals together, and forge those bonds. Implementing organizations (especially international ones), often have an easier path to collecting spatial data through MEAL structures, but usually don’t have the capacity to conduct evaluations in-house.  This makes partnerships, alongside the planning and process guidance mentioned above all the more important.

• Peer exchange and cross organizational learning: In most of our conversations with peers, all sides felt energized by the connection points – sharing challenges and successes with each other, and we were able to do the same from the Mercy Corps end.  Focusing peer learning around geospatial is often difficult when it’s considered more as a niche technology among implementing organizations, but in talking with our peers, we also realized it’s ubiquitous to some degree – not every organization has a demand signal from headquarters for collecting and using spatial data, but every organization is at least experimenting with it, or has a program using coordinates of program location or assets to inform their work.  Forging these connections among technology actors in the humanitarian and development sector is not new – NetHope and others have been doing it for decades, but having an (even informal) network of those on various stages of geospatial integration into programs would be especially beneficial for making things like GIE feel more attainable for many organizations.

The conversations we had with peers across these four organizations were energizing, not because the path to GIE readiness is short, but because the people working on these problems are thoughtful, motivated, and eager to learn from each other. The barriers are real, but so are the models that work: sector-embedded champions, data tied to reporting requirements, communities of practice that keep field teams connected to a shared vision. What's missing most isn't technology, it's the connective tissue of partnerships, peer exchange, and accessible resources that help organizations take the next step, wherever they're starting from. The GeoField community is well positioned to build that connective tissue and this post is intended as an open invitation to that conversation. If your organization is working through any of these challenges, we'd love to hear from you!

‍