This report is made possible by the support of the American people through the United States Agency 
for International Development (USAID). The contents of this presentation are the sole responsibility of 
the DEPA-MERL consortium and do not necessarily reflect the views of USAID or the US government.
TABLE OF CONTENTS
LIST OF ABBREVIATIONS AND ACRONYMS 2
ACKNOWLEDGMENTS 2
CONSORTIUM INFORMATION 2
SUGGESTED CITATION 2
EXECUTIVE SUMMARY 3
INTRODUCTION 6
METHODOLOGY 7
THE DIGITAL STRATEGY DE 8
OVERVIEW OF PHASES OF THE DIGITAL STRATEGY DE 9
TRANSITIONING FROM WORKING 100% TO 50% ON THE DE 9
USE OF DE PRINCIPLES IN THE DIGITAL STRATEGY DE 11
ROLE OF THE DE ADMINISTRATOR IN THE DIGITAL STRATEGY DE 11
EMBEDDING REMOTELY 12
HIGHLIGHTS OF THE DE 13
SUSTAINABILITY OF THE DE 15
PILOT FINDINGS 16
WHAT ARE THE BARRIERS AND ENABLERS TO IMPLEMENTATION OF DE IN 
THE USAID CONTEXT? 16
WHAT DO KEY STAKEHOLDERS CONSIDER TO BE THE VALUE OF CONDUCTING 
A DE COMPARED TO A TRADITIONAL EVALUATION APPROACH? 30
CONCLUSIONS 33
REFERENCES 35
LIST OF ABBREVIATIONS AND ACRONYMS 
CDD Center for Digital Development
CLA Collaborating, Learning, and Adapting 
DE Developmental Evaluation
DEPA-MERL Developmental Evaluation Pilot Activity-Monitoring, Evaluation, Research, and Learning 
FY Fiscal Year
ITR Innovation, Technology, and Research 
LER Learning Evaluation and Research
PPL Policy, Planning and Learning
Search Search for Common Ground
USAID United States Agency for International Development 
WDI William Davidson Institute at the University of Michigan
ACKNOWLEDGMENTS
The Developmental Evaluation Pilot Activity-Monitoring, Evaluation, Research, and Learning (DEPA￾MERL) consortium would like to thank Laura Ahearn, the Digital Strategy Developmental Evaluator, for 
her enthusiastic support and dedication to Digital Strategy developmental evaluation. 
CONSORTIUM INFORMATION
The DEPA-MERL consortium consists of Social Impact, the prime awardee; Search for Common Ground;
and the William Davidson Institute at the University of Michigan. 
Consortium Contact: Danielle de Garcia, ddegarcia@socialimpact.com
USAID Contact: Sophia van der Bijl, svanderbijl@usaid.gov
SUGGESTED CITATION
The William Davidson Institute (WDI) at the University of Michigan. (2021). A study of the Digital Strategy
developmental evaluation: What factors help and hinder developmental evaluation success in USAID, and what 
is its value to stakeholders? Retrieved from: https://www.usaid.gov/PPL/MERLIN/DEPA-MERL
EXECUTIVE SUMMARY
PROGRAM BACKGROUND
The Developmental Evaluation Pilot Activity-Monitoring, Evaluation, Research, and Learning (DEPA￾MERL) consortium—situated in Policy, Planning and Learning (PPL) and Learning Evaluation and Research 
(LER)’s Monitoring, Evaluation, Research, and Learning Innovations Program at the United States Agency 
for International Development (USAID)—is testing the effectiveness of developmental evaluation (DE)
in the USAID context. DE was created to evaluate innovative programs that operate in complex 
environments and are thus expected to adapt over time. From May 2020 through May 2022, DEPA￾MERL conducted a DE of USAID’s Digital Strategy, which was launched in early 2020. USAID’s 
Innovation, Technology, and Research (ITR) Hub, formerly the US Global Development Lab, is 
responsible for leading the strategy implementation. The Digital Strategy is focused on aligning the 
Agency’s vision for development assistance with the world’s evolving digital landscape. From May 2020 
to June 2021, the DE supported four of the Digital Strategy’s 17 implementation initiatives by providing 
insights into how the initiatives work within their teams and with each other, and by sharing timely data 
and input for decision-making. The DE was then extended through May 2022, allowing the consortium 
to broaden the DE’s focus to work more across the Digital Strategy, in addition to focusing on a subset 
of initiatives. The extension also reduced the Developmental Evaluator’s time from 100% to 50%.
1 As a 
result of the COVID-19 pandemic, for the first time at USAID, the Developmental Evaluator was 
embedded remotely. As the USAID team implemented the Digital Strategy, the DE helped them learn 
by collecting and sharing data to inform strategy and operations. The DE also helped them collaborate,
as well as communicate the very ambitious and complex strategy to stakeholders.
EVALUATION BACKGROUND AND PURPOSE
DE is an evaluative approach aimed at facilitating continuous adaptation of interventions. In the context
of this project, it involves having one or more Developmental Evaluators integrated into the 
implementation team, usually on a full-time basis. This report seeks to facilitate learning on the 
implementation of DEs in the USAID context by sharing results from the Digital Strategy DE. The 
intended audience of this report includes USAID stakeholders, organizations funding or implementing 
DE, and Developmental Evaluators themselves. Using the information collected, the DEPA-MERL 
consortium aims to build on existing literature and offer readers targeted data and guidance to improve 
the effectiveness of DE. 
METHODOLOGY AND LIMITATIONS
During the entire duration (24 months) of the implementation of the Digital Strategy DE, the William 
Davidson Institute (WDI) team at the University of Michigan collected data to answer the following 
research questions:
• What are the barriers and enablers to implementation of DE in the USAID context?
1 The reduction of her effort was so she could continue to support the DE for a longer duration than if she stayed at 100% effort.
• What do key informants consider to be the value (added or lost) of conducting a DE compared 
to a traditional evaluation approach?
To answer these questions, the WDI team used mixed methods. The WDI team conducted a document 
review, held semi-structured interviews with the Developmental Evaluator and stakeholders, and
administered an electronic survey to stakeholders. Limitations of the study included selection bias and 
lack of a counterfactual. 
FINDINGS
Throughout the DEPA-MERL DEs, the top barriers and enablers were consistent.i,ii Given this was DEPA￾MERL’s fourth attempted DE, WDI investigated the top enablers and barriers across the DEs, which 
were also the top enablers and barriers in the Digital Strategy DE, to provide insights for future DE 
implementation within USAID. 
In the report, the WDI team shares examples from the Digital Strategy DE of barriers and enablers
associated with USAID dynamics, leadership, and skills of the Developmental Evaluator. The WDI team
also shares recommendations based on how the Developmental Evaluator overcame barriers or ways 
others can mitigate such barriers in the future at USAID. Key takeaways are listed below by each research 
question.
Data from interviews with the Developmental Evaluator and with stakeholders revealed the following:
Three main dynamics within USAID influenced the Digital Strategy DE: bureaucratic processes, lack 
of clarity of decision-making power, and social norms. A Developmental Evaluator’s skill set can help 
overcome a number of barriers created by these dynamics within USAID. Additionally, making 
person-to-person connections within USAID can help overcome barriers to DE created by 
bureaucratic processes.
USAID's learning-oriented and collaborative culture is a strong enabler of DE. At the same time,
there are norms (such as annual slowdowns, too many meetings, and limited trust of outsiders, 
among others) that can slow down DE activities, making it difficult for the Developmental 
Evaluator to build relationships and become embedded in the program. There are also
information preferences such as a preference for quantitative data (compared to qualitative data) 
that can serve as barriers to data sharing and utilization.
DE champions within USAID have the following characteristics: genuine openness to learning, 
collaboration, understanding, and willingness to convey USAID dynamics to the Developmental 
Evaluator, understanding the technical aspects of the DE, and investing in the success of DE within 
the USAID context. 
What are the barriers and enablers to implementation of DE in the USAID context?
A survey was distributed to stakeholders involved in the DE to assess its value. Based on the analysis, 
which included responses from five stakeholders, the WDI team found the following:
1. Overall, respondents reported positive interactions with the Developmental Evaluator. One 
hundred percent (n = 5) of respondents reported that they always felt comfortable sharing 
information with the Developmental Evaluator and that the Developmental Evaluator always 
understood the challenges they faced and always provided them with timely information.
2. Generally, survey respondents reported that the Digital Strategy DE was more valuable than a 
traditional evaluation. One hundred percent of respondents reported that DE was much better 
when compared to traditional evaluation as it relates to time savings (n = 4; one person did not 
respond to the question), providing timely feedback (n = 5), and facilitating adaptations (n = 5) to 
the program.
3. One hundred percent (n = 4) of respondents would like to see continued use of DE within 
USAID, and 100% (n = 4) of respondents would recommend the DE approach to other 
organizations.
CONCLUSION
The main goals of the DE were met. Those goals included helping those who were implementing the 
Digital Strategy learn as they went along, as well as collaborate and communicate across what was a very 
ambitious and complex strategy. The Developmental Evaluator accomplished this by working one-on￾one with initiative leads to inform innovations and strategic decision-making, as well as identifying and 
implementing strategic activities that cut across the initiatives. The Digital Strategy DE demonstrates that 
DEs can design and implement solutions that increase communication and coordination across USAID.
What do key informants consider to be the value (added or lost) of conducting a DE 
compared to a traditional evaluation approach?
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 6
INTRODUCTION
THE DEVELOPMENTAL EVALUATION PILOT ACTIVITY
Managers of programs being implemented in complex settings or with untested theories of change often 
face a challenge when trying to use a traditional midterm or end-term evaluation to assess the program’s
impact. In such programs, traditional evaluations2 may fail to provide useful information in a timely fashion 
or capture important outcomes not defined at the outset. To help address this issue, the United States 
Agency for International Development (USAID) funded the Developmental Evaluation Pilot Activity￾Monitoring, Evaluation, Research, and Learning (DEPA-MERL) program—a mechanism to pilot the use 
of developmental evaluation (DE) and assess its feasibility and effectiveness in the USAID context. DEPA￾MERL is an initiative under the PPL’s Monitoring, Evaluation, Research, and Learning Innovations Program 
and is implemented by Social Impact (prime awardee), Search for Common Ground (Search), and the 
William Davidson Institute (WDI) at the University of Michigan. 
DE is an evaluative approach aimed at facilitating continuous adaptation of interventions. In this context, 
it involves having one or more Developmental Evaluators integrated into the implementation team,
usually on a full-time basis. These Developmental Evaluators contribute to modifications in program 
design and targeted outcomes throughout implementation. They participate in team meetings; document 
decisions, processes, and dynamics; and collect and analyze data, feeding it back to the program on a 
regular basis. The DE approach is methodologically agnostic and utilization focused. Developmental 
Evaluators adjust research questions and methodological and analytic techniques as the project changes 
and deliver contextualized and emergent findings on an ongoing basis.
THE DEPA-MERL LEARNING AGENDA
The WDI team’s role in the DEPA-MERL consortium is to facilitate learning on the implementation of 
DEs in the USAID context. To accomplish this objective, the WDI team analyzed the Digital Strategy DE
during its 24-month duration, from May 2020 through May 2022. Using the data collected, the DEPA￾MERL consortium aims to build on existing literature focused on the practice of DE.
iii Readers of this 
report—including USAID stakeholders, other organizations implementing DE, and Developmental 
Evaluators themselves—can use the data and recommendations to strengthen their own use of this 
approach. Additionally, the findings from this study will be compared to findings from other pilots 
conducted by DEPA-MERL.
iv A comparative report is forthcoming in 2024.
FOCUS AREAS OF THIS REPORT
This report discusses evaluation results from the DEPA-MERL consortium’s learning agenda, led by the 
WDI team. This report includes an overview of the DE followed by a detailed analysis of barriers and 
enablers to DE implementation. Finally, the report shares stakeholders’ perceptions of the value of DE 
and concludes with recommendations. 
2
For the purposes of this study, “traditional evaluation” refers to any formative or summative evaluation approach. This includes evaluations that measure 
the difference between pre- and post-program activities on a subset of the population that receives a treatment or intervention, or “one-off” evaluations
that provide a snapshot or cross-sectional analysis of a program.
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 7
METHODOLOGY
The WDI team used mixed methods to understand the effectiveness of the DE approach, how it can be 
strengthened in practice, and what value it provides for its stakeholders within the USAID context.v
Table 1 lists the research questions developed by the DEPA-MERL consortium and the associated data 
collection methods. 
Table 1: DEPA-MERL Assessed the Digital Strategy DE Based on Two Research Questions Using a Mixed￾Methods Approach
Research Question Methods Data Sources
What are the barriers and enablers to implementation 
of DE in the USAID context?
Semi-structured 
interviews 
(qualitative) 
• Monthly reflection interviews with the 
Developmental Evaluator (20 monthly interviews)
• Interviews with keys stakeholders at end line (n =
3)
What do key stakeholders consider to be the value 
(added or lost) of conducting a DE compared to a 
traditional evaluation approach? 
Survey 
(quantitative and 
qualitative)
• Value of DE survey with stakeholders at end line (n
= 5)
The WDI team conducted over 23 hours of interviews and analyzed over 350 pages of qualitative data.
Outside of DEPA-MERL, research of this kind—a systematic evaluation of DE—has been conducted only 
in a handful of other instances.vi,vii,viii
METHODS AND DATA TREATMENT 
To answer the first research question, What are the barriers and enablers to implementation of DE in the 
USAID context?, WDI conducted monthly reflection and key stakeholder interviews. During the 
interviews, the WDI team asked open-ended questions about factors that the interviewees experienced 
or faced that were particular to the program and/or owing to the local context. The WDI team used 
NVivo software and conducted line-by-line coding3
of interview transcripts for barriers and enablers.
The WDI team ensured inter-coder reliability of 80%–90% on all codes or factors. Before the launch of 
the Digital Strategy DE, the WDI team conducted a literature review to identify 13 factors that could 
influence the implementation of DE (using a deductive approach). The WDI team also identified and 
added new factors by using an inductive approach to carefully review the incoming data.
To answer the second research question, What do key informants consider to be the value (added or lost) of 
conducting a DE compared to a traditional evaluation approach?, the WDI team distributed the Value of 
Developmental Evaluation Survey to 26 key stakeholders ahead of the DE Wrap-Up Learning Session, of 
which five responded fully or partially. The Developmental Evaluator identified stakeholders who should 
receive the survey based on their role in the Digital Strategy DE, with the objective of selecting persons 
3 Coding is an analytical process in which data, in qualitative form (such as interview transcripts), are categorized to facilitate analysis. See Miles et al. (2014).
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 8
from different initiatives and different levels of involvement within the Digital Strategy DE.4 The data 
were analyzed using Qualtrics and Microsoft Excel software.
LIMITATIONS OF THE STUDY 
The WDI team faced challenges in collecting data during the Digital Strategy DE. These are related to
the following:
• Selection bias: Because of resource constraints, the WDI team could not interview all DE 
stakeholders. Instead, when identifying interview participants, the team used purposive sampling 
to select a few stakeholders who had robust knowledge of the Digital Strategy DE. To reduce 
the possibility of selection bias, the WDI team worked with the Developmental Evaluator to co￾select individuals who would have both positive and negative perceptions of the Digital Strategy
DE. While perspectives from interview respondents are included in this report, the majority of 
the information in the report is from the Developmental Evaluator’s perspective. Whenever 
possible, the WDI team triangulated the Developmental Evaluator’s perspective with information 
shared in interviews with other key stakeholders.
• Lack of a counterfactual: There was no counterfactual available for this study. As an 
alternative, the WDI team triangulated data through verification from multiple sources and 
conducted ongoing data collection to reduce recall bias.
• Small survey sample size: A small number of individuals responded to the survey associated 
with our research question What do key informants consider to be the value (added or lost) of 
conducting a DE compared to a traditional evaluation approach? The small sample limits the external 
validity (i.e., generalizability) of the findings from the survey. 
THE DIGITAL STRATEGY DE
In early 2020, USAID launched its Digital Strategy to align the Agency’s vision for development assistance 
with the world’s evolving digital landscape. The Technology Division (first known as the Center for 
Digital Development (CDD)) within USAID’s Innovation, Technology, and Research (ITR) Hub, formerly
part of the US Global Development Lab, is responsible for leading the implementation of the Digital 
Strategy. From May 2020 to May 2022, DEPA-MERL conducted a DE of USAID’s Digital Strategy. Because 
of the COVID-19 pandemic, the Developmental Evaluator was embedded remotely.
Several factors made DE a good fit for the Digital Strategy:
• The nascent nature of some parts of the strategy, as well as the uncertain direction of particular 
aspects of the initiatives, provided an opportunity for embedded learning to guide 
implementation.
• The Digital Strategy’s need to adapt, test, and promote new programming, all while the digital 
landscape continues to quickly evolve.
4 The DEPA-MERL consortium validated this list of stakeholders, but it was not reviewed by any stakeholders themselves.
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 9
• The need to work across USAID, as well as with numerous initiatives to develop, refine, and 
implement the Digital Strategy. 
OVERVIEW OF PHASES OF THE DIGITAL STRATEGY DE5
From May 2020 to June 2021, the DE supported four of the Digital Strategy’s 17 implementation 
initiatives by collecting data to inform strategy, operations, and learning, as well as help the initiatives 
collaborate and communicate across what was a very ambitious and complex strategy. The four digital 
initiatives included Digital Ecosystem Fund, Digital Payments, Digital Skills, and Digital Literacy.
In July 2021, the DE was extended through May 2022, allowing the DE’s focus to be broadened further 
across the Digital Strategy, in addition to focusing on a subset of initiatives. The extension also reduced 
the Developmental Evaluator’s time from working 100% on the DE to working 50% on the DE. The 
reduction of her effort was so she could continue to support the DE for a longer duration than if she 
stayed at 100% effort. In both phases, the Developmental Evaluator was supported by the DE 
Administrator. The DE Administrator role was fulfilled by Social Impact staff.
The COVID-19 pandemic forced the DE transition to a fully virtual format. The most difficult aspect of 
the virtual environment was embedding remotely. Otherwise, the DE was able to adapt easily to the 
virtual work environment. For instance, instead of holding a multiday or multihour acculturation 
workshop, the Developmental Evaluator significantly shortened the acculturation workshop duration and 
used the technologies (such as using the white space of Google slides as a whiteboard for group work)
to allow attendees to engage seamlessly and work together virtually. She also helped the Digital Literacy 
initiative think through how to set up and facilitate the Digital Literacy working group meetings virtually 
so participants found the meetings useful and meaningful. 
TRANSITIONING FROM WORKING 100% TO 50% ON THE DE
In July 2021, the Developmental Evaluator shifted from working 100% to 50% on the DE. In this section,
details describing the Developmental Evaluator’s experience with this transition are shared to help 
inform other DEs that may be thinking about or planning to undergo a similar transition.
Scope of Work: The DE scope of work was revisited for the transition, and some of the objectives 
were made more general so the Developmental Evaluator could be approached by other teams and no 
longer be restricted to the original four digital initiatives. Originally, everyone thought it was a good idea,
but shortly after the transition began, the Developmental Evaluator was approached with a number of 
requests which required her to set more precise parameters related to the DE’s focus. By the third 
month of working on the DE 50% of the time, she wished that the scope only had included overarching 
or cross-cutting activities, as there was more than enough for her to do at that level, and at that point 
in time, she was only working with two of the four original initiatives. 
5
For a more comprehensive list of activities completed by the Digital Strategy DE, please see the final report written by the Developmental Evaluator 
entitled Digital Strategy Developmental Evaluation: Final Report, May 2022 which can be found on the DEPA-MERL 
webpage (https://www.usaid.gov/PPL/MERLIN/DEPA-MERL)
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 10
Delegating Time: Originally the Developmental Evaluator had thought about trying to delegate 
certain days of the week for the DE, but she found early on that such a format did not work, as it was 
difficult to contain her DE and non-DE work to specific days. She also had considered reserving mornings 
for the DE because of her other non-DE work commitments. However, she found limiting herself to 
working on the DE in the mornings was not realistic if she needed to attend the initiative lead’s biweekly 
meetings. 
During the first month of the transition, the Developmental Evaluator was able to accommodate the 
extra demands on her time from increased requests for DE activities, as the other 50% of her non-DE 
work had not ramped up yet. However, the following month, her non-DE work became demanding, and 
as a result, she reflected she may have worked more slowly on the DE because of when she had time to 
schedule work or meetings. Perhaps serendipitously the transition occurred near an annual slowdown 
at USAID (i.e., August), so although a lot of requests came in as she transitioned, because of the 
slowdown and the Developmental Evaluator’s own paid time off, new requests decreased during the 
second month of working on the DE 50% of the time.
Shifting Gears: In the sixth month of working 50% on the DE, the Developmental Evaluator
shared she was finding it increasingly challenging to shift gears between the DE and non-DE work, 
especially since her non-DE work was with ITR as a hub. She found herself really missing working full
time on the DE. It also made her reflect that if the DE had been scoped at 25% instead of 50% time, the 
DE would have likely fallen to the bottom of her to-do list for a good portion of the time, and she would
not have had enough concentrated time to make adequate progress on the DE.
Connection: The Developmental Evaluator was not able to be as proactive as she used to be 
when she was working full time on the DE regarding suggesting things or offering to do things for people. 
As a result, she felt less connected to people in the DE and the DE overall because she felt her allegiance 
and attention had been pulled toward the non-DE work. She reflected that when she worked full time 
on the DE and was immersed, things would occur to her or she’d remember to follow up – and although 
she could not point out moments where she had not done those things, she shared that it did feel 
different working 50% on the DE. 
Additionally, at the time of the transition, the Digital Strategy lead also began attending fewer meetings, 
so the Developmental Evaluator found because she also was attending fewer meetings due to working 
50% on the DE, she was less connected to the Digital Strategy lead as well. The Developmental Evaluator
found herself hearing information secondhand and thus felt she had lost some of the situational awareness 
and relationships that were more tangential to her core work on the DE. Because of events in her private 
life, the Developmental Evaluator also had to take a good amount of time off starting in month 8 of 
working 50% on the DE, which resulted in her feeling even less connected.
Given the Developmental Evaluator’s experience from working 100% and 50% on the DE, she would not 
recommend anyone starting a DE at 50% time. The complexity of learning about the Digital Strategy and 
the roles and responsibilities of those involved would have taken much longer if she had been only 50% 
at the start of the DE. However, extending the DE at 50% allowed the Developmental Evaluator to 
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 11
arrange for and then track the handover of learning and the uptake of various DE activities for longer 
than would otherwise have been possible. 
USE OF DE PRINCIPLES IN THE DIGITAL STRATEGY DE
The Developmental Evaluator did not consciously or regularly refer to the DE principles6
to inform the 
DE; rather, she reported that the principles were internalized into everything she did. The most 
frequently mentioned DE principles shared by the Developmental Evaluator during the monthly 
interviews included complexity perspective, co-creation, and systems thinking. For example, during the 
November 2020 monthly interview, she shared that as an anthropologist, she used systems thinking and 
considered complexity in how she approached everything. In February 2021, she reflected that the Digital 
Strategy initiatives operated within numerous systems (the Mission, headquarters, and implementing 
partners), which created varying levels of complexity. In May 2021, she described the Digital Strategy 
initiatives as a matrix within a complex bureaucracy. 
DE principles she referred to the least included developmental purpose, innovation niche, and evaluation 
rigor. In November 2020, she recognized that one of the main purposes of the DE was to support the 
Digital Strategy in innovating how USAID operates digitally. In May 2021, she shared that she had received 
positive feedback on the Key Issue Narrative Analysis for conducting a thorough, rigorous analysis. 
Interestingly, one of the DE stakeholders was not familiar with the DE principles until the Developmental 
Evaluator shared them with her during the eighth month of the DE. The USAID stakeholder was struck 
by the overlap between the Digital Strategy and DE principles. She suggested that the DE principles 
should be incorporated into future scopes of work. 
ROLE OF THE DE ADMINISTRATOR IN THE DIGITAL STRATEGY DE
The DE Administrator helped scope and staff the DE and provided administrative and technical support 
throughout the DE. Prior to the start of the DE, the DE Administrator worked with USAID to develop 
a preliminary scope of work and budget, as well as recruit the Developmental Evaluator. 
During the DE, the DE Administrator was responsible for managing the DE, such as overseeing the 
budget, contracts, timelines, and liaising with the funder. The DE Administrator also worked with the 
Developmental Evaluator to provide surge support as needed for DE technical tasks, such as qualitative 
coding and workshop facilitation. The DE Administrator was also available to fill any gaps in the 
Developmental Evaluator’s skills since it can be difficult for one person, even a seasoned Developmental 
Evaluator, to have all the skills needed to conduct a DE. For instance, during the DE, the DE 
Administrator identified another staff member at SI to provide technical support on the Digital 
Development Network Analysis on behalf of the Developmental Evaluator. 
Beyond offering technical support, the DE Administrator also held routine meetings with the 
Developmental Evaluator to serve as a sounding board for thinking through complex and emergent 
issues. The Developmental Evaluator also shared that the monthly interviews with the WDI team served 
6 The DE principles include developmental purpose, evaluation rigor, utilization focus, innovation niche, complexity perspective, systems thinking, cocreation,
and timely feedback
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 12
as a valuable opportunity for the Developmental Evaluator to step back, reflect on the DE, and see the 
bigger picture. Outside of these meetings, the Developmental Evaluator also reached out to the DE 
Administrator whenever she needed to get additional feedback and input on ideas and products. 
Additionally, the DE Administrator provided a managerial role in the DE, which included working with 
the USAID before the start of the DE to develop the preliminary scope of work and budget for the DE, 
as well as recruit the Developmental Evaluator.7 During the DE, the DE Administrator helped to ensure 
adherence to the budget, timelines, and contracts; conducted quality assurance; oversaw reporting to 
USAID; and oversaw conversations related to extending the DE. The DE Administrator was also available 
to serve as an intermediary to help problem solve any management challenges. 
EMBEDDING REMOTELY
Because of the COVID-19 pandemic, the Developmental Evaluator had to embed remotely. Since the 
DE began early in the pandemic, there were not many lessons learned at that point around how to work
effectively virtually or around embedding remotely; as a result, the Developmental Evaluator had to 
navigate embedding virtually with little guidance. With that stated, early on in the pandemic, everyone 
faced new challenges of adapting to a fully virtual environment. The Developmental Evaluator attributed 
the fact that she had a usaid.gov email address at the start of the DE to helping her embed quickly, as it 
helped people within USAID see her as an insider. Similar to previous DEs, as people became more 
familiar with the Developmental Evaluator, they increasingly pulled her into meetings and included her 
as a copied recipient on emails. Indeed, the Developmental Evaluator attributed the openness of key DE 
stakeholders to providing access to documents and meeting invitations early on to helping her embed. 
The Developmental Evaluator reflected that a challenge to embedding remotely was the loss of tacit 
knowledge as “there is no substitution for walking around an office.” To remedy this, she would join Zoom 
meetings early and stay after they ended to hear some of the informal conversations.8 Despite the 
Developmental Evaluator’s interest in getting to know DE stakeholders more deeply and having a desire 
to meet one-on-one, as she embedded, she was careful to take into consideration the Agency’s efforts 
to try to limit meetings since a lot of staffers were parents responsible for overseeing their children’s 
education during school shutdowns. One of the only communication challenges that arose was when the 
Developmental Evaluator became aware that a lot of communication was occurring through informal 
channels (e.g., Google chat) that she was not included in. Upon realizing this, the Developmental 
Evaluator requested that DE stakeholders do their best to convey relevant information to her discussed 
through Google chat. Overall, the Developmental Evaluator reflected that operating remotely was a 
barrier to the DE implementation at the start, but with time, as staff adjusted to the virtual environment, 
it no longer affected the DE negatively. The virtual setup did not serve as an enabler at any point in time.
7 DEPA-MERL’s Implementing Developmental Evaluation: A Practical Guide for Evaluators and Administrators includes step-by-step guidance for designing a DE 
(including developing a job description for a Developmental Evaluator and selection criteria to consider) while our Developmental Evaluation in Practice: Tips, 
Tools, and Templates handbook includes detailed guidance on how to recruit for DEs, including interview questions for this position.
8 This strategy and others for conducting remote DEs can be found in the Remote Developmental Evaluation Guide for Funders and Practitioners. 
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 13
HIGHLIGHTS OF THE DE9
Throughout the DE, the Developmental 
Evaluator captured, promoted, and enabled the 
utilization of emergent learnings to support the 
DE. Key activities and outputs of the DE are 
featured in Figure 1, which highlights what the 
Developmental Evaluator was most proud of 
each month. The list begins with the network 
maps, which were the first product the 
Developmental Evaluator created during the DE. 
The Developmental Evaluator met and 
interviewed a number of DE stakeholders to 
understand the different USAID staff and 
contractors involved in the Digital Strategy,
including which individuals were involved in 
multiple digital initiatives. The network maps 
were shared with key DE stakeholders, which 
helped users see how disconnected some 
initiatives were from others and which individuals 
were connected to multiple initiatives and thus 
might be informative Digital Strategy sources (but 
also may be stretched thin). The maps also were 
used to help orient new hires on the organization 
of the Digital Strategy initiatives. 
Another key output of the DE included the 
Connect & Reflect sessions. The Developmental 
Evaluator’s observations of how disconnected 
initiatives were from one another resulted in the 
recommendation for and then implementation of
weekly 30-minute Connect & Reflect sessions 
where Digital Strategy staff crowdsourced
solutions to a particular challenge or shared
more about their work to find synergies and 
encourage collaboration between the digital 
initiatives. The Connect & Reflect sessions 
increased communication and coordination 
across the Digital Strategy initiatives. They also
helped initiatives think about their relationship to 
one another. For example, the Digital Payments
initiative lead conducted an interactive session 
during one of the Connect & Reflect sessions in 
which attendees shared how their initiatives 
could intersect with Digital Payments or amplify 
what they were doing. The Developmental
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 14
Evaluator reported that “in eight minutes, a phenomenal amount of connections were made,” and given that 
a number of decision-makers were on the call, the session resulted in an offer to include questions 
related to Digital Payments in the Digital Ecosystem Country Assessments moving forward, something 
the initiative had been trying to do without much success prior to the call. Additionally, the digital 
initiative leads internalized some of the writing and brainstorming activities used in the Connect & Reflect 
sessions and used them in their own meetings.
Another key activity during the DE was a review of USAID Key Issue Narratives to better understand 
digital programming in Missions. The Developmental Evaluator developed a coding scheme and elicited 
support from the DE Administrator to complete the coding. The Developmental Evaluator presented 
and shared findings from the analysis with various USAID groups and used their feedback to inform 
additional analyses. Beyond serving the needs of the Digital Strategy, the Key Issue Narrative Analysis 
has been used by a number of other USAID entities from the National Security Council to the 
Administrator’s Office and other groups. For instance, the Administrator’s Office wanted to better 
understand the type of digital programming used in the Northern Triangle region of Honduras, El 
Salvador, and Guatemala. The Developmental Evaluator was able to quickly pull the relevant information
from her analysis. It was included in an informational document that the heads of the Digital Strategy
presented to the Vice President. The Developmental Evaluator also worked with key DE stakeholders 
to rewrite the Key Issue Narrative instructions for Fiscal Year (FY) 2021. She also conducted the Key 
Issue Narrative Analysis for FY 2021. She adapted the process based on learnings from the previous year. 
Changes included streamlining codes to be more targeted toward data utilization. The Developmental 
Evaluator shared the findings from the FY 2021 analysis at the end of the DE.
The final key activity of the DE was conducting a Digital Development Network Analysis of individuals 
across the Agency working in digital development. The total sample consisted of 105 people, who listed 
a total of 705 staff members, working groups, and external organizations related to digital development 
work, among whom there were 1,628 connections. The Developmental Evaluator held a learning session 
for people to comment on the analysis. It was also circulated to the rest of the Technology Division
beyond the Digital Strategy. The head of the Technology Division sent it out in an email that stated there 
are a lot of possible uses for the analysis. Indeed, the Digital Development Network Analysis identified 
and improved digital connections within the Digital Strategy and across the entire Agency. For example,
the analysis showed that the Technology Division staff was not as well connected to other 
USAID/Washington Bureaus or overseas Missions as would be necessary to implement the Digital 
Strategy successfully. The analysis helped illustrate something that the Agency likely was aware of: USAID 
Digital Strategy initiatives need to improve outreach to Missions and maintain engagement with other 
Bureau colleagues. The map was also helpful for assessing what gaps are created in the network when 
someone leaves.
Beyond these key activities, the initiative leads reported that their one-to-one meetings with the 
Developmental Evaluator were a valuable part of the DE. In the meetings, they discussed tasks they were 
working on together through the DE. The meetings provided initiative leads the opportunity to step 
9
For a more comprehensive list of activities completed by the Digital Strategy DE, please see the final report written by the Developmental Evaluator entitled 
Digital Strategy Developmental Evaluation: Final Report, May 2022 which can be found on the DEPA-MERL webpage (https://www.usaid.gov/PPL/MERLIN/DEPA￾MERL)
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 15
back and reflect, something that is rare across USAID and especially for those with many other 
responsibilities. The initiative leads found it helpful to be able to “think out loud” with the Developmental 
Evaluator, bounce ideas off of her, and hear her perspective, constructive criticism, and guidance. The 
leads cherished the meetings because they allowed them to dig into some of the strategic activities they 
wanted to do but did not have the time to focus on frequently.
Stakeholders of the DE reported that the main goals of the DE were met. Those goals included helping 
those who were implementing the Digital Strategy learn as they went along by collecting and sharing data 
to inform strategy and operations and to help them collaborate and communicate across what was a 
very ambitious and complex strategy. Some stakeholders also reported that the DE exceeded their 
expectations.
SUSTAINABILITY OF THE DE
During the Developmental Evaluator’s transition to working 50% on the DE, she started meeting more 
with the two individuals likely to take over her work once the DE ended—the Monitoring and Evaluation 
Advisor for the Digital Strategy, as well as the Monitoring and Evaluation Program Officer for the 
Technology Division. Additionally, the Developmental Evaluator conducted one-on-one meetings with 
initiative leads and a final learning session with key DE stakeholders to discuss accomplishments and next 
steps. 
As part of the transition, in September 2021, the Monitoring and Evaluation Advisor for the Digital 
Strategy took over as facilitator of the Connect & Reflect sessions. After the facilitator role shifted, over 
time, there was a small decrease in attendance at the sessions, but attendance was still strong at the end 
of the DE, with no plans to stop the sessions. During the final learning session, stakeholders noted they 
should rethink whether the audience for the sessions should only include initiative leads or a wider 
audience. Currently, about two thirds are from the Technology Division. 
Both the Monitoring and Evaluation Advisor for the Digital Strategy and the Monitoring and Evaluation 
Program Officer for the Technology Division expressed interest in continuing the Digital Development 
Network Analysis. The Monitoring and Evaluation Advisor for the Digital Strategy was interested in using 
it as an annual metric, while the Monitoring and Evaluation Program Officer for the Technology Division
was interested in conducting more participatory stakeholder mapping exercises in the future. A challenge 
to replicating the Digital Development Network Analysis in the future is that the Developmental 
Evaluator used Kumu (a platform for mapping people, systems, and concepts), which is not approved for 
use by USAID. Thus, access to the previous analyses will rely on the Developmental Evaluator keeping 
her Kumu account. During the final learning session, there was discussion about not using surveys to 
conduct the analysis in the future since response rates to surveys are low at USAID, and because 
contractors tend to analyze survey data, it can result in the data not being internalized as much as it 
could be by USAID staff since they don’t spend as much time with the data.
The Monitoring and Evaluation Program Officer for the Technology Division also expressed interest in 
continuing the Key Issue Narrative analyses. A challenge to this is that the previous analyses were 
conducted using Dedoose, a platform for analyzing qualitative and mixed-methods research. Dedoose is 
not approved for use within USAID, so the Developmental Evaluator will need to keep her paid account 
active for USAID staff to have continued access to the files. Given USAID staff’s inability to have their 
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 16
own Dedoose project or subscription account, the Developmental Evaluator discussed alternatives to 
using Dedoose to analyze the Key Issue Narratives in the future such as using a spreadsheet or holding
a hackathon. 
Additionally, to contribute to the sustainability of the DE, an American Association for the Advancement 
of Science & Technology Policy Fellow drafted a research and learning plan for the Digital Ecosystem
Fund team that included a number of activities the Developmental Evaluator started with the team. The 
plan organizes these activities into actions and an associated timeline with responsibilities. 
In an attempt to ensure USAID staff retain access to files she created throughout the DE, the 
Developmental Evaluator included links to the files in her final report. She also added a note to Google 
documents she owned that stated, “Please contact me at my Social Impact email address if you need 
something related to the document.” The Developmental Evaluator also resent the links to all the people 
who currently have access to the files to remind them they have access and let them know that they 
cannot relocate or move the files, but they can add another—essentially a shortcut on their own drive 
so it is somewhere more obvious to them if they need it in the future.
In her final report, the Developmental Evaluator also made the recommendation to hire or assign a full￾time permanent learning advisor either internal to USAID (preferred) or a contractor to assist with 
continuing the DE activities. This recommendation resulted from a recognition of the fact that the 
individuals mentioned above have full-time positions and the aforementioned DE activities are an 
additional responsibility they would need to take on, which could result in the de-prioritization of DE 
activities given how busy USAID staff already are.
PILOT FINDINGS
Findings and associated key takeaways are organized by the two research questions in this report. 
FINDINGS FOR RESEARCH QUESTION: WHAT ARE THE BARRIERS AND 
ENABLERS TO IMPLEMENTATION OF DE IN THE USAID CONTEXT?
The top two barriers our DEs have encountered are the effect of USAID dynamics and the influence of 
leadership on the DE. The top enabler across our DEs has been the skills of the Developmental Evaluator. 
Throughout the DEPA-MERL DEs, the top barriers and enablers were consistent. ix,x Given this was 
DEPA-MERL’s fourth attempted DE, we decided to further investigate the top enablers and barriers 
across the DEs (which were also the top enablers and barriers in the Digital Strategy DE) to answer this 
research question more deeply to provide insights for future DE implementation within USAID. Thus, in 
the following section, we share examples from the Digital Strategy DE of barriers and enablers associated 
with USAID dynamics, leadership, and skills of the Developmental Evaluator.
USAID DYNAMICS
To more deeply understand the effect of USAID dynamics on DE implementation, we conducted 
secondary coding of the “USAID dynamics” code. Three main themes emerged during secondary coding: 
lack of clarity of decision-making power, navigating bureaucratic processes, and social norms (see Figure 
2). 
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 17
Figure 2: USAID Dynamics Subcodes
Next, we provide examples of barriers and enablers within each of the three themes noted in Figure 2
associated with USAID dynamics. Based on how the Developmental Evaluator overcame barriers 
associated with each of these themes we also share recommendations or ways others can potentially 
mitigate such barriers in design or implementation of future DEs within USAID. Please note that since 
these recommendations are based on a single DE, we advise that readers do not overly generalize them, 
but rather carefully consider them based on the context in which they hope to or are currently 
implementing DE. Thus, we share these recommendations in an effort to further learning and improve 
DE design and implementation within USAID.
Effect of Lack of Clarity of Decision-Making Power on the DE
Early in the DE in August 2020, the Developmental Evaluator observed situations where there was a lack 
of clarity of roles, which created uncertainty and tension around decision-making. The Kumu network 
maps the Developmental Evaluator developed of the Digital Strategy showed that there were six or 
seven standing teams with a cross-cutting matrixed Digital Strategy set of initiatives. Seventeen of these 
initiatives were superimposed on top of the standing teams. In some initiatives, the initiative team lead’s 
supervisor was the only other direct-hire member, which caused structural ambiguity. Additionally, some 
of the Digital Strategy initiatives were involved in things that were already being done by one of the 
standing teams. As a result, some structural ambiguities and inefficiencies resulted in individuals, including 
initiative leads, feeling like they did not have the authority to make decisions. It also created a challenge 
of who a contractor should listen to within an initiative. When delays in decision-making affected the 
Lack of 
Clarity of 
Decision￾Making 
Power
Navigating 
Bureaucratic 
Processes
Social Norms
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 18
DE, it helped that the DE was designed to work with four Digital Strategy initiatives so that when there 
were decision-making delays in one initiative, the Developmental Evaluator could focus on the other 
initiatives.
The structural inefficiencies were further complicated by USAID’s complex staffing structure and use of 
a range of different types of contractors and consultants within the initiatives. Three of the initiatives
were staffed with only one full-time staff member, which required a heavy reliance on contractors.
Sometimes contractors from different implementing partners were assigned different pieces of the same 
overall project, leading to communication and collaboration challenges. The Developmental Evaluator
shared that there appeared to be a hierarchy of non-direct-hire staff with direct-hire Foreign Service
Officers at the top, followed by direct-hire general service civil servants, then full-time embedded 
institutional contractors, and finally non-embedded contractors and consultants. Although the 
contractors often lacked decision-making authority because of this hierarchy, they tended to be more 
knowledgeable because they spent more time on the initiatives given direct hires’ limited bandwidth. The 
wide use of contractors influenced the DE in subtle ways, such as the Developmental Evaluator’s work 
being put on hold when initiative leads needed to finish time-bound work with contractors. The 
Developmental Evaluator proposed potential solutions to help address the challenges of working with a 
large number of contractors in her First Quarter Reflections in November 2020. These suggestions 
included enabling frequent communication and coordinating with contractors from different 
implementing partners, as well as identifying opportunities for consistency across tasks, either by adding 
full-time staff or assigning a single contractor to an entire project whenever possible.
Recommendations: (1) When a DE is expected to work across a large program or 
the Agency, consider designing the DE to work across USAID teams/initiatives so
that when DE activities are affected by delays in decision-making, contractor 
timelines, or other bureaucratic processes, the DE can shift focus to another USAID 
team/initiative. (2) When a contractor’s work may affect the DE’s ability to conduct 
activities, enable frequent communication and coordination to avoid delays.
After identifying these challenges early in the DE, the Developmental Evaluator implemented two 
solutions, the Connect & Reflect sessions and the Digital Development Network Analysis. Despite these
solutions being put in place, the Developmental Evaluator and key DE stakeholders continued to note 
challenges resulting from structural tensions throughout the DE. In May 2021, the Developmental 
Evaluator noted that a solution was proposed to transfer oversight of the implementation of some of 
the Digital Strategy initiatives from the Digital Strategy leadership team to the Technology Division team 
leads, and by November 2021, she reported that some of the initiatives had in fact been returned to the 
managers for implementation and would no longer be under the Digital Strategy lead’s direct supervision 
or control. During an open topic Connect & Reflect in early 2022, however, a number of attendees were 
unaware of the fact that the management of the Digital Strategy initiatives had been reorganized. As a 
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 19
result, notes from the session were sent to management, and management led the next Connect & 
Reflect to explain the management changes. 
Other USAID dynamics also influenced the DE in subtle ways during the USAID Transformation as the 
Global Development Lab was dismantled. The division’s name changed from CDD to Innovation, 
Technology, and Research/Technology (ITR/T) to Technology Division, and during that time, a rescoping 
occurred of what counts as work for the Digital Strategy and what work falls outside that scope. Some 
staff members reported hearing rumors that they might have to compete for their existing jobs. They 
were told that no one would lose their jobs, but they didn’t know what their new positions or roles
would be, or if they would stay together. In the end, there were only a few changes in who reports to 
whom, but the uncertainty caused anxiety and required the revision of clearance processes, causing 
delays in the approval of some DE products.
Effect of Navigating Bureaucratic Processes on the DE
Conducting a DE within a bureaucratic organization presents a number of challenges to DE activities 
from collecting, sharing, and utilizing data to embedding the Developmental Evaluator. The 
Developmental Evaluator reflected on the complexity of the bureaucracy and how to mitigate its effect 
on the DE during an interview in May 2021: “It's so complex because there are multiple systems. In order to 
make any changes that have any real stick to them, there are multiple levels of this bureaucracy. Because it's a 
matrixed, sort of organization, on top of what was already a complex bureaucracy, it's just really complicated to 
make change. It happens slowly. Keeping that in mind and recognizing why things move slowly, that helps. Also 
having that perspective and knowing that there are other places to go in this large system where you might be 
able to tweak a lever or gain a new perspective is useful too.” Additionally, throughout the DE, the 
Developmental Evaluator kept the DE principles around complexity perspective and systems thinking 
front of mind to help remind her what she could influence especially within a bureaucracy.
One particular challenge to conducting this DE was the effect of bureaucratic processes on the ability to 
implement the DE principle of timely feedback. Many bureaucratic processes slowed down the ability to 
collect, share, and utilize data because the Developmental Evaluator and teams needed to follow USAID 
procedures and processes. Bureaucratic processes also slowed down the Key Issue Narrative Analysis
and the Digital Development Network Analysis. For instance, the Developmental Evaluator wanted to 
share the Key Issue Narrative data but since the data is derived from internal reporting, it required
permission from the Missions to share externally. Therefore, in the case of the Key Issue Narratives 
related to Digital Payments, the Developmental Evaluator pivoted to share the data internally through a 
basic map on the USAID internal website. However, although she had access to edit the website, security 
features prevented the creation of a map. In the end, they decided to seek permission from the Missions 
to post a small number of cases on a public-facing website. Despite the experience, the Developmental 
Evaluator emphasized that although bureaucratic processes can slow DE activities down, the information 
is still appreciated and useful when it is shared because being timely is relative in a bureaucratic context. 
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 20
She also reflected that timely feedback was not an issue in smaller events such as workshops with 
individual initiative leads, as she could quickly reach out for information, analyze the data, get feedback, 
and then share the data with the initiative leads. Throughout the DE, the Developmental Evaluator was 
creative in finding ways to gather and share feedback in a way that abided by but did not get bogged 
down by bureaucratic processes.
Recommendation: If timely feedback is critical to the success of the DE or the 
program, consider structuring the DE with small rather than large initiatives.
Bureaucratic procedures can not only slow down processes but also reduce the amount of data gathered. 
For instance, the same day the Developmental Evaluator shared the Digital Development Network 
Analysis survey with the Digital Sector Council, an Agency-wide notice came out reminding everyone of 
the survey clearance process. However, the Digital Strategy lead had gotten a fast track process approved
through the Paperwork Reduction Act the previous year, so requests would have to be turned around 
in a week rather than 6 to 7 months, but the fast track approval process had not yet been announced 
across the Agency. Unfortunately, whoever put out the Agency-wide reminder did not appear to be 
aware that this other process existed. As a result, the Digital Development Network Analysis went on 
hold as the Digital Strategy lead tried to coordinate with the Management Bureau. It took over 2 months 
to clear both the content of the survey and the overall content of the notice, which described the 
purpose of the survey. The survey was then included as a single link in a long daily notice—resulting in 
very low response rates—only seven additional survey responses. 
In the words of the Developmental Evaluator, “It was a complete bust, a waste of time. We could have been 
doing it earlier.” The Developmental Evaluator noted that she could have collected the required data via 
informational interviews with people working in digital initiatives to inform snowball sampling instead of 
waiting for the survey to be approved. In the end, the Developmental Evaluator also sent the survey to 
the Evaluation Interest Group; Collaborating, Learning, and Adapting (CLA) community practice; and 
two other working groups. That brought the total up to 69 responses, all of which identified other 
individuals working in digital development. The Developmental Evaluator then sent out individual emails 
and ended up with 119 responses in total (105 responses after the data was cleaned). 
Recommendation: DEs that consider using surveys across the Agency should 
advocate for a separate notice encouraging staff to complete the survey or have a 
backup method for data collection. 
People within USAID served as enablers in navigating USAID’s bureaucratic processes—see Box 1 for 
related recommendations
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 21
BOX 1: PEOPLE AS ENABLERS IN NAVIGATING USAID’S BUREAUCRATIC PROCESSES
Navigating the bureaucracy has its own set of challenges that can slow timelines and lead to an inability to move 
forward without the right connections. For example, the Developmental Evaluator shared that it was amazingly 
complex and opaque to figure out how to provide input on improving the instructions and guidance for Missions 
completing the Key Issue Narratives. However, in the end, the Developmental Evaluator, along with two other 
key DE stakeholders (the DE buyer and an initiative lead), figured out who to ask, and as a result, the 
Developmental Evaluator and others got access to rewrite the instructions. As the Developmental Evaluator
reflected on the experience, she shared, “It takes patience; it takes who you know; it takes banging your head against 
the wall.”
Recommendation: Developmental Evaluators can better navigate USAID’s bureaucratic 
processes through well-connected people who have Agency knowledge and clout. 
Indeed, one of the key enablers for the Developmental Evaluator to overcome several barriers associated with 
bureaucratic processes was connections with people. For example, a key recommendation she gave for 
improving the quality of the Key Issue Narrative data beyond rewriting the instructions was to reach out to 
Missions about their responses to the Key Issue Narratives so that they know someone is reading and using
the data they provide. The Developmental Evaluator shared, “That is my not-so-hidden agenda that I really am 
serious about. We need to let people know in Missions that this onerous bureaucratic, seemingly pointless task actually 
generates extremely valuable data. The only way we have any visibility over much of what goes on in Missions. Ironically, 
it's just so decentralized. There's no other place … how do you find out because money goes to Missions, then Missions 
program most of the money, and they go to implementing partners, and they report, but there's no way of centralizing 
all of the information.”
Recommendation: Making person-to-person connections within a large bureaucracy can 
go a long way, including improving the richness of data that DEs are able to collect.
Another source of pride for the Developmental Evaluator during the DE was contributing to changes in meeting 
norms. The remote work environment made it easy to stack five or six meetings in a row since accounting for 
time to physically transition attendees to meetings was not necessary. During the Connect & Reflect sessions,
staff shared ideas for ways to reduce the number of meetings. In parallel, parts of the broader Agency put in 
place a meeting-free week. As a result, there was recognition that they needed to work differently, and the time 
was ripe for change. As more people put in practice and advocated for conventions to reduce and improve 
meetings, the overall environment around meeting expectations shifted for the better.
Recommendation: Bureaucratic processes can be improved through collective action of 
people within the Agency. 
Together the above recommendations illustrate ways people can serve as enablers in navigating USAID’s 
bureaucratic processes.
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 22
USAID’s processes also influence the ability of the Developmental Evaluator to embed within teams. The 
process of obtaining basic facilities clearance to obtain a USAID email address or to get security clearance 
takes time. Luckily, the Developmental Evaluator still had her USAID email address from her previous 
engagement and was able to reach out to DE stakeholders immediately. 
Recommendation: When determining the DE start date, account for the time 
required for the Developmental Evaluator to obtain basic facilities clearance and/or 
security clearance. 
Before any new technology or platform can be used by USAID staff to collect, analyze, or share data, it 
has to be approved by USAID. During the DE, this approval process slowed down the adaptation to 
remote work, with Jamboard, a digital whiteboard platform that facilitates collaboration, not being 
approved until a number of months into working remotely because of the pandemic. 
Recommendation: Developmental Evaluators should use platforms and technologies 
for collecting, analyzing, and sharing data (including facilitating meetings) that are 
already approved by the Agency, as approval processes for new platforms take time.
This DE was unique as a national election occurred during it, which indirectly influenced the DE. Since 
USAID is a governmental organization, the election was brought up in every meeting in one way or 
another, even before the election happened, though participants were careful not to violate the Hatch 
Act, which prohibits expressions of partisan preferences. In addition, some initiative members were 
pulled away from their day-to-day responsibilities to develop write-ups for the current Administration. 
Changes in Administration require even more work. For instance, when Samantha Powers became the 
new USAID Administrator and given the Biden Administration’s interest in digital, the focus on digital 
was elevated across the Agency. Staff members were tasked to develop proposals for future large-scale 
digital work, which increased their workload.
Recommendation: Anticipate USAID staff may be distracted during an election year,
which can influence their commitment to the DE.
Effect of Social Norms on the DE
The Developmental Evaluator raised a number of social norms within USAID units or teams that 
influenced the DE. The Developmental Evaluator shared, “In the USAID context realizing that there are 
these cultures, I think they actually [are in] any context, there are these cultures or cultural practices, that I would 
prefer to call it. There are subcultural practices, and that may go against the larger one or sometimes may 
exacerbate them.” 
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 23
Part of what made the Digital Strategy a good fit for DE was that the culture within the Lab (before it 
was dismantled) overlapped with many DE principles and practices, such as adaptive management, rapid 
feedback, and co-creation. Additionally, the Principles for Digital Development overlapped with the DE 
principles. As one stakeholder described it, “USAID is an unusually, intellectually and emotionally safe place 
to work. …The DE can recognize or point out ‘maybe if you did something this way’ or ‘this is what I heard you 
saying maybe if you tried x, y, or z’ people are totally safe to accept that. There’s no scorekeeping other than as 
a collective, we’re doing better. …No one feels threatened by anybody else. That kind of fearlessness or trust or 
safety is hugely important for the DE to be successful.”
Recommendation: Ensure a learning-oriented and collaborative culture exists before 
starting a DE. 
Another enabling norm was that the Developmental Evaluator did not have any authority over anyone. 
Stakeholders reported this as being immensely helpful for her ability to communicate with people. It 
created a depowered evaluation dynamic since she wasn’t a supervisor or even part of their team, but 
rather, she was just someone to help get work done. 
Table 2 categorizes some of the norms that go against the largely positive culture for DE that exists 
within USAID. These norms were identified during secondary coding of the monthly interviews and were 
then triangulated in interviews with key DE stakeholders. Table 2 categorizes these norms by their 
effect on the DE from slowing down DE activities, to making it difficult to embed and build relationships, 
to creating barriers to sharing information. Each of these norms is explored in these three categories
followed by recommendations that could help reduce their effect on future DEs facing similar situations.
Table 2: Social Norms Within USAID That Can Negatively Influence DE
Norms That Can Slow DE 
Activities Down
Norms That Can Make It 
Difficult for the 
Developmental Evaluator to 
Build Relationships and 
Embed
Norms That Are Barriers to 
Sharing Information
◼ Annual slowdowns
◼ Practice of rescheduling 
meetings 
◼ Too many meetings 
◼ Emphasis on consensus 
and collaboration
◼ Evaluation fear
◼ Low trust of USAID 
outsiders
◼ Information preferences 
around trusting 
quantitative data more 
than qualitative data
Norms that can slow down DE activities include being consensus-based, having annual 
slowdowns, the practice of rescheduling meetings, and having too many meetings. People 
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 24
within USAID are generally open to collaborating, and the Agency tends to be a consensus-based 
organization, which can result in slow decision-making. Sometimes it results in going back and forth so 
much that decisions are never even made. Despite slowdowns, having a consensus-based foundation is a 
valuable asset for the DE and in particular for the co-creation principle.
Recommendation: Leverage USAID’s consensus-based culture for co-creation 
throughout the DE but also consider the timing needed for collaborative activities. 
The DE encountered two periods of slowdown across the Agency during the months of August and 
December. Initially, the Developmental Evaluator tried to schedule the acculturation workshops in 
August, which was challenging because many people were on vacation. Additionally, during the first year 
of the DE, the Developmental Evaluator took one week off during the December holiday period, which 
left her with a week of “really quiet time for work.” However, she noted that the DE timing and her 
productivity were affected more during the second December when some staff took the full month off 
and many took 2 or more weeks off. 
Recommendation: When planning DE activities, keep in mind that August and 
December are difficult months to engage staff but can be a good time to get ahead 
in planning and analyses that do not involve staff. 
As referenced earlier, the DE attempted to help reduce the meeting culture by limiting the number and 
duration of meetings to allow staff to have more time to do deep, uninterrupted work. One stakeholder 
we interviewed shared that he had 27.5 hours of meetings scheduled for the week, which was probably 
on the high end for USAID staff but not terribly so. Throughout the DE, the Developmental Evaluator
needed to delay activities or reschedule meetings when retreats (whether at the Digital Strategy level or 
because of the reorganization of the Bureau) or mandatory sessions occurred and required all involved 
staff to participate for large amounts of time. At the same time, staff members grew frustrated when 
leadership had too many meetings to attend and as a result did not attend meetings they were originally 
scheduled to participate in, which included Connect & Reflect sessions.
DE activities were also slowed down because of the common practice of rescheduling meetings. One of 
the key stakeholders interviewed explained that to create time to do deep work, if there is a day where 
he has one meeting scheduled and the prior 2 days he had a ton of meetings, he will reschedule that one 
meeting so he can have a day where he can fully focus on whatever he is behind on. 
Recommendation: Anticipate delays or reduced ability for USAID staff to engage 
deeply in DE activities because of USAID staff being pulled into many meetings or 
needing to reschedule meetings.
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 25
Norms that can make it difficult for the Developmental Evaluator to build relationships and 
embed include evaluation fear and low trust of outsiders. Monitoring, Evaluation, and Learning 
have been built into much of what USAID does, and as a result, there is a culture of measuring and 
evaluating all the Agency’s work to ensure it is effective and uses taxpayer dollars responsibly. However,
individuals can become fearful of evaluation if they feel like they are being judged. However, the 
Developmental Evaluator did not encounter much of this culture. Instead of using the title 
“Developmental Evaluator,” she used the title “Senior Learning Advisor” to preemptively make DE less 
threatening. As part of this effort, she also emphasized that a large part of DE is around learning and 
supporting adaptive management. She also leveraged DE champions within USAID, such as the DE buyer,
to help key stakeholders overcome fears that the DE could make them look bad. 
Recommendation: Consider using a different title than Developmental Evaluator 
and emphasizing the learning component of DE if evaluation fears are present in 
your key DE stakeholders. Leverage DE champions within USAID to help USAID 
staff overcome their fears of DE.
The Developmental Evaluator noted that she observed that USAID staff have a low trust of people 
outside of USAID. A stakeholder noted that part of this is because many people at USAID are reluctant 
to engage in anything additional such as the DE because they don’t have time. The Developmental 
Evaluator shared that it helps to demonstrate knowledge of how USAID works, as “there's a certain level 
of trust that is built when you can use the lingo or you know that certain things are impossible, or you can know 
who to talk to about things.” The Developmental Evaluator felt she was able to build trust with most people 
relatively quickly because she was seen as an insider, or a former insider, because she used to be in the 
Lab.
Recommendation: When hiring Developmental Evaluators for a USAID DE, look for 
candidates that are familiar with how USAID operates and can “speak their 
language,” or be sure to train your Developmental Evaluator on these topics before 
they begin interacting with USAID stakeholders. 
Information preferences can create barriers. The Developmental Evaluator found some USAID staff 
expressed opinions that quantitative findings are more valuable and rigorous than qualitative findings. 
However, there are many constructs on which you cannot collect quantitative data. 
Recommendation: When collecting data for a DE, Developmental Evaluators should 
consider using mixed methods to appeal to data preferences within USAID. When 
sharing qualitative data, Developmental Evaluators should emphasize the rigor of 
the methods used, trustworthiness of the data, and, when appropriate, why 
qualitative methods provided more value than quantitative methods in that 
particular case. 
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 26
The Developmental Evaluator also found that USAID staff prefer to receive data in small digestible pieces 
such as one pagers, checklists, and data visualizations. Many USAID staff shared that they do not have 
time to read reports. 
Recommendation: Developmental Evaluators should share data in short and easy￾to-digest formats in order to increase the likelihood of USAID staff engaging with 
and using the data.
LEADERSHIP
Leaders hold much potential to enable DE activities as DE champions, but at the same time, they also 
have the ability to serve as a barrier to DE and undercut DE activities. In past DEs, leadership has served 
as a top barrier. The Digital Strategy DE also from time to time encountered barriers to DE activities 
created by leadership. As one DE stakeholder shared, “When you are thinking about making broader changes 
or front office type of changes to implementation, you need leadership buy-in, and I think that could be a challenge 
sometimes.”
During the Digital Strategy DE, we found that leadership could influence the DE’s intended focus and 
timeliness, as well as the Developmental Evaluator’s contextual understanding in a negative way. Indeed, 
some of the work the Developmental Evaluator intended to do was put on hold because of changes in 
focus directed by leadership. For example, the Digital Skills initiative’s learning personas were put on 
hold when the Digital Strategy lead asked the initiative lead to focus on something else first. Nevertheless, 
by being responsive to shifts in priorities, the Developmental Evaluator was able to provide timely 
support for work that the Digital Strategy leadership deemed most important. Leadership engagement 
can also influence the Developmental Evaluator’s depth of contextual understanding. When the 
implementation lead of the Digital Strategy started to be pulled into higher-level meetings and began to 
prepare for her next posting, she became less involved in the day-to-day management of the Digital 
Strategy. This resulted in the Developmental Evaluator being less in tune with some of the Digital 
Strategy–wide issues. The Developmental Evaluator still had other sources to obtain contextual 
information from, but they were not comprehensive. 
Recommendation: When leadership creates barriers to DE, the Developmental 
Evaluator should assess the related risks to the DE and implement strategies to 
mitigate the negative effects.
Leadership can also be an enabler. In particular, individuals within USAID who serve as DE champions 
are major enablers of DE activities. Within the Digital Strategy DE, the two individuals who originally 
initiated the DE from USAID’s side held high leadership positions and were DE champions throughout 
the DE (see Box 2 for characteristics of Digital Strategy DE champions). The Developmental Evaluator
reflected that having a strong partner as the main point person was key to the DE’s success, as without 
their support, the DE activities likely would not have been as successful, even if there were other 
supporters of the work. The DE buyers were generous with their time and stayed involved throughout 
the DE through regular communication. Their involvement ranged from working with the Developmental 
Evaluator to apply lessons from workshops to the next iteration of workshops, to holding regular 
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 27
meetings with the Developmental Evaluator. With that being said, the Developmental Evaluator had 
other DE champions throughout the DE who also advocated for and supported DE activities. In fact, the 
Developmental Evaluator reflected that DE champions evolved throughout the DE. 
Although the DE buyers remained champions throughout the DE, new DE champions also emerged with 
time. For instance, the Digital Strategy initiative team leads became champions early in the DE, while the 
Digital Strategy leadership became champions a few months into the DE as they began to see value.
Through frequent meetings or communications, individuals typically begin to see the value of DE activities 
and then start to internalize and adapt some of the DE activities into their own work. For example, the 
Developmental Evaluator observed that staff members started using more interactive approaches, such 
as Jamboard, after becoming comfortable with them and seeing their value in Connect & Reflect sessions. 
Another example is that the Digital Ecosystem Fund initiative started to apply some of the writing 
activities the Developmental Evaluator used in the Connect & Reflect sessions to enhance their 
collaboration in other meetings.
Champions were also created indirectly during the DE. For example, the Developmental Evaluator
offered to serve as a sounding board for an individual working with Missions to co-create according to 
the Principles for Digital Development. The individual was quick to accept the offer, as she was already 
familiar with the DE from attending the Connect & Reflect sessions. The interactions with the 
Developmental Evaluator led to the individual becoming a major DE champion. 
Recommendation: The Developmental Evaluator should continue to develop 
champions for their cause. To do so, they should consider holding open meetings, 
seeking opportunities for connection, and maintaining open communication 
channels. This will help socialize DE and make it more accessible to a broader set of 
USAID stakeholders to create more DE champions throughout the lifetime of the 
DE.
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 28
BOX 2: DE CHAMPION CHARACTERISTICS
Secondary analysis found the following key characteristics of DE champions:
• Genuine openness to learning—DE champions have a curiosity and desire to learn, as well as a 
willingness to trust the DE process. They are not fearful of the evaluation label. Rather, they want input on 
whether there is anything they can change that would let them do their work better. 
• Collaborative—DE champions have a willingness to engage and problem solve. For instance, in this DE,
the DE buyers participated in after-action reviews to inform adaptive management of DE activities and
reviewed and iterated on network maps, in addition to other activities. Other DE champions arranged 
regular meetings with the Developmental Evaluator to bring her into their work. 
• Understand and convey USAID dynamics to the Developmental Evaluator—DE champions have 
knowledge of USAID stakeholders’ information preferences and how to best approach USAID stakeholders 
and convey this information to the Developmental Evaluator. 
• Understand the technical aspects of DE—DE champions understand the technical aspects of DE and 
can thus advocate for the methodology and limit misunderstandings on its purpose. For instance, during this 
DE, a DE champion alerted the Developmental Evaluator that there were discussions that a DE activity she 
was leading might be changed in a way that did not align with her vision. Thus, in this case, the DE champion 
advocated on the Developmental Evaluator’s behalf internally to leadership regarding the DE purpose.
Champions can also help DE stakeholders understand how to take advantage of the DE and best engage 
with it to create the most value for their work. 
• Invested in DE success—DE champions are generous in using their connections within USAID to 
advocate for DE, as well as to overcome barriers to DE. As expected, the DE buyers were invested in the 
DE succeeding and thus were proactive in soliciting feedback on the DE from key DE stakeholders and 
sharing relevant information with the Developmental Evaluator. DE champions also advocate for DE even 
when they are not involved. For example, the Developmental Evaluator shared that on the CLA Community 
of Practice listserv, which has 865 people on it, someone asked, “Does anybody have any good examples of 
adaptive management techniques?” Before the Developmental Evaluator could respond, the prior DEPA￾MERL Contracting Officer Representative responded that the individual should look at DE and shared 
relevant links. USAID staff also trust internal DE champions and thus feel comfortable expressing any 
concerns related to the DE. As mentioned earlier, DE champions helped people involved in the DE 
overcome their evaluation fears. 
Recommendation: Developmental Evaluators should target leaders and people within the DE 
who have some or all of the above characteristics to increase the likelihood of leadership serving 
as enablers rather than barriers to the DE.
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 29
SKILLS OF THE DEVELOPMENTAL EVALUATOR
Skills of the Developmental Evaluator have been a top 
enabler of DE implementation
.
xi
,xii This DE was no different
.
Two DE stakeholders mentioned the Developmental 
Evaluator’s skill set, experience, demeanor
, and approach 
as the largest enabler to the DE. A Developmental 
Evaluator’s skill
set can help overcome some of the 
barriers noted earlier related to USAID 
dynamics. For 
instance, it is important for a Developmental Evaluator to 
have the ability to build strong relationships with USAID 
staff to successfully facilitate use of DE data and insights. 
Thus, an understanding of USAID’s culture and an ability to 
gain staff members’ trust are critical Developmental 
Evaluator skills.
See Figure
3 for a list of skills used by the 
Developmental Evaluator during the Digital Strategy DE. 
The DE required that the Developmental Evaluator not 
only have a range of robust technical skills and knowledge 
but also a great deal of communication, social
, and 
personal skills. 
Verbal communication included teaching 
new skills to a variety of audiences
, such as how to make 
meetings or presentations more engaging
, as well as 
coaching and mentoring DE stakeholders. Indeed, 
together the Developmental Evaluator’s technical CLA
skills and verbal facilitation skills during a 10
-minute 
brainstorm session for the Key Issue Narrative Analysis
resulted in responses from participants such as “this is one 
of the most shockingly productive brainstorming sessions I’ve 
ever been in
.” And others said the following related to the 
Connect & Reflect sessions
:
“
You know, this is the most 
productive meeting all week, for sure,” and
“yeah, we
’re 
having all these two
-hour long retreat meetings
, and we don't 
really accomplish anything in those two hours. In 30 minutes, 
you can kind of crowdsource in the short amount of time, a 
huge amount of information.
” An important technical skill 
discussed earlier is familiarity with USAID, as the 
Developmental Evaluator emphasized
,
“
There’s a certain 
level of trust that is built when you can use the right lingo
, or 
you know certain things are impossible, or you can know who 
to talk to about things
.
”
On the social side
, the Developmental Evaluator was 
constantly balancing the priorities of multiple stakeholders 
since the DE required working with multiple digital 
initiatives. The DE also required a range of personal skills 
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 30
including a willingness to learn on the go since the DE ended up being entirely virtual, resulting in the 
Developmental Evaluator needing to test and apply new digital communication techniques to allow for 
effective and productive digital interactions. For instance, during the acculturation workshops in 
September 2020, the Developmental Evaluator taught teams how to use the white space around each 
Google slide as a virtual whiteboard by zooming in and out. She also taught them how to use virtual 
Post-It notes so they could brainstorm questions together. 
RECOMMENDATION: Hiring a skilled Developmental Evaluator (one who has strong 
technical, interpersonal, social, and communication skills) will be one of the most 
important activities to enable success of the DE.
FINDINGS FOR RESEARCH QUESTION: WHAT DO KEY STAKEHOLDERS 
CONSIDER TO BE THE VALUE (ADDED OR LOST) OF CONDUCTING A DE 
COMPARED TO A TRADITIONAL EVALUATION APPROACH?
The WDI team distributed the Value of Developmental Evaluation Survey to 26 key stakeholders ahead 
of the DE Wrap-Up Learning Session to assess the DE approach in the context of the Digital Strategy 
DE. Five individuals responded fully or partially.
INTERACTION WITH THE DEVELOPMENTAL EVALUATOR
The WDI team assessed the value of survey respondents’ interactions with the Developmental Evaluator
through a question with five subitems. Respondents could answer using a five-point scale that ranged 
from 1 (never) to 5 (always). Results are shown in Figure 4.
Overall, respondents reported positive interactions with the Developmental Evaluator. On all except 
two of the five subitems, respondents used the highest rating on the scale (i.e., always). Notably, none of
the respondents used about half the time, sometimes, or never in response to a subitem of this question. 
Figure 4: Survey Respondents Reported Positive Interactions With the Developmental Evaluator (n = 5)
5
4
5
4
5
1
1
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
I felt comfortable sharing information with the Senior Learning
Advisor
I received useful information from the Senior Learning Advisor
The Senior Learning Advisor understood the challenges I faced
The Senior Learning Advisor addressed the challenges I faced
The Senior Learning Advisor provided me with timely information
Always Most of the time About half the time Sometimes Never Not Applicable
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 31
DE COMPARED TO TRADITIONAL EVALUATION
The WDI team assessed how the Digital Strategy DE compared to traditional evaluation through a 
question with eight subitems. To rate the Digital Strategy DE, respondents used a five-point scale that 
ranged from 1 (much worse) to 5 (much better). Results are reported in Figure 5. Generally, survey 
respondents reported that the Digital Strategy DE was more valuable than a traditional evaluation. On
all but one of the eight subitems, more than 50% of the respondents reported that the Digital Strategy
DE was much better than a traditional evaluation. Across all subitems, none of the respondents said that 
the Digital Strategy DE was either somewhat worse or much worse compared to a traditional evaluation.
However, only two respondents answered the question about cost effectiveness. This could suggest that 
respondents did not feel like they had enough information to answer this question. One respondent who 
chose not to respond to this particular question stated later in the survey, “I don't know the cost, but it 
seems like a really good value to me. I'd be interested in knowing the ROI compared to the cost, but even without 
that information, I can pretty easily endorse the DE without any hesitation.” A sizable percentage of 
respondents (80%; n = 4) responded that the extent to which the DE allowed for evidence-based 
decision-making was much or somewhat better (compared to a traditional evaluation). 
Figure 5: Survey Respondents Perceived the Digital Strategy DE as More Valuable Than Traditional 
Evaluation (n = 5)
4
4
1
5
5
4
3
1
1
3
1
1
1
1
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Addressed the needs of my organization…
Was aware of complexities in the local environment...
Allowed for evidence-based decision-making...
Facilitated adaptations to the program...
Was timely in providing feedback...
Resulted in time savings...
Was able to uncover inefficiencies...
Was cost-effective...
Compared to traditional evaluation, the extent to which the 
Developmental Evaluation...
Much better compared to traditional evaluation Somewhat better
About the same Somewhat worse
Much worse compared to traditional evaluation
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 32
MOST VALUABLE ASPECTS OF THE DE
In response to the question, “In what ways was 
the DE most valuable?” respondents (n = 2) 
emphasized ways in which the Digital Strategy DE 
created value for their particular initiative. As 
one respondent shared, “Helping identify 
challenges and issues with the upskilling initiative and 
helping to come up with effective solutions.”
Additionally, another respondent commented 
that the most valuable aspect of the DE was 
making connections across the initiatives.
During the final learning session and interviews 
with key DE stakeholders, each initiative lead 
shared that their one-to-one meetings with the 
Developmental Evaluator were one of the most 
valuable parts of the DE. The meetings provided 
initiative leads the opportunity to overcome a 
paralysis of critical thinking that comes from 
being too busy, being overworked, and being 
underfunded. The initiative leads found it helpful 
to be able to think out loud with the 
Developmental Evaluator, bounce ideas off of 
her, and hear her perspective, constructive 
criticism, and guidance. 
LEAST VALUABLE ASPECTS OF THE DE
According to the respondents who answered the question (n = 2), the least valuable aspect of the DE
approach was knowing how to optimally use some of the cross-cutting DE activities like the Digital 
Development Network Analysis and the Key Issue Narrative Analysis. 
PERSPECTIVES ON USING DE AGAIN
Respondents were asked two separate closed-ended questions about whether they would recommend 
the DE approach. First, they were asked whether they would like to see DE continued at their own
organization. All four of the respondents who answered this question said yes. Three of these 
respondents had not participated in a DE until the Digital Strategy DE. 
The second question asked respondents whether they would recommend the DE approach to other
organizations. All four respondents (100%) said yes. 
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 33
CONCLUSIONS
Stakeholders of the DE reported that the main goals of the DE were met. Those goals included helping 
those who were implementing the Digital Strategy learn as they went along and helping them collaborate 
and communicate across what was a very ambitious and complex strategy. The Developmental Evaluator 
accomplished this by working one-on-one with initiative leads to inform innovations and strategic 
decision-making, as well as identifying and implementing strategic activities that cut across the initiatives 
such as the Key Issue Narrative, the Digital Development Network Analysis, and developing meeting 
norm culture changes. The Digital Strategy DE demonstrates that DEs can design and implement 
solutions that increase communication and coordination across USAID.
Investigating the top enablers and barriers across the DEPA-MERL DEs, which were also the top enablers 
and barriers in the Digital Strategy DE, led to the following high-level takeaways:
1. Three main dynamics within USAID influenced the Digital Strategy DE: bureaucratic processes,
lack of clarity of decision-making power, and social norms. A Developmental Evaluator’s skill set 
can help overcome a number of barriers created by these dynamics within USAID. 
Additionally, making person-to-person connections within USAID can help overcome barriers 
to DE created by bureaucratic processes.
2. USAID's learning-oriented and collaborative culture is a strong enabler of DE. At the same 
time, there are norms (such as annual slowdowns, too many meetings, and limited trust of 
outsiders, among others) that can slow down DE activities, making it difficult for the 
Developmental Evaluator to build relationships and become embedded in the program. There 
are also information preferences such as a preference for quantitative data (compared to 
qualitative data) that can serve as barriers to data sharing and utilization.
3. DE Champions within USAID have the following characteristics: genuine openness to learning, 
collaboration, understanding, and willingness to convey USAID dynamics to the 
Developmental Evaluator, understanding the technical aspects of the DE, and investing in the 
success of DE within the USAID context.
Based on the analysis of the Value of Developmental Evaluation Survey, which included responses from 
five stakeholders, the WDI team found the following:
4. Overall, respondents reported positive interactions with the Developmental Evaluator. One 
hundred percent (n = 5) of respondents reported that they always felt comfortable sharing 
information with the Developmental Evaluator and that the Developmental Evaluator always 
understood the challenges they faced and always provided them with timely information.
5. Generally, survey respondents reported that the Digital Strategy DE was more valuable than a 
traditional evaluation. One hundred percent of respondents reported that DE was much better 
when compared to traditional evaluation as it relates to time savings (n = 4; one person did not 
respond to the question), providing timely feedback (n = 5), and facilitating adaptations (n = 5) to 
the program.
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 34
6. One hundred percent (n = 4) of respondents would like to see continued use of DE within 
USAID, and 100% (n = 4) of respondents would recommend the DE approach to other 
organizations.
The learnings from the Digital Strategy DE support the continued use of DE within the USAID context 
and provide insights for those designing DEs within USAID to consider.
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 35
REFERENCES
i Baylor, R., Fatehi Y. K., & Esper, H. (2020). Advancing the use of developmental evaluation: Key questions 
answered during a multiyear study of developmental evaluations implemented with USAID. DEPA-MERL 
Consortium. USAID.
https://www.usaid.gov/sites/default/files/documents/Analysis_Across_Pilots_Report.pdf
ii Baylor, R., Fatehi Y. K., & Esper, H. (2019). A study of the sustained uptake developmental evaluation: How 
does developmental evaluation work in the USAID context, what factors help and hinder its success, and what is 
its value to stakeholders? https://dec.usaid.gov/
iii Lam, C. Y. (2016). A case study on a design-informed DE. QSpace. http://hdl.handle.net/1974/14929
iv Baylor, R., Fatehi Y. K., & Esper, H. (2020). Advancing the use of developmental evaluation: Key questions 
answered during a multiyear study of developmental evaluations implemented with USAID. DEPA-MERL 
Consortium. USAID.
https://www.usaid.gov/sites/default/files/documents/Analysis_Across_Pilots_Report.pdf
v Miles, M. B., Huberman, A. M., & Saldaña, J. (2014). Qualitative data analysis: A methods sourcebook. 
SAGE Publications.
vi Langlois, M., Blanchet-Cohen, N., & Beer, T. (2012). The art of the nudge: Five practices for 
developmental evaluators. The Canadian Journal of Program Evaluation, 27(2), 39. 
https://www.researchgate.net/publication/289246043_The_art_of_the_nudge_Five_practices_for_deve
lopmental_evaluators 
vii Patton, M. Q., McKegg, K., & Wehipeihana, N. (2016). DE exemplars: Principles in practice. The 
Guilford Press.
viii Lam, C. Y., & Shulha, L. M. (2015). Insights on using developmental evaluation for innovating: A case 
study on the cocreation of an innovative program. American Journal of Evaluation, 36(3), 358–374. 
https://doi.org/10.1177/1098214014542100
ix Baylor, R., Fatehi Y. K., & Esper, H. (2020). Advancing the use of developmental evaluation: Key questions 
answered during a multiyear study of developmental evaluations implemented with USAID. DEPA-MERL 
Consortium. USAID.
https://www.usaid.gov/sites/default/files/documents/Analysis_Across_Pilots_Report.pdf
x Baylor, R., Fatehi Y. K., & Esper, H. (2019). A study of the sustained uptake developmental evaluation: How 
does developmental evaluation work in the USAID context, what factors help and hinder its success, and what is 
its value to stakeholders? https://dec.usaid.gov/
A STUDY OF DEVELOPMENTAL EVALUATION USAID.GOV 36
xi Baylor, R., Fatehi Y. K., & Esper, H. (2020). Advancing the use of developmental evaluation: Key questions 
answered during a multiyear study of developmental evaluations implemented with USAID. DEPA-MERL 
Consortium. USAID.
https://www.usaid.gov/sites/default/files/documents/Analysis_Across_Pilots_Report.pdf
xii Baylor, R., Fatehi Y. K., & Esper, H. (2019). A study of the sustained uptake developmental evaluation: How 
does developmental evaluation work in the USAID context, what factors help and hinder its success, and what is 
its value to stakeholders? https://dec.usaid.gov/