DISCLAIMER: The views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. DIGITAL STRATEGY DEVELOPMENTAL EVALUATION Final Report, June 2022 Laura M. Ahearn, Ph.D. Developmental Evaluator and Learning Advisor Social Impact KETCIA ORILIUS, HAITIAN HEALTH WORKER; PHOTO CREDIT: DAVID ROCHKIND FOR USAID TABLE OF CONTENTS Introduction: What Was The Digital Strategy Developmental Evaluation? 1 What Did We Want To Learn, And How Did We Go About Learning It? 2 Digital Ecosystem Fund (DEF) Learning Questions And Activities 3 Digital Literacy Learning Questions And Activities 4 Digital Payments Learning Questions And Activities 4 Digital Skills Learning Questions And Activities 5 Cross-Cutting Learning Questions And Activities 6 What Did We Learn, And What Difference Did It Make? 9 Recommendations For Continued Learning Throughout The Implementation Of The Digital Strategy 9 TABLE OF FIGURES Figure 1: Michael Quinn Patton’s diagram of the eight DE principles................................................................... 1 Figure 2: Participatory Action Plan for Formulating DE Learning Questions..................................................... 2 Figure 3: Digital Strategy DE Highlights........................................................................................................................ 3 ACRONYMS AAR After Action Review CLA Collaborating, Learning, and Adapting COP Community of Practice DDA Digital Development Advisor DE Developmental Evaluation DECA Digital Ecosystem Country Assessment DEF Digital Ecosystem Fund DFS Digital Finance Systems DSC Digital Sector Council GEWE Gender Equality and Women’s Empowerment KIN Key Issue Narrative MEL Monitoring, Evaluation, and Learning OU Operating Unit PPR Performance Plan and Report USAID U.S. Agency for International Development 1 | DIGITAL STRATEGY DE FINAL REPORT USAID.GOV INTRODUCTION: WHAT WAS THE DIGITAL STRATEGY DEVELOPMENTAL EVALUATION? The developmental evaluation (DE) of the U.S. Agency for International Development (USAID) Digital Strategy sought to gather evidence on an ongoing basis to inform innovations and strategic decision making across four of the Strategy's 17 initiatives. The Digital Strategy DE also sought to explore opportunities for cross-cutting implementation of strategic efforts. As USAID’s Implementing Developmental Evaluation: A Practical Guide for Evaluators and Administrators notes, “DE is particularly useful in programs with untested or incomplete theories of change, where objectives may shift in response to contextual changes and where implementers and/or program managers are ‘building the plane in the air.’” The Digital Strategy implementation posed exactly this sort of challenge. The DE ran full-time from July 2020 to July 2021, at which point, the embedded evaluator switched to a half-time implementation schedule until May 2022. Due to the pandemic, the entire DE was conducted remotely. Similar to other DEs, at the helm of the DE’s efforts was a dedicated embedded evaluator. Unlike other DEs, this position was rebranded as a “Learning Advisor” to better convey the individual’s value-add to the Digital Strategy ecosystem and avoid any watchdog connotations associated with the term “embedded evaluator.” Figure 1: Michael Quinn Patton’s diagram of the eight DE principles Throughout the DE, the Learning Advisor was guided by the eight DE principles (see Figure 1), with different principles becoming more or less salient at various points throughout the DE. For example, “Developmental Purpose,” “Innovation Niche,” and “Co-creation” were most important at the outset of the DE as the Learning Advisor held acculturation workshops to explain the DE’s objectives, discuss how the change processes put forth by the Digital Strategy require innovation, and co-create learning questions and activities with the Digital Strategy initiative leads. Later in the DE as learning activities were implemented, “Rigorous Evaluation,” “Timely Feedback,” and “Utilization-focused” were central principles. Throughout the DE, “Complexity Lens” and “Systems Thinking” provided the Learning Advisor with essential perspectives on the Digital Strategy implementation. The DE supported four focal initiatives, one from each of the Digital Strategy’s four key tracks of implementation: 1. Track 1 – Adopt an Ecosystem Approach to Development in a Digital Age: a. Digital Ecosystem Fund (DEF) initiative USAID.GOV DIGITAL STRATEGY DE FINAL REPORT | 2 2. Track 2 – Help Partners Navigate Opportunity and Risk: a. Digital Literacy initiative 3. Track 3 – Shift to Digital by Default: a. Digital Payments initiative 4. Track 4 – Build the USAID of Tomorrow: a. Digital Skills initiative While there was a team constituted to implement the DEF initiative, the other three focal DE initiatives were implemented primarily by single individuals serving as initiative leads, with occasional support from other USAID staff members or contractors. Although the Digital Strategy is an agency-wide strategy intended to be implemented eventually by all USAID staff members, the primary responsibility for overseeing the implementation fell to the Technology Division within the Bureau for Development, Democracy, and Innovation’s Innovation, Technology, and Research Hub. Digital Strategy initiative leads took on their roles in addition to the other work assignments they had as members of various standing teams within the Technology Division. Challenges stemming from this matrixed management structure will be discussed further below. WHAT DID WE WANT TO LEARN, AND HOW DID WE GO ABOUT LEARNING IT? At the beginning of the DE, the Learning Advisor held two sets of two-hour “acculturation workshops” for each of the four focal Digital Strategy initiatives. The first of these sessions involved formulating learning questions. Participants brainstormed on virtual sticky notes the questions which would enable them to implement their initiatives more successfully if they knew the answers to them. The second acculturation workshop focused on creating a Participatory Action Plan for learning activities that would address the learning questions formulated in the first session. (See Figure 2 for a template of the table that the groups filled out together.) Figure 2: Participatory Action Plan for Formulating DE Learning Questions 3 | DIGITAL STRATEGY DE FINAL REPORT USAID.GOV Following the two workshop sessions, the Learning Advisor met individually with each initiative lead to co-create a DE Work Plan in which the Learning Advisor recorded activities addressing the learning questions of each of the initiatives as well as cross-cutting activities related to the Digital Strategy overall. A change log at the bottom of the work plan indicated new activities and shifts in direction. The Learning Advisor also created a DE Products, Adaptations, and Decisions Tracker listing 123 products and any adaptations or decisions they generated or informed. (See Figure 3 for the DE’s highlights.) Figure 3: Digital Strategy DE Highlights DEF LEARNING QUESTIONS AND ACTIVITIES According to its fact sheet, the DEF “equips the Agency’s Operating Units (OU) with catalytic financing to design and implement activities that foster open, inclusive, and secure digital ecosystems.” Although the DEF did not have a regular source of funding in the Digital Strategy’s initial years, the team was nevertheless able to administer two rounds of funding for USAID overseas Missions using repurposed funds. The first application round, which the team termed the Beta Round, had already taken place by the time the DE began, so the learning questions the DEF team formulated revolved around assessing the application process and preparing for future rounds: ● How did reviewers as well as selected and unselected applicants perceive the whole application/selection process? ● How have OUs with similar programs (such as Local Works, Financing Self-Reliance, Consortium for Elections and Political Process Strengthening, Development Innovation Ventures, Global Climate Change, etc.) engaged with Missions, both before and after awarding funds? ● What might motivate another OU to invest their funds in the DEF? ● How can the application review process and consultation / co-creation process with Missions be improved? To address these questions, the Learning Advisor facilitated an After Action Review (AAR) with the team following Round One and conducted focus group discussions with Foreign Service Officers to receive feedback on three scenarios the team had created for future funding rounds and to obtain insight on how Missions respond to calls for proposals from centrally managed funds such as the DEF. The Learning Advisor also surveyed Beta Round applicants and Round One external reviewers, then shared the findings with the teams. Toward the end of the DE, a new staff member joined the DEF team to help the team address future learning around the outcomes of the funded projects, which provided the Learning Advisor an USAID.GOV DIGITAL STRATEGY DE FINAL REPORT | 4 opportunity to encourage the creation of a DEF Research and Learning Plan that would ensure that the team continued to learn well beyond the end of the DE. DIGITAL LITERACY LEARNING QUESTIONS AND ACTIVITIES According to the Digital Strategy, the Digital Literacy initiative was designed to strengthen the digital capabilities of people participating in USAID’s programming: “USAID’s programming that includes digital technology regardless of sector or geography must include considerations around digital literacy in our design, from helping smallholder farmers track commodity prices on their mobile phones to helping newspapers operate on multimedia platforms.” The Digital Literacy initiative lead therefore prioritized a wide-ranging set of learning questions related to standing up an internal Digital Literacy Working Group and determining how to identify effective digital literacy practices: ● How can a working group best be utilized for digital literacy and how should we be strategically involving the working group? ● How do we deal with individual requests for technical assistance in the short run? How best can and should we support as an initiative? ● How can we best help Mission staff understand best practices in digital literacy? ● What’s the best way to measure the success of this initiative? To support the Digital Literacy initiative’s learning, the Learning Advisor co-created facilitation plans with the initiative lead for sessions with the Digital Literacy Working Group to determine the focus of the group, obtain feedback on USAID’s Digital Literacy Primer, and elicit ideas from the group about how to disseminate the Primer and develop shorter, more utilization-focused products from its contents. The Digital Literacy initiative lead found this support to be very helpful, noting in her final DE reflection, “The DE really helped establish a strong cadence for the working group and a meeting flow that made the meetings meaningful to join.” Other Digital Literacy learning activities included an interview with a Mission staff member who had worked with the Digital Literacy initiative lead, which informed discussions of how the initiative lead could best achieve the initiative’s goals, and a Connect & Reflect session to identify intersections between Digital Literacy and the other initiatives. The Learning Advisor also created a spreadsheet containing excerpts from the FY20 Performance Plan and Report (PPR) Key Issue Narrative (KIN) related to the subject of digital literacy. This analysis is described further in the section below describing cross-cutting learning activities. DIGITAL PAYMENTS LEARNING QUESTIONS AND ACTIVITIES The Digital Payments initiative seeks to make digital payments the default method of payment under all USAID-funded awards (with appropriate exceptions). The initiative’s learning questions therefore revolved around how best to accomplish this goal: ● What are the best ways to integrate digital payments into the Program Cycle and other Agency bureaucratic processes? ● What reactions do USAID’s Implementing Partners (IP)s and members of InterAction, the Professional Services Council, and other organizations have to the findings of the landscape assessment? What surprises them? How can USAID best use the findings to make it easier for IPs to use digital payments? ● How can Mission staff and IPs be encouraged to integrate digital payments across different sectors' programs? 5 | DIGITAL STRATEGY DE FINAL REPORT USAID.GOV To address these questions, the Learning Advisor analyzed a sample of solicitations and award documents to determine whether they contained boilerplate language requiring digital payments. The Learning Advisor also reviewed Digital Ecosystem Country Assessment (DECA) report recommendations to identify opportunities for follow-up to the DECA Missions from the Digital Payments initiative. The analysis resulted in DECA and Digital Payments staff members putting in place processes that facilitate better support in the future for Missions whose DECAs contain recommendations regarding Digital Payments programming. They also intentionally created a stronger relationship with the DECA team, which now reaches out to the Digital Finance Services (DFS) team to review DFS-related recommendations prior to publication, alerting the team to where future programming resources and attention should be directed. The Digital Payments initiative lead stated the following in the final DE learning session about connecting with the DECA initiative: “I would highlight this to all the initiative leads because we found it particularly useful…And now because we got some funding we’re trying to move some of the DECA recommendations forward with those Missions. That’s been an amazing experience.” In addition to these activities, the Learning Advisor also elicited information from Digital Development Advisors on the activities related to digital payments taking place in their Missions. And finally, the Learning Advisor identified excerpts from the FY20 PPR KINs related to the subject of digital payments, then created a MyUSAID webpage for the most interesting cases. This analysis is described further in the section below describing cross-cutting learning activities. DIGITAL SKILLS LEARNING QUESTIONS AND ACTIVITIES The Digital Skills (or “Upskilling”) initiative seeks to integrate skills in digital development across the technical and programmatic occupational categories of the USAID workforce. The initiative’s DE learning questions therefore revolved around assessing USAID staff members’ training needs and determining the most appropriate core competencies related to digital development: ● What are the digital skill levels and perceived needs of former Digital Development trainees, staff in DECA Missions, and staff in non-DECA Missions? ● How did the intake form and other aspects of the consulting process with initiative leads work as trainings were planned for those initiatives? ● How can information about trainings, the learning personas, the certification program, and other products best be shared? ● What feedback do participants have about the new Digital Development training? ● How can the new learning personas be validated and utilized? ● What are the most appropriate digital core competencies for Digital Development Advisor (DDA)s and other USAID staff members? To address the Digital Skills initiative’s learning questions, the Learning Advisor designed and administered a survey to USAID staff members who had attended one or more of the Digital Development trainings over the past five years. After sharing the findings of the survey, the Learning Advisor interviewed a subset of survey respondents to obtain more detailed information for the creation of learning personas that have subsequently informed the development of new trainings. The Learning Advisor also helped to formulate core competencies to be incorporated into Foreign Service Officer position descriptions. USAID.GOV DIGITAL STRATEGY DE FINAL REPORT | 6 CROSS-CUTTING LEARNING QUESTIONS AND ACTIVITIES The Learning Advisor also looked for opportunities to support learning across the Digital Strategy as a whole, especially during the second phase of the DE. As a result of discussing a “First Quarter Reflections” document with Digital Strategy leadership and the individual initiative leads, the following question was posed and guided cross-cutting learning activities for the remaining year and a half of the DE: ● How can collaboration be enhanced across the Digital Strategy initiatives and beyond? In addition to encouraging informal knowledge sharing across initiatives where appropriate, the Learning Advisor also conducted four learning activities to enhance collaboration across the Digital Strategy initiatives as well as between staff implementing the Digital Strategy and other USAID staff. 1. Connect & Reflect sessions. After the first few months of the DE, the Learning Advisor put forward a proposal in response to an observed gap in collaboration for five activities designed to improve communication, coordination, and collaboration across all the Digital Strategy initiatives. One of the five proposed activities, Connect & Reflect Fridays, became a central practice that contributed to all the Digital Strategy initiatives as well as to the other three cross￾cutting DE learning activities. Connect & Reflect sessions took place for 30 minutes every Friday morning and were facilitated mainly by Digital Strategy initiative leads but also by others at times, including Digital Strategy leadership, Monitoring, Evaluation, and Learning (MEL) Specialist, and the Learning Advisor. A sign-up sheet allowed individuals to choose a convenient date, indicate the topic of their session, share resources afterwards, and make note of any actions or decisions that were informed by their Connect & Reflect session. Connect & Reflect sessions were most commonly organized around status updates, requests for feedback, and brainstorming activities. The Learning Advisor emphasized that facilitating the sessions required very little preparation, and she encouraged facilitators to incorporate interactivity, writing, and reflection, often using Jamboards. After the first three months, the Learning Advisor held an AAR to determine whether Connect & Reflect sessions should continue and, if so, whether there should be any changes. Feedback from the participants was overwhelmingly positive. One representative Jamboard comment said, “The Connect and Reflect Fridays have been awesome in my book as it is a dedicated time (even if only for 30 minutes) to all come together and discuss various opportunities/challenges for each of our initiatives.” Another participant wrote, “I feel like these are one of the few opportunities for thought-partnership and substantive feedback.” Participants also made suggestions for changes in time and approach, many of which were implemented. In reflecting back on the DE, several of the focal initiative leads named Connect & Reflect sessions as one of the most valuable DE learning activities, with one participant noting the following: I think the Connect & Reflect was the most important cross-cutting activity. It gave all of the initiative leads an opportunity to benefit from group think on challenges or issues where they needed some fresh thinking and it was also a problem solving tool. For example, I benefited from the Connect and Reflect at a challenging time for the initiative. By presenting to the Connect and Reflect group–which was also made up of a lot of the folks that I needed to get feedback from–I heard from them on what needed to change and also got to bounce some ideas off of them on what we could do to improve the process. 7 | DIGITAL STRATEGY DE FINAL REPORT USAID.GOV Further information can be found in a USAID Learning Lab blog piece co-authored by the Digital Strategy MEL Specialist and the DE Learning Advisor, “Knowledge Lives in People: Using ‘Connect & Reflect’ to Learn from Each Other.” A few months before the end of the DE, the Learning Advisor handed over responsibility for coordinating Connect & Reflect sessions to the Digital Strategy MEL Specialist. 2. Analysis of annual PPR KINs To increase USAID staff members’ visibility of, and engagement with, digital development work taking place in Missions and other USAID/Washington OUs, the Learning Advisor, with the assistance of two other Social Impact staff members, conducted a qualitative analysis of four sets of the FY20 KINs: ○ Digital Technology KINs ○ Science & Innovation KINs ○ Cybersecurity KINs ○ Gender Equality and Women’s Empowerment (GEWE Primary) KINs The team coded the KINs according to themes that participants suggested in a Connect & Reflect session in January 2020 using Dedoose software. After the completion of the analysis, the Learning Advisor shared findings from the analysis at a Technology Division brown bag session in March 2021 and at a Digital Sector Council in April 2021. The Learning Advisor and Digital Strategy MEL Specialist also presented on the PPR process in general and the analysis of the KINs in particular at a May 2021 gathering of MEL specialists from the Latin America and Caribbean region. Despite the partial and imperfect nature of the KINs dataset, attendees at all these events expressed enthusiasm for the potential of KINs to shed light on Mission activities related to digital development, and they also suggested possibilities for more effective engagement and technical support to Missions around these programs. As Technology Division staff members became aware of the analysis, a number of them asked to see compilations of KINs excerpts related to specific topics or regions, such as the following: ● Artificial intelligence and machine learning ● Global (public) digital goods ● Humanitarian assistance ● Resilience ● Digital payments and digital financial services ● Digital literacy ● Africa ● Latin America ● Ukraine ● Philippines ● Indonesia ● Northern Triangle (El Salvador, Guatemala, and Honduras) -- These excerpts were incorporated into a brief by the Digital Strategy MEL Specialist and shared with the Office of the Vice President. USAID.GOV DIGITAL STRATEGY DE FINAL REPORT | 8 For two of the focal Digital Strategy initiatives – Digital Literacy and Digital Payments – the Learning Advisor compiled relevant KINs, then organized them into helpful resources for those initiatives. In the case of the Digital Literacy initiative, the Learning Advisor, with the support of a Social Impact staff member, further categorized the KINs excerpts according to types of digital literacy programs. For the Digital Payments KINs, the initiative lead chose the best examples, then obtained permission from Missions, after which the Learning Advisor posted them on an internal MyUSAID webpage. In order to elicit more detailed and relevant information in the FY21 KINs, the Learning Advisor collaborated with several Technology Division staff members to revise the instructions to Missions on how to complete the Digital Technology KINs and how to distinguish these KINs from the Science & Innovation KINs. As a result, the FY21 KINs were more relevant to the work of the Digital Strategy’s initiatives. In early 2022, during the DE’s extension period, the Learning Advisor conducted a streamlined analysis of three sets of FY21 KINs: Digital Technology, Science & Innovation, and Cybersecurity, with the goal of not only surfacing further insights about digital development programming across the Agency but also of establishing a process that could be implemented on an annual basis by Technology Division staff members themselves because there was a consensus among staff members that the KINs should be analyzed regularly. As one initiative lead noted, “The KINs were extremely helpful in understanding the breadth of tech-related activity.” Further information about the DE’s KINs analysis processes and findings can be found in the Analysis of KINs folder and in a USAID Learning Lab blog piece co-authored by the Digital Strategy MEL Specialist and DE Learning Advisor, “Learning from PPR KINs.” 3. Establishment of Division-wide meeting norms. In response to a Connect & Reflect session on the topic, and in collaboration with a Digital Frontiers staff member, the Learning Advisor convened gatherings to agree upon a set of meeting norms that would reduce the number of unnecessary meetings and increase the effectiveness of meetings that do take place. These gatherings resulted in long and short meeting norms documents for the Technology Division. The Learning Advisor and Digital Frontiers staff member presented these documents to the Division managers and at a brown bag session open to all staff members. While the documents have been included in onboarding materials for new hires, it is not clear how many changes have been made as a result of them. Anecdotal evidence indicates that some teams have set aside meeting-free time on a weekly basis for uninterrupted work, and some staff members have more frequently canceled unnecessary meetings. A Technology Division staff member also developed guidance for hybrid meetings. 4. Digital Development Network Analysis. Due to Digital Strategy initiative leads had limited visibility over which other USAID staff members were involved in digital development activities, the Learning Advisor facilitated a Connect & Reflect session in June 2021 to solicit feedback on a draft of a survey designed to collect data for a social network analysis. The Learning Advisor subsequently revised the survey according to the feedback and put it through a time-consuming clearance process to be approved for Agency-wide dissemination, which was granted in September 2021. To increase the survey response rate, the Learning Advisor circulated it on 9 | DIGITAL STRATEGY DE FINAL REPORT USAID.GOV various USAID listservs and used a snowball method to send individual requests to staff mentioned in others’ surveys – a process that could have commenced earlier had there not been an assumption that the Agency Notice would result in a high response rate. In the end, there were 119 survey respondents, who listed a total of 705 staff members, working groups, and external organizations related to digital development work, among whom there were 1,628 connections. The Learning Advisor, with the assistance of two Social Impact colleagues, analyzed the survey data using Kumu, a social network visualization platform for mapping systems and relationships, then shared the findings in a March 2022 brown bag session open to all Technology Division staff. The focus of the session was a “What? So what? Now what?” Jamboard exercise in which the Learning Advisor encouraged participants to identify actions they could take to strengthen or expand their connections to other individuals, working groups, and organizations involved in digital development. In an email following the session, one staff member wrote, “This is really cool stuff, Laura. I would definitely advocate for the Division and others across USAID to do more analyses like these.” The final section of the report contains recommendations for how to continue to learn from all four of these cross-cutting activities in ways that are sustainable given limited staffing. WHAT DID WE LEARN, AND WHAT DIFFERENCE DID IT MAKE? The DE helped to further embed a learning orientation into the Digital Strategy implementation, both for the four focal DE initiatives and for the Digital Strategy implementation overall. In the final individual learning sessions with the four focal initiative leads, all expressed an intention to continue to learn in ways that would improve the effectiveness of their initiatives. The DEF team, thanks to its new member, had even created a robust DEF Research and Learning Plan to help the team learn from DEF-funded programs. Initiative leads commented in the final learning session that one of the most valuable aspects of the DE was the opportunity the DE provided initiative leads to step back and reflect on a regular basis on the initiatives they were leading. Doing so allowed them to think and act more strategically and make important connections to other staff members and programs. As the Digital Literacy lead stated in that session, “One of the most valuable aspects [of the DE] was being able to work with [the Learning Advisor], bounce ideas off her, and think through overall strategy and direction with the initiative. The DE offered that space to reflect at a broader level than even our bimonthly or monthly meetings.” While many of the DE’s learning activities for the four focal initiatives informed the adaptations and decisions that the initiative leads subsequently made, it is fair to say that the DE’s cross-cutting activities will probably have the most lasting impact on the Digital Strategy implementation. The following session contains recommendations for how to sustain these cross-cutting activities and to incorporate a learning orientation even more thoroughly into the implementation of the Digital Strategy going forward. RECOMMENDATIONS FOR CONTINUED LEARNING THROUGHOUT THE IMPLEMENTATION OF THE DIGITAL STRATEGY Many of the DE’s regular practices will continue because of the staff members’ strong orientation to learning, but it will be a challenge to support some of the Learning Advisor’s activities given staffing shortages. The recommendations below nevertheless encourage continued efforts around learning and USAID.GOV DIGITAL STRATEGY DE FINAL REPORT | 10 make some suggestions for how to proceed with some of the learning activities in a more streamlined fashion. 1. Create a permanent Learning Advisor position, or at least explicitly assign portions of this role to one or more current staff members. The Digital Strategy and Technology Division’s MEL staff members have full workloads already, so perhaps the responsibility for coordinating Connect and Reflect sessions, or helping initiative leads formulate future learning questions and activities, or making sure future KINs are analyzed and shared should be distributed across multiple individuals. 2. Hold a midcourse stocktaking for the Digital Strategy. The retreat already scheduled for July 2022 can serve this purpose and provide an opportunity for initiative leads and managers to step back, reflect on what has gone well and not so well with the Strategy’s implementation, and make plans to adapt accordingly for the future. This will be especially important given recent changes in the management structure of the Digital Strategy, with many of the initiatives moved back under the aegis of the standing Technology Division team leads. 3. Harmonize MEL between the Digital Strategy and the Technology Division and focus on utilization. The Digital Strategy and the Technology Division’s MEL specialists have made good progress in revisiting and reconciling the two entities’ Results Frameworks, indicators, goals, and milestones in consultation with the initiative leads and teams. This is important work that should be accompanied by an emphasis on putting the data collected to use to inform decisions and future program designs. 4. Continue to improve communication and collaboration through Connect & Reflect and/or other practices. Coordinating Connect & Reflect each week is not a heavy lift, but it does require time and effort to ensure there is a leader for each session, and that session topics and formats are responsive to the needs of the participants. It may be time to step back and reflect on Connect & Reflect to determine whether any changes in the topics, the timing, the format, or the participants are called for. If Connect & Reflect is discontinued, other regular opportunities to communicate and collaborate across the Digital Strategy initiatives must take its place because there is a risk given the new management structure that the initiatives could become even more siloed. 5. Conduct a streamlined analysis of 3 sets of PPR KINs each year (Digital Technology, Science & Innovation, and Cybersecurity). In the final months of the DE, the Learning Advisor spoke with a number of staff members, including the Technology Division and Digital Strategy MEL Specialists, to explain the FY20 and FY21 KINs analysis process and drafted descriptions of the methodology for both years, with particular focus on the FY21 analysis since it was streamlined and therefore more likely to be implementable by current staff members. Having a contractor conduct the analysis is also a possibility, but there is greater benefit in having one or more Technology Division staff members conduct the analysis because the information about digital development programs in Missions and other USAID/Washington OUs then resides internal to the Division and can be more easily used to inform technical support and other decisions. The Learning Advisor discussed creative ways to crowdsource the analysis, such as holding one or more “KINs Hackathons,” perhaps following an initial quick pre-analysis by a contractor, so that staff members can internalize the content of the narratives. 6. Consider repeating or extending Digital Development Network Analysis. As part of her handover and sustainability activities, the Learning Advisor has discussed with the two MEL 11 | DIGITAL STRATEGY DE FINAL REPORT USAID.GOV Specialists the possibility of holding follow-up “What? So what? Now what?” sessions to discuss some of the Kumu maps from the analysis and decide how they should inform actions, such as outreach efforts to Missions or attempting to reestablish important connections when a staff member leaves. For the future, the analysis could be repeated or extended, or, perhaps more usefully, other more participatory approaches to stakeholder mapping could generate more action-oriented insights. 7. Recirculate the long and short meeting norms, and combine them with the new guidance for hybrid meetings. There are still many pain points around the frequency and effectiveness of meetings, so revisiting these norms could result in better and fewer meetings. 8. Tap more regularly into the potential of DDAs for mutually beneficial learning and coordination. The DDAs are potential gold mines of information about digital development activities in their Missions, which could be of great use to Technology Division staff members. And in turn, Technology Division staff members could provide DDAs with even more technical support than they already do. These exchanges could take place as part of the Digital Development Community of Practice proposed below. 9. Be more intentional about sequencing and strengthening the ties among DECA, DEF, and the other initiatives. In the next phase of the Digital Strategy’s implementation, it will be important to sequence and leverage more consistently the potential interconnections across all the initiatives, especially (but not exclusively) between DECA and DEF and the other initiatives. Because of the broad scope of these initiatives, there is great potential for sequencing activities around them and having them inform other initiatives. This could be accomplished by arranging cross-initiative conversations, appointing liaisons, or utilizing Connect & Reflect to identify opportunities to connect. 10. Acknowledge the challenges of a matrixed management structure and continue to work to mitigate confusion or tensions. Challenges around who has the authority to assign tasks to others, or how resources are distributed, or which activities get prioritized are inherent to matrixed organizations, but they need not derail the work. Frank conversations at the July retreat could help to surface issues that need to be addressed, and regular opportunities to communicate further should help to resolve them. 11. Create a Digital Development Community of Practice similar to the CLA COP. The DDAs could serve as the backbone to the COP, and Digital Sector Council (DSC) members should be invited to participate. The goal would be to expand the community beyond these two groups, however, to include others across the Agency who are involved or interested in digital development. Getting a COP up and running takes some effort, but it could be extremely valuable in helping to achieve the objectives of the Digital Strategy. This COP would differ from the existing monthly meetings of the DSC in being driven by peer-to-peer asynchronous interactions. 12. Develop effective onboarding and offboarding processes to help transfer tacit knowledge and important network connections. This resource on managing staffing transitions contains a lot of good suggestions, such as having outgoing staff members record short videos about different aspects of their job. USAID’s Knowledge Retention and Transfer Model is also an excellent comprehensive resource.