USAID Center for Education EDUCATION PERFORMANCE, IMPROVEMENT, COMMUNICATION, AND KNOWLEDGE (EPIC) Evaluation Report USAID Contract Number: GS-23F-8071H/AID-OAA-M-15-00018 Education Performance, Improvement, Communications, and Knowledge (EPIC) Implemented by: Training Resources Group, Inc. (TRG) 4301 Wilson Boulevard, Suite 400 Arlington, VA 22203 Phone: +1 703-875-8909 Fax: +1 703-875-9409 www.trg-inc.com Submitted: March 11, 2021 This publication was produced for review by the United States Agency for International Development. It was prepared by Training Resources Group, Inc. (TRG) and Social Solutions International. i | EPIC EVALUATION REPORT USAID.GOV TABLE OF CONTENTS ACRONYMS II EXECUTIVE SUMMARY 1 INTRODUCTION 13 BACKGROUND 13 EVALUATION METHODOLOGY 13 EVALUATION DESIGN 13 SAMPLING AND EVALUATION METHODS 16 ANALYSIS, INTERPRETATION, REPORTING, AND LEARNING 18 EVALUATION FINDINGS AND RECOMMENDATIONS 20 ORGANIZATIONAL EFFECTIVENESS 20 PROFESSIONAL DEVELOPMENT 50 KNOWLEDGE MANAGEMENT 71 ENGAGEMENT, COMMUNICATIONS, AND PARTNERSHIP 82 PROJECT WIDE RECOMMENDATIONS AND LESSONS LEARNED 95 CONCLUSION 101 ANNEXES 102 ANNEX 1: EVALUATION WORK PLAN 102 ANNEX 2: DATA COLLECTION MATRIX 105 ANNEX 3: DATA COLLECTION INSTRUMENTS 108 ANNEX 4: TIMELINE OF KEY EVENTS BY WORKSTREAM 109 ANNEX 5: FACTORS THAT FACILITATED AND HINDERED ACHIEVEMENT OF EPIC RESULTS 114 ANNEX 6: DETAILED ORGANIZATIONAL WELLNESS RESULTS (FEBRUARY, MAY, DECEMBER 2018) 118 ANNEX 7: LEADERSHIP DEVELOPMENT PROGRAM & PROFESSIONAL ENHANCEMENT SERIES EVALUATION RESULTS - LIFE OF PROJECT 119 ANNEX 8: PD ACTIVITIES AND PRODUCTS BY YEAR 120 ii | EPIC EVALUATION REPORT USAID.GOV ACRONYMS AAR After-Action Review ADS Automated Directives System AOR Alternate Contracting Officer Representative BE2 Building Evidence for Education CLA Collaborating, Learning, and Adapting COL Continuum of Learning COR Contracting Officer’s Representative COVID-19 Coronavirus Disease 2019 DDI Bureau for Democracy, Development, and Innovation DDI/EDU Bureau for Democracy, Development, and Innovation/Center for Education DEEP Data and Evidence for Education Programs E3 Bureau for Economic Growth, Education, and Environment ECCN Education in Crisis and Conflict Network ECP Engagement, Communications, and Partnerships ED Office of Education ED-CVE Education for Countering Violent Extremism EDTA Educational Technical Assistance Support EEPM Effective Education Portfolio Management EiCC Education in Crisis and Conflict EMT Executive Management Team EOC Education Overview Course EPIC Education Performance, Improvement, Communications, and Knowledge ESC Education Sector Council FSN Foreign Service National FSO Foreign Service Officer GBA Global Book Alliance GEC Global Education Conference GELS Global Education Learning Series GRN Global Reading Network HEYWFD Higher Education, Youth, Workforce Development ICT4E Information and Communication Technology for Education IR Intermediate Result KM Knowledge Management iii | EPIC EVALUATION REPORT USAID.GOV LDP Leadership Development Program LPA Bureau for Legislative and Public Affairs MBTI Myers-Briggs Type Indicator MCI Making Cents International M&E Monitoring and Evaluation MEL Monitoring, Evaluation, and Learning MSC Most Significant Change OD Organizational Development OE Organizational Effectiveness OU Operating Unit PAD Project Appraisal Document PD Professional Development PDWG Professional Development Working Group PES Professional Enhancement Series PMMELP Performance Management, Monitoring, Evaluation, and Learning Plan PMP Performance Monitoring Plan POP Period of Performance PYD Positive Youth Development QUEST Quality, Equity, and Sustainability Team SDD Session Design and Development SME Subject Matter Expert SMG Senior Management Group SO Strategic Objective SOTA State of the Art TCMST Transformation Change Management Support Team TDY Temporary Duty TRG Training Resources Group USAID United States Agency for International Development USG United States Government YWFD Youth Workforce Development 1 | EPIC EVALUATION REPORT USAID.GOV EXECUTIVE SUMMARY The Education Performance Improvement, Communications, and Knowledge (EPIC) contract was awarded on October 1, 2015 by the U.S. Agency for International Development’s (USAID) Office of Education (ED), within the Bureau for Economic Growth, Education, and Environment (E3). Through the last phase of implementation, the Office of Education transformed to become reorganized into the Bureau for Development, Democracy, and Innovation, Center for Education (DDI/EDU). EPIC’s contract structure supports collaborative and adaptive implementation across EPIC’s four workstreams: organizational effectiveness (OE), professional development (PD), knowledge management (KM), and engagement, communications, and partnerships (ECP), with the ultimate primary goal of transforming DDI/EDU and strengthening the USAID education ecosystem to be able to achieve USG’s Education Strategy goals and USAID’s Education Policy goals. EVALUATION PURPOSE This summative evaluation has two overarching objectives: 1. To document program achievements against the EPIC theory of change as well as expected results against objectives of the Strategic Results Framework; and 2. To identify best practices and lessons learned from the EPIC project. FIGURE 1: THEORY OF CHANGE Additionally, this evaluation synthesizes EPIC program data and experience related to the project’s learning agenda questions. The evaluation questions are presented in Box 1 followed by Table 1, which shows which evaluation questions contribute to answering the learning agenda questions. 2 | EPIC EVALUATION REPORT USAID.GOV Box 1: Evaluation Questions ● To what extent has EPIC contributed to DDI/EDU’s effectiveness and work quality, particularly as related to its four workstreams: organizational effectiveness; knowledge management; professional development; and engagement, communications, and partnerships? ● What unintended positive or negative outcomes, if any, emerged from EPIC’s support to DDI/EDU and other USAID Operating Units (OUs)? ● How did the project affect intended beneficiaries? What factors contributed to or hindered its success? ● Which tools and strategies were more effective in achieving the intended project objectives, and what were the contributing factors? ● Did workstream collaboration lead to better or extended benefits? In what ways? ● How did the project shift and adapt in approach, theory of change, and assumptions over time? TABLE 1: LEARNING AGENDA AND EVALUATION QUESTIONS LEARNING AGENDA QUESTIONS EVALUATION QUESTIONS Overall Project How did the design and structure of the EPIC project contribute to increased capacity of the USAID Education Office and sector to perform effectively? What kinds of changes lead to education strategy/policy goal achievement (though perhaps not directly attributable to EPIC)? To what extent has EPIC contributed to DDI/EDU’s effectiveness and work quality, particularly as related to its four workstreams: organizational effectiveness; knowledge management; professional development; and engagement, communications, and partnerships? What learnings, challenges-responses, success factors, frameworks and tools, or good principles and practices from EPIC can be used by other USAID OUs to improve their effectiveness? How did the design and structure of the EPIC project enable these things to be learned? How did the project affect intended beneficiaries? What factors contributed to or hindered its success? Which tools and strategies were more effective in achieving the intended project objectives, and what were the contributing factors? How does the confluence or integration of the workstreams lead to greater service to the USAID education sector, particularly implementation in partner countries? Did workstream collaboration lead to better or extended benefits? In what ways? How did the project shift and adapt in approach, theory of change, and assumptions over time? What were the steps, pivotal moments, shifts, breakthroughs/turning points, mistakes/misdirection, and key decisions over the journey? What was the lived experience? How did the project shift and adapt in approach, theory of change, and assumptions over time? 3 | EPIC EVALUATION REPORT USAID.GOV LEARNING AGENDA QUESTIONS EVALUATION QUESTIONS What unintended positive or negative outcomes, if any, emerged from EPIC’s support to DDI/EDU and other USAID OUs? Organizational Effectiveness What behavioral changes have occurred among USAID education sector staff as a result of EPIC activities? How have these behavioral changes contributed to effectiveness of individuals, teams, cross-teams, leadership, Office, and education sector as well as organizational culture/wellness? To what extent has EPIC contributed to DDI/EDU’s effectiveness and work quality? How did the project affect intended beneficiaries? What are the enabling factors that facilitate organizational effectiveness, and what does it take to bring them out? What are the deterring factors that are obstacles to organizational effectiveness, and how can they be addressed? Which tools and strategies were more effective in achieving the intended project objectives, and what were the contributing factors? What factors contributed to or hindered its success? Professional Development What are the most effective learning modalities needed throughout the career path of Education Officers? Which tools and strategies were more effective in achieving the intended project objectives, and what were the contributing factors? What are the most effective learning strategies to promote behavior and attitudinal change? Which tools and strategies were more effective in achieving the intended project objectives, and what were the contributing factors? What aspects of EPIC have helped or hindered field-led professional development and learning? What factors contributed to or hindered its success? What are the competencies that Education Officers need to succeed in their role? Knowledge Management Are knowledge products reaching intended users? If so, do they find them useful (defined as high-quality and high relevance)? Which tools and strategies were more effective in achieving the intended project objectives, and what were the contributing factors? 4 | EPIC EVALUATION REPORT USAID.GOV LEARNING AGENDA QUESTIONS EVALUATION QUESTIONS How did the project affect intended beneficiaries? What factors contributed to or hindered its success? What sources of knowledge are most useful (types of products, organizations, or people)? Which tools and strategies were more effective in achieving the intended project objectives, and what were the contributing factors? What are the most effective information formats? Which tools and strategies were more effective in achieving the intended project objectives, and what were the contributing factors? What are the most effective engagement channels? Which tools and strategies were more effective in achieving the intended project objectives, and what were the contributing factors? What are the most popular information topics? Engagement, Communications, and Partnerships How does the EPIC project contribute to increased capacity of DDI/EDU staff to become ambassadors and champions for USAID Education? To what extent has EPIC contributed to DDI/EDU’s effectiveness and work quality? What are the most effective information formats? Which tools and strategies were more effective in achieving the intended project objectives, and what were the contributing factors? What are the most effective engagement channels? Which tools and strategies were more effective in achieving the intended project objectives, and what were the contributing factors? What are the most popular information topics? METHODOLOGY The evaluation applied a mixed-method approach, gathering, consolidating, analyzing, and synthesizing both quantitative and qualitative data. Data included inputs collected since the beginning of the project, as well as information gathered from DDI/EDU staff. As appropriate, data were collected through individual interviews, group discussions, online surveys, and document reviews using an iterative process 5 | EPIC EVALUATION REPORT USAID.GOV to facilitate learning and reflection from key findings. Planning for the evaluation began in May 2020, interviews were conducted in June and July 2020, and the evaluation concluded in March 2021, with the submission of this evaluation report. Note on terminology: On November 16, 2020, the Office of Education officially transitioned to become the Center for Education under the new Bureau for Development, Democracy, and Innovation. While most of the project was implemented before the transformation took place, this evaluation report will refer to the Center for Education (DDI/EDU) for consistency. KEY PROJECT-LEVEL FINDINGS Implementation of the EPIC project occurred from October 1, 2015 and is due to end on March 30, 2021. During this period, there have been significant external conditions such as a transition to a new administration in 2017, the shift to the Journey to Self-Reliance policy, and challenges such as a government shutdown, a U.S. Government (USG) hiring freeze, and COVID-19 resulting in a pivot to virtual work. Internally, the DDI/EDU Center grew from approximately 25 to more than 40 staff, and senior and mid-level leadership changed multiple times. Additionally, USAID was in the midst of an Agency-wide reorganization. These factors have had both negative and positive impacts on project implementation, which are reflected in the findings. Since the successes and challenges of DDI/EDU are closely intertwined with the successes and challenges of EPIC, it is sometimes difficult to parse attribution, but to the extent possible, the full report attempts to make these distinctions. Table 2 presents a high-level summary of key results based on EPIC’s strategic results framework. More details along with representative interview quotes are presented in the body of this report. TABLE 2: KEY RESULTS AND RECOMMENDED PRACTICES BY STRATEGIC OBJECTIVE KEY RESULTS SO1: Strengthen DDI/EDU’s organizational culture, structure, systems, and processes EPIC implemented multiple interventions in collaboration with DDI/EDU that resulted in notable improvement in office culture over time from a “toxic,” “competitive,” and “dysfunctional” office climate to a more open, transparent, collaborative climate where there is more evident cross-team integration and collaboration. EPIC interventions most cited by DDI/EDU interview respondents as having contributed to this result include design and facilitation of all staff and off-site meetings (45 percent), organizational wellness and staff surveys with feedback mechanisms (45 percent), the Leadership Development Program (LDP) and Professional Enhancement Series (PES)(39 percent), coaching (33 percent), consulting/being a thought partner/informal conversations (18 percent), support to the Executive Management Team (EMT) (12 percent), and process and systems design (12 percent). Some DDI/EDU staff note that more still needs to be done, particularly in light of COVID-related challenges, but that the work environment is much improved from when EPIC started. Leadership capacity in general has been strengthened, although DDI/EDU respondents at different levels varied in their assessment of the degree to which this intermediate result (IR) was achieved. Of the five senior management interviewees, four stated that good progress had been made; one interviewee did not address this question. Of the five Team Leads interviewed, two said results were mixed, two did not address the question, and one did not believe this objective had been achieved. Technical Specialists were about equally split in stating 6 | EPIC EVALUATION REPORT USAID.GOV KEY RESULTS that the objective had been achieved or that results were mixed, with one person believing the objective had not been achieved. EPIC interventions that respondents cited as having contributed to leadership strengthening included: coaching (45 percent), consulting/being a thought partner/informal conversations (42 percent), LDP/PES (36 percent), organizational wellness and staff surveys (36 percent), support to EMT and their support to Team Leads (18 percent), Office-wide off-site meetings (18 percent), process and system design (15 percent), cross workstream technical meetings (15 percent), staff meetings (12 percent), work planning (12 percent), and strategic planning (12 percent). A few respondents noted that senior leaders were perceived as more open and transparent because they sought out and responded to staff feedback from the organizational wellness and staff surveys. Some leaders strengthened their skills in holding themselves and other team members accountable, while others have not yet been as successful in this regard. Better alignment of DDI/EDU’s purpose, Office-wide priorities, objectives, resources, plans, and activities has been an ongoing effort influenced by the transition in the strategy and policy as well as growth of the Office. Sixty-six percent of DDI/EDU interview respondents stated that the Center has made notable progress on this intermediate result with EPIC’s support, 21 percent did not specifically address this question, and 13 percent believed that results were mixed, with progress in some but not all areas. The current education policy is broader and more flexible, allowing for greater collaboration, but also requires greater attention to setting Center priorities within that broader scope. In collaboration with DDI/EDU, EPIC designed and facilitated an intentional mix of ongoing annual off-site retreats, monthly EMT Strategy Sessions, monthly all-hands meetings, quarterly town halls, and annual work planning sessions to help the Center come together to set priorities and strengthen alignment. Some interview respondents indicated that more attention needs to be paid to using approved work plans as operational tools and as a means to hold teams accountable for progress. This is a particularly important recommendation to the Center as it embarks on a new and iterative planning process and seeks structured moments for reflection, learning, feedback, adjustments, and replanning over time. EPIC integrated collaborating, learning, and adapting (CLA) into everything it did in its work with DDI/EDU and intentionally learned with DDI/EDU from their shared experiences through regular after-action reviews, pause￾and-reflect sessions, and ongoing data gathering and consultation. However, EPIC did not have an explicit, documented CLA strategy, which could have been helpful as a tool to articulate their approach to their work with the Center and to measure progress in this area. SO2: Improve USAID education staff’s technical capability and skills to achieve the Agency’s education sector goals. Forty-two percent of DDI/EDU interview respondents stated that EPIC had contributed to improvement of USAID Education staff’s technical capability and skills, 50 percent believed that progress had been made but results were mixed with less progress on institutionalizing a quality continuum of learning (COL) and generally more progress on strengthening core competencies with notable improvement over the last year, one person did not believe that this objective had been achieved, and one person did not comment on this. EPIC has contributed to DDI/EDU’s progress on institutionalizing a quality continuum of learning strategy that reaches all actors in the USAID education ecosystem by conducting a needs assessment and meta-analysis of existing resources; making the decision to use consultants to contribute to deliverables; and collaborating with DDI/EDU on the development of specific core courses as well as additional events, webinars, and blogs. Training design has benefitted from more collaboration with Regional Bureaus and from Mission input. However, there have been a number of fits, starts, and missed opportunities in this workstream due to transitions in USAID’s activity leads and EPIC staff as well as shifting DDI/EDU priorities, and a lack of agreement on a core set of competencies for the sector and a comprehensive PD strategy. Strengthening core competencies of DDI/EDU and education field staff to design, manage, and evaluate programs proved to be challenging due to the Office not having up-to-date core competencies articulated and 7 | EPIC EVALUATION REPORT USAID.GOV KEY RESULTS agreed upon. EPIC tried to clarify which competencies the Office wanted to use, but there appeared to be more pressing priorities. EPIC collaborated with DDI/EDU to update and support the delivery of the Education Overview Course (EOC) and the Effective Education Portfolio Management (EEPM) in-person courses, which were of high quality and highly rated by participants. The course designs and deliveries improved over time due to EPIC’s intentional CLA approach, which integrated feedback from course evaluations and after-action reviews into the design of the next iteration of the course. EPIC’s pivot to the online Global Education Learning Series (GELS) sparked by COVID-19 ignited optimism about the positive momentum of this series, which has the potential to grow into a more comprehensive COL strategy. One early indicator of this success is that in the first three months, 206 individuals have already attended a course offered in concert with the GELS, which is more than the sum of all in-person course participants over the life of EPIC (175). SO 3: Create a sustained, global system of sector-wide learning and knowledge exchange. DDI/EDU and Missions have improved access to the latest and most relevant education resources as a result of the joint efforts of the EPIC team and DDI/EDU. Critical to access was the creation, launch, and maintenance of the EducationLinks external platform under the leadership of the project Contracting Officer’s Representative (COR) and KM Portfolio Manager with EPIC’s support to design, build and launch the website. Since EducationLinks was migrated from the Education Toolkit in Year 3 (September 2018), users have praised the user-friendly platform that enables them to easily find information of interest via blogs, resources, announcements, events, and toolkits. The creation and establishment of EducationLinks as a “go to” hub has been a game changer. EPIC contributed to progress in the capture and synthesis of relevant evidence and learning regarding gender, disability, information, and communication technology for education (ICT4E), and education finance programs by collaborating on the creation of knowledge products for each of these topics. Both the volume of Center for Education knowledge products disseminated, and the usage of these products suggest an increased visibility of the Office through EducationLinks. The 2020 EducationLinks Analytics Summary reported that overall site performance metrics were “strong and increasing” with 161 percent increase from 2019 to 2020 in the number of sessions (120,230), a 132 percent increase in pageviews (194,165), a 144 percent increase in users (79,416) and a 4.5 percent decrease in bounce rate (54.2 percent). The average session duration was 1 minute and 55 seconds. Email traffic increased 413 percent from 2019 to 2020. Traffic from LinkedIn increased 2,000 percent and Twitter traffic increased 413 percent from 2019 to 2020. As a result of KM collaboration with other workstreams and mechanisms, information was shared across teams and products were developed in an iterative process that focused on feedback and effectiveness. In collaboration with DDI/EDU, EPIC’s KM team developed templates and guidance on how to create KM materials, which contributed to a unified, coordinated Office. EPIC intentionally built trusting relationships with other mechanisms, which resulted in a successful launch of ECCN’s microsite and socialization of Data and Evidence for Education Programs (DEEP) Indicator Report Guidance. This collaboration created a more practical, engaging, and modern learning approach to support the effective work of USAID Mission and implementing partner staff. SO 4: Communicate DDI/EDU’s work and support more effective engagement with USAID/Washington, Missions, other USG offices, global stakeholders, and partners. Understanding of DDI/EDU’s brand and messaging was increased as a result of EPIC’s support through creation of standard templates, newsletters, and communications products that DDI/EDU staff could use during 8 | EPIC EVALUATION REPORT USAID.GOV KEY RESULTS temporary duty (TDY) and engaging with multiple stakeholders (e.g., in the development of the GEC and GELS). Interview respondents suggested that engagement and communications activities would have benefited from a more in-depth stakeholder analysis and a comprehensive Center-wide engagement and communications strategy. Many DDI/EDU staff that were interviewed commented that they were better able to advocate for education sector work as a result of the many KM and communications products and communications training that EPIC supported, as well as being able to refer people to EducationLinks. Many of these activities were led by other workstreams that coordinated with ECP. EPIC offered communications training and follow-up coaching to DDI/EDU staff. Many staff either did not know about this opportunity or did not take advantage of it. As a result, only 11 people attended these coaching sessions, but they rated the sessions as very beneficial. RECOMMENDATIONS AND LESSONS LEARNED The summary of lessons learned and recommendations presented in this Executive Summary are cross￾cutting and project wide. There are many other lessons learned within each individual workstream presented in the body of this report. EPIC’S PROJECT DESIGN WORKED WELL AND SHOULD CONTINUE, WITH A FEW ADJUSTMENTS. Twenty-two of the 24 DDI/EDU interview respondents (92 percent) said they would not change the four workstreams as articulated in EPIC’s theory of change. They saw the synergistic and interconnected value of these four workstreams. Thirty-three percent of respondents commented that it is a large scope for any one contract. Five respondents believed that the overall goal articulated in EPIC’s theory of change was too high level and that the objectives should be more within the manageable control of the contract and reflect specific Office priorities. Two DDI/EDU respondents thought it would be helpful to create a more detailed theory of change that more specifically spelled out the assumptions and sequencing of desired results. One DDI/EDU respondent suggested that diversity, equity, and inclusion be added to the OE workstream in the future. All EPIC team respondents supported the package of the four workstreams. Two EPIC staff proposed that KM and PD be combined or integrated, and another proposed that the TOC reflect in some way the greater integration among KM, PD, and ECP that has evolved through EPIC. One proposed that OE be presented as a catalyst or enabler of the other three workstreams. DDI/EDU’s experience implementing this theory of change through EPIC will be a great example for other parts of USAID. IDENTIFY CHAMPIONS WHO CAN EFFECTIVELY LEAD WORKSTREAMS AND WORK SYNERGISTICALLY ACROSS TEAMS AND WITH OTHERS. Having champions that could bring together the various workstreams and technical teams for a larger purpose was helpful (e.g., an effective PD Activity Manager who leads the PD Working Group and the KM Activity Manager who convenes staff from other workstreams and teams to co-develop knowledge products and improve the EducationLinks platform). The most effective champions had a clear and strategic vision of where they wanted to go, communicated it well, got the right people involved, understood the relative strengths of the various contributors, established clear processes and expectations, and held people accountable for progress. 9 | EPIC EVALUATION REPORT USAID.GOV BE INTENTIONAL AND EXPLICIT ABOUT USING THE OE WORKSTREAM TO STRENGTHEN THE EFFECTIVENESS OF ALL WORKSTREAMS. While some DDI/EDU staff did not initially understand the purpose or scope of the OE workstream, and some resisted initial leadership development and teambuilding workshops, many came to view it as foundational to the work that the other workstreams were able to do individually and together. Nearly everyone commented on the improvement in the organizational culture of the Office over the life of the EPIC project and acknowledged the contributions of the OE workstream to this positive change. Clear outcomes emerged from most of the OE interventions that EPIC supported (see specific interventions and outcomes below in the OE section), but these interventions also often worked synergistically to contribute to the achievement of intermediate results in the OE workstream as well as contribute to results in other workstreams. For example, EPIC provided strategic planning and stakeholder consultation assistance to DDI/EDU to develop the USG Education Strategy and USAID Education Policy. The strategy and policy provided a framework that informed the Center’s priority-setting process, which occurred during the 2018 Center’s off-site meeting co-designed and co-facilitated by EPIC and DDI/EDU. The Center priorities informed each of the workstreams’ annual work plans. EPIC’s work with DDI/EDU to develop a standard work planning template made it easier and more efficient for teams and workstreams to collaborate. The creation of periodic cross-team content review meetings ensured that there was consistency in content across workstreams. This practice was not continued. LEADERS NEED TO BE COMMITTED TO CHANGE, UNDERSTAND THE MECHANISM’S POTENTIAL VALUE, STRATEGICALLY TAKE ADVANTAGE OF THE FULL RANGE OF SERVICES THAT IT OFFERS, AND HOLD THEMSELVES AND THE CONTRACTOR ACCOUNTABLE FOR PROGRESS. Leadership transitions at USAID are common. These transitions are challenging, both for the leaders and for the staff. DDI/EDU experienced several leadership transitions over the life of the EPIC project at the Office Director level as well as the Deputy and Team Lead levels. The EPIC team worked to support these transitions by offering coaching, being a thought partner, sharing institutional knowledge about the Office, its culture, and team dynamics, and offering support in the design and facilitation of Office gatherings and meetings in addition to the leadership development series that EPIC delivered. Nearly all who took advantage of this support found it helpful; a few chose not to engage. (See descriptions of these interventions and results presented in the OE IR 1.2 section below). If leaders are not interested in leading change and/or believe there is nothing they need to learn (e.g., don’t attend trainings, don’t use a coach, etc.), interventions such as these will be of little value. As one DDI/EDU leader commented: Be invested in change. If you’re invested in status quo, don’t do the work. Be willing to take risks. When it goes south, be willing to clean it up. — DDI/EDU Sr. Management To take full advantage of a mechanism like EPIC, leaders need to be committed to change, understand the mechanism’s potential value, and strategically take advantage of the full range of skills, experience, and services that it offers. Additionally, when Office objectives and initiatives are agreed to, leadership needs to hold themselves and their teams/groups accountable for progress and results. 10 | EPIC EVALUATION REPORT USAID.GOV HAVE THE RIGHT PEOPLE IN THE RIGHT JOBS AT THE RIGHT TIMES AND TAKE ADVANTAGE OF THEIR EXPERTISE. When the right people are in the right jobs at the right times and people are clear about what they can offer each other and how they can work together, the work is much more productive and effective. It took some time for EPIC to gain a clear understanding of DDI/EDU needs and to clarify the roles that EPIC staff were expected and not expected to take on to meet those needs. Some adjustments in staffing were made in the first two years to ensure that the right people were in the right positions. The flexibility of the EPIC mechanism allowed the team to quickly bring in “surge capacity” resources as needed. Similarly, on the USAID side, both EPIC and USAID interview respondents reflected on the importance of having the right people in the COR and Activity Manager roles. They cited examples of CORs and Activity Managers who provided excellent leadership characterized by having a clear vision and guidance so that EPIC could align its support and move forward on objectives. A few examples were provided that showed there was either a lack of strategic vision or there were personality differences that resulted in limited progress. It was clear that the COR and Activity Managers can have a big impact on the overall vision and progress of the project and of any given workstream. WORK IN STRATEGIC PARTNERSHIP. As mentioned above, having an effective strategic partner on the USAID side was critical to making EPIC’s support successful. When the Office Director, COR, and Activity Managers were strategic partners and champions, they could work as thought partners; generate new ideas, strategies, and plans to address Office priorities and needs; translate them into “AID-speak”; promote them internally to get buy-in; and monitor data on their progress to inform decision-making and adjustments in strategic and operational plans. This applies to all workstreams. START WITH A FULL ANALYSIS OF HOW YOU ARE FUNCTIONING AND DO PERIODIC ASSESSMENTS OF PROGRESS ALONG THE WAY. As EPIC did, do an in-depth analysis of how the Office or unit is structured and how it is functioning from diverse perspectives by involving everyone and building trust along the way. “Don’t assume anything about people on your team; start working with them from where they are as real people.” This is good, evidence-based organizational development (OD) practice, and it was mentioned by interview respondents as a lesson they learned. CREATE OPEN, INCLUSIVE SPACES FOR REFLECTION, LEARNING, AND CO-CREATION ON A REGULAR BASIS. Twenty-seven percent of DDI/EDU interview respondents stated that the annual off-site meetings and other Office-wide events such as town halls contributed to improved organizational effectiveness. Respondents said that it helped the Office to step out of the day-to-day operations to reflect on what was happening, how they were doing and what they were learning, get to know each other better, build trusting relationships, speak frankly about problems, and work together to create solutions and a collective picture of how to move forward. Having EPIC design and facilitate these sessions provided the value that an external perspective and honest broker role brings. At the same time, the ongoing relationship that EPIC had with the Office created a level of trust and understanding that was foundational to a more open and safe space for productive strategic, cross-team technical, and organizational conversations that helped move the Office toward greater organizational effectiveness. It 11 | EPIC EVALUATION REPORT USAID.GOV was also important to include diverse voices and views of key stakeholders within and outside of the Office (Mission staff, Regional and other Bureaus, host country and implementing partners, etc.) to build relationships and trust, enable better informed decision-making, and facilitate implementation of activities. (See OE IR 1.1 below for more specifics.) INSTITUTIONALIZE AND SOCIALIZE NEW AND UPDATED PROCESSES. Many new and updated Office processes were introduced over the life of the EPIC project, and many of these processes were documented. However, staff were not always aware of them or of changes when they occurred. Ensure that process documentation is easily accessible to all staff and that any new or updated processes are socialized throughout the Office. For these updated processes to be effective, the contractor should work with DDI/EDU to provide more support to socialization when USAID is short￾staffed. BUILD FLEXIBILITY INTO CONTRACT DESIGN AND MANAGEMENT TO FACILITATE RESPONSIVENESS, ADAPTIVE MANAGEMENT, AND QUICK PIVOTING WHEN NECESSARY. The flexibility and adaptability of the EPIC contract mechanism and its management enabled the project to accommodate emerging priorities and rapidly shifting contextual factors such as the government shutdown and COVID-19. This was due in large part to the close collaboration across workstreams and maintaining a large pool of resources (internal to TRG and its subcontractor partners as well as external consultants) that could be accessed for surge capacity. Contract finances were managed so that funding could be moved between activities as needed. In conclusion, EPIC serves as a compelling and useful example for USAID of the value of a highly flexible and comprehensive mechanism to strengthen a Center’s organizational culture and effectiveness, professional development, knowledge management, engagement, communications, and partnerships in support of Agency sector and development goals. This mechanism required a high level of collaboration and co-creation between the EPIC team and DDI/EDU leadership and staff. The EPIC team worked hard to develop trusting relationships with everyone in the Center by listening to them; acquiring a deep understanding of DDI/EDU’s people, their needs, goals, priorities, pain points, and aspirations together with the very dynamic and often challenging context in which they worked; and consistently delivering high quality, professional services. When things did not go as well as intended, EPIC team members were open to constructive feedback and worked hard to improve. When presented with major challenges such as COVID-19, EPIC was able to quickly adapt its support to meet the Center’s changing needs. The scope of the EPIC project was broad and ambitious. We cannot say that EPIC helped the Center fully achieve all of the strategic objectives and intermediate results articulated in EPIC’s Theory of Change and strategic results framework. This was probably unrealistic given the project’s broad scope and the many challenges the Center and EPIC team faced over the life of the project, including a change in government administrations, two government shutdowns, a hiring freeze followed by rapid Center growth, Agency restructuring, and a global pandemic that required all work to be conducted virtually. However, it is clear that EPIC has been successful in contributing to significant and noteworthy advances in DDI/EDU’s effectiveness as an emerging Center for Education. While it has not always been easy and there were setbacks, the Center has demonstrated resilience in the face of all these challenges. With EPIC’s support, the Center has improved its organizational culture, strengthened its leadership, systems, 12 | EPIC EVALUATION REPORT USAID.GOV and processes, led an inclusive process to develop and launch the USG Education Strategy and USAID Education Policy, increased collaboration across teams and with Regional Bureaus and Missions, launched an online Global Education Learning Series, established EducationLinks as a “go to” hub for education information to strengthen the capacity of field practitioners, supported the development of several global partners, and produced high quality communications and knowledge products that are helping USAID tell its education story. 13 | EPIC EVALUATION REPORT USAID.GOV INTRODUCTION BACKGROUND The Education Performance Improvement, Communications, and Knowledge (EPIC) project is a five￾year activity of support services for the U.S. Agency for International Development’s (USAID) Center for Education (DDI/EDU). The EPIC contract was awarded to Training Resources Group, Inc. (TRG), along with subcontractors Forum One and Social Solutions International, Inc. (Social Solutions), on October 1, 2015. Making Cents International (MCI) was added as a subcontractor in January 2017. The EPIC contract includes a one-year base period, followed by two, two-year option periods. To date, USAID has exercised both option periods and obligated a total of $19,104,582. EPIC’s objective is to improve education sector programming, measurement, and management. The project was designed to make paradigm shifts within and across workstreams as well as to model how a large organization can transform itself, build quality sector-wide learning, become more productive and strategic, develop internal and external partnerships, and more effectively pursue its mandate and goals. EPIC consists of support services in four workstreams, which are mutually reinforcing and implemented simultaneously: • Performance improvement through organizational effectiveness (OE); • Professional development (PD) and learning for all staff; • Knowledge management (KM), generation, and use; and • Engagement, communications, and partnerships (ECP). EPIC’s operations and management structure tracks progress and budget utilization of each activity under EPIC’s four workstreams. EPIC maintains a flexible structure to enable the team to respond to emerging priorities as necessary, within the confines of time and resources. The EPIC staffing approach has been to organically augment the team in response to specific project needs. Through the EPIC core team of eight full-time staff, TRG organizational development and training consultants, subcontractor staff, and 30+ independent consultants, the broad and deep EPIC bench of experts has enabled EPIC to mobilize the right people/expertise at the right time to be responsive to USAID needs while also being cost efficient. EVALUATION METHODOLOGY EVALUATION DESIGN The evaluation design is informed by the EPIC Performance Management, Monitoring, Evaluation, and Learning Plan (PMMELP) and takes into consideration priority areas of focus as well as practical and budgetary parameters in the context of the Coronavirus Disease 2019 (COVID-19) pandemic. The evaluator facilitated an evaluation design meeting with key DDI/EDU and EPIC team counterparts to review and revise the evaluation plan and framework, finalize the data collection and sampling strategy, and review the evaluation implementation process. The complete evaluation work plan is found in Annex 1. 14 | EPIC EVALUATION REPORT USAID.GOV EVALUATION FRAMEWORK Figure 1 graphically presents the theory of change incorporating key strategies that, when achieved together, contribute to the overall program goal. The theory of change informs the evaluation questions, data collection instruments, and presentation of data. The strategic results framework (Figure 2) provides more detail on EPIC’s strategic objectives (SOs) and intermediate results (IRs). This version of the results framework is slightly modified from the original included in EPIC’s PMMELP. Changes made mid-project included subsuming what was originally OE IR 1.4, “Capacity to inclusively plan, implement, monitor, and evaluate ‘Office Strategic Priorities’ strengthened” under the current OE IR 1.3, and adding KM IR 3.3, “Office of Education visibility is enhanced through the design and launch of an external education sector platform.” The theory of change makes clear that EPIC, by definition, would only be successful if DDI/EDU was successful in strengthening its OE, PD, KM, and ECP functions. FIGURE 1: EPIC THEORY OF CHANGE 15 | EPIC EVALUATION REPORT USAID.GOV FIGURE 2: EPIC STRATEGIC RESULTS FRAMEWORK 16 | EPIC EVALUATION REPORT USAID.GOV SAMPLING AND EVALUATION METHODS PRIORITY AREAS OF FOCUS Documentation of: • Progress made toward SOs and IRs • The process and results achieved through each individual workstream • The process and results achieved through collaboration across workstreams • The process and results achieved through collaboration between USAID and EPIC team • Best practices and lessons learned throughout the life of the program KEY EVALUATION QUESTIONS 1. What have been the project achievements as compared with expected results and objectives of the Strategic Results Framework? 2. To what extent has EPIC contributed to DDI/EDU’s effectiveness and work quality, particularly as related to its four workstreams of organizational effectiveness, knowledge management, professional development, and communications/engagement/partnerships? 3. What unintended positive or negative outcomes, if any, emerged from EPIC’s support to DDI/EDU and other USAID Operating Units (OUs)? 4. What are best practices and lessons learned from the EPIC program? 5. How did the project affect the intended beneficiaries? What factors contributed to or hindered its success? 6. Which tools and strategies were more effective in achieving the intended project objectives, and what were the contributing factors? 7. Did workstream collaboration lead to better or extended benefits? In what ways? 8. How has USAID’s collaboration with the EPIC program contributed to changes that have occurred over the life of the program? What did the EPIC team do in its work with USAID to contribute to bringing about these changes? 9. What external factors have made a positive contribution/have had a negative influence on the design, implementation, and outcomes of the program? DATA COLLECTION METHODS To respond to the evaluation questions, the evaluation team drew on a mix of methods, including semi￾structured interviews with key program informants; focus group discussions; a training follow-up 17 | EPIC EVALUATION REPORT USAID.GOV application survey with those who have been involved in and were intended to benefit from the program; review of project documents (e.g., annual and evaluation reports) and existing data; and construction of a life of program timeline based on document review and input from project staff and participants. Monitoring and evaluation (M&E) data that were collected throughout the EPIC program were consolidated, summarized, and analyzed. A data collection matrix, which includes the evaluation questions, illustrative measures, and a data collection methodology, is presented as Annex 2. The data collection tools are included in Annex 3. Virtual semi-structured interviews were conducted in June and July 2020 with 24 DDI/EDU staff and nine EPIC staff who represent various roles and perspectives in relation to the project. These interviews primarily collected qualitative information about program implementation, results, intended and unintended changes and outcomes, best/good practices, and lessons learned. Table 2 presents a summary of the characteristics of those who were interviewed. The interviews included elements of the Most Significant Change methodology, so interviewees were asked about the most significant changes they observed as well as small, meaningful changes that have occurred over the life of the program in the individual workstreams, cross-workstream efforts, and more broadly in relation to organizational effectiveness and collaborating, learning, and adapting (CLA). TABLE 2: INTERVIEW PARTICIPANT CHARACTERISTICS ORGANIZATION GENDER POSITION/FUNCTION TEAMS DDI/EDU Center n = 25 Male (24%) Female (76%) Sr. Management (20%) Team Lead (12%) Technical Specialist (64%) Operations (4%) EiCC (12%) EMT (20%) Engagement (12%) Evidence (12%) HEYWFD (12%) Program (20%) QUEST (8%) Reading (4%) EPIC n = 9 Male (33%) Female (67%) Sr. Management (22%) Team Lead (34%) Technical Specialist (22%) Operations (22%) Management (44%) OE (11%) PD (11%) KM (11%) KM/ECP (23%) Note: The participating teams include Education in Crisis and Conflict (EiCC); Executive Management Team (EMT); Higher Education, Youth, and Workforce Development (HEYWFD); and Quality, Equity, and Sustainability Team (QUEST) A follow-up survey was administered in November 2020 to 167 DDI/EDU staff to gather additional information on the utility, application, and results of EPIC-supported learning opportunities and knowledge resources. Seventeen staff (10 percent) completed the survey. Table 3 presents a summary of the characteristics of those who completed the survey. 18 | EPIC EVALUATION REPORT USAID.GOV TABLE 3: FOLLOW-UP SURVEY RESPONDENT CHARACTERISTICS. TENURE GENDER AFFILIATION TRAINING ATTENDED (COULD TAKE MORE THAN ONE COURSE) 1-2 years (11.7%) 3-6 years (41%) 7-9 years (17.6%) 10+ years (29.4%) Male (41%) Female (58%) Mission (47%) Regional Bureau (11.7%) Office (41%) Ed-CVE (46%) EEPM (61%) EOC (38%) ETHICS The evaluators abided by the American Evaluation Association Guiding Principles for Evaluators.1 INFORMED CONSENT All interview and focus group participants were informed that their participation in data collection was voluntary, that they did not have to answer questions that made them uncomfortable, and that they could halt their participation at any time. Protection of research participants is paramount in an evaluation, and the team took steps to minimize any risks associated with participation in data collection activities. Confidentiality procedures were explained to all participants involved in data collection. PROTECTION OF DATA The evaluation team used encryption to protect the qualitative data set and scrubbed identifying respondent information from all transcripts prior to analysis. Transcripts were numbered, with associated identifying information stored in a separate document. The evaluation team has not and will not share the data set with any stakeholder without DDI/EDU’s permission and without conducting a rigorous data scrubbing process to ensure protection of evaluation participants. ANALYSIS, INTERPRETATION, REPORTING, AND LEARNING ANALYSIS AND INTERPRETATION Qualitative data were coded and analyzed (both content and thematic) in Dedoose, a cross-platform application for qualitative and quantitative analysis. The evaluation team discussed the coding structure, applied codes, and identified salient themes to ensure agreement. Quantitative data were analyzed using Excel. Qualitative and quantitative data were triangulated across sources, where feasible. The Evaluation 1 The American Evaluation Association’s Guiding Principles for Evaluators can be found online: http://www.eval.org/p/cm/ld/fid=51 19 | EPIC EVALUATION REPORT USAID.GOV Team Lead shared preliminary findings with EPIC’s Contracting Officer Representative (COR) to ensure the relevance and utility of initial findings. The team ensured data reliability and integrity using the data collection practices described above. Data from multiple sources were reviewed and analyzed to address the potential limitations of self-reported data. Some stakeholders expressed survey fatigue over the course of the project, which affected completion of questionnaires. To address this challenge and collect robust data, the evaluation team developed surveys that were as concise as possible, used online administration methods for ease of completion, and sent invitations and reminders that appreciated and encouraged participation in post￾activity and follow-up surveys for learning purposes. The data collection tools are included as Annex 3. Critical priorities for this evaluation were to document project results and outcomes and to identify best practices and lessons learned that can be applied to similar programs. To validate the findings, conclusions, and lessons learned, the evaluation team shared the preliminary findings with key DDI/EDU counterparts and the EPIC team. REFLECTION, FEEDBACK, AND REPORTING To ensure a useful report, the evaluation team developed a report template early in the evaluation process to share with DDI/EDU and the EPIC team for feedback. When the analysis was complete, the team drafted a report for DDI/EDU’s review. This first draft report was shared with DDI/EDU and EPIC senior management. Those who received the draft participated in a videoconference for a high-level overview of the findings and they provided feedback to inform further development of this report. It is important to note that validation of recommendations with project implementers is an evaluation good practice to ensure that recommendations are useful for project strategy going forward. The Evaluation Team Lead validated draft recommendations with DDI/EDU and the EPIC team before they are finalized. CONFIDENTIALITY Representative quotes from data collection are used in the report to add context to findings. Participants, however, are not identified by name, and some quotes have been edited not for substance but to protect confidentiality. 20 | EPIC EVALUATION REPORT USAID.GOV EVALUATION FINDINGS AND RECOMMENDATIONS This section presents evaluation findings in response to key questions by workstream and cross-cutting themes. Each section includes outcomes presented according to the strategic results framework, effective strategies, tools and practices, lessons learned, and recommendations. Annex 5 presents a summary of factors cited by interview respondents and in EPIC reporting that facilitated and hindered achievement of EPIC’s strategic objectives and intermediate results overall, and those that affected specific workstreams. ORGANIZATIONAL EFFECTIVENESS SO 1: STRENGTHEN DDI/EDU’S ORGANIZATIONAL CULTURE, STRUCTURE, SYSTEMS, AND PROCESSES The strategic objective of this workstream was to strengthen DDI/EDU’s organizational culture, structure, systems, and processes by improving the Office’s organizational culture; strengthening leadership capacity; and aligning the Office’s resources, plans, and activities with its purpose, priorities, and objectives while strengthening its capacity to collaborate, learn, and adapt. This workstream was, in many ways, foundational to the success of the other workstreams in that it affected the ways in which the Center’s individuals and teams worked together to achieve DDI/EDU’s goals and objectives. This was a new form of support for the Center, and it took some time for leadership and staff to define and understand how they could use this type of support. Seventy-five percent of the 24 DDI/EDU interview respondents believed that, overall, EPIC had contributed to strengthening DDI/EDU’s organizational culture, structure, systems, and processes and 25 percent either did not feel they had enough experience with the Center to judge or did not specifically address this question. While there were many advances, interview respondents commented on the non-linear nature of progress made in improving organizational effectiveness, sometimes taking one step forward and two steps back as they encountered various challenges such as government shutdowns, COVID-19, transitions in leadership and other Center personnel, the Agency reorganization, and others along the way. In general, respondents noted improvements in organizational culture and the Center’s improved ability to work collaboratively internally and with others, the establishment of systems, processes, and tools, and leadership and management skills development, while noting that there is still room for improvement in terms of follow up in the more consistent use of these skills, systems, processes, and tools. The following sections describe the interventions that EPIC co-designed and implemented with DDI/EDU, the outcomes they produced that contributed to each Organizational Effectiveness (OE) IR, and the factors that facilitated and hindered progress. Key events over the life of the project in this workstream are presented in Annex 4, Table 1. A summary of the factors that facilitated and hindered the success of EPIC’s OE efforts is presented in Annex 5. IR 1.1: DDI/EDU’S ORGANIZATIONAL CULTURE IMPROVED. Seventy-nine percent of the 24 DDI/EDU staff interviewed noted improvement in organizational culture over time from a “toxic,” “competitive,” and “dysfunctional” Office climate to a more open, transparent, collaborative climate where there is more evident cross-team integration and collaboration. One person (4 percent) did not believe that culture improved, one person (4 percent) commented that improvement “goes up and down depending on leadership,” and three people (13 percent) felt that they 21 | EPIC EVALUATION REPORT USAID.GOV were too new to the Center to judge whether the culture had improved. While acknowledging that culture had improved from when EPIC started working with the Center, five people (21 percent) stated that more still needs to be done, particularly in light of COVID-related challenges and morale issues related to work overload. Six people (25 percent) emphasized the important role that leadership plays in setting the tone for culture, following through with agreements, and holding themselves and staff accountable for creating and sustaining a positive work environment. “When EPIC first came in on the OE side, there was a lot of bringing people together as human beings and trying to create a healthy Office ambiance. When I reflect on it, I think that needed to happen. Some may have thought what are we doing? Wasting our time here. Clearly there needed to be cultural change to create a space where people took care of one another, respect one another. There were situations that were disrespectful. It was hugely challenging. ...The culture has improved through the town halls and the off￾site events – created more breathing room space to talk about bigger picture issues. I think that we had new management when EPIC came in. It was good timing with a fresh team and a new leader. Took people a while to get their footing from a dysfunctional situation where people cranked out work and weren’t respectful of each other. It’s still a challenge because the workload hasn’t gone down. We’ve had attrition through the USG, and it’s hard to keep people. There are still some challenges but those have been mitigated by an emphasis on better communication” — DDI/EDU Team Lead “Before I came to DC, the Office of Education was a disaster. When I was in the field, I didn’t even want the Office to support me. That has dramatically changed. We have so many requests from the field now. That has to do with the changes in the Office culture and climate. The difference is really dramatic. A whole new group of team members with tremendous technical skills and a tremendously improved climate in how people work together, which has improved our support to the field, which is our main objective.” — DDI/EDU Sr. Management “I don’t think that the organizational health and culture improved but not for lack of effort by EPIC. EPIC was empowered and proactive to bring around changes but it’s hard when Office leadership doesn’t walk the talk. The project did provide a lot of training activities and coaching, which were valuable investments. There was a definite commitment to improve the Office, and I liked the members of the EPIC team. It wasn’t a reflection of EPIC. The consultations with staff and the meetings didn’t manifest in a change in culture. I’m not sure I saw the leadership and management skills [of team leads] strengthened [over the five years]. It may have been the personalities of the team leads. I thought MBTI [Myers-Briggs Type Indicator] was really helpful. It may be partly that we were in our silos. Guilty as charged. Decisions would be made by the same set of individuals without transparency. Factors that contributed to lack of transparency in decision-making were that the Office was faced with all sorts of demands from on high, political appointees. If your plate is full, it’s hard to look beyond that and take a step back, move forward with other things you desire. Highly stressful positions for leadership. Need to hire people who strengthen Office morale and culture.” — DDI/EDU Technical Specialist “There have been changes in how we work together, in organizational wellness, the support they provide our leadership, and the cascade of how that increases staff morale (and sometimes staff turnover) making them feel empowered and the staff feeling heard.” — DDI/EDU Technical Specialist “I recognize where we were and where we are, and I recognize the role EPIC played in that. If folks wondered what’s the purpose here, it was a cultural change from a transactional situation to one that takes time to talk to one another, break bread together, enjoy one another, and take advantage of each person’s expertise to move forward together. In the beginning, people didn’t recognize the value of the OE work. From my perspective, the Office was ailing, and I don’t know what a better approach would have been. To 22 | EPIC EVALUATION REPORT USAID.GOV get people out of what is a very narrow mental space of their own concerns to a more open creative space.” — DDI/EDU Team Lead In collaboration with the Center, EPIC designed and implemented or facilitated interventions to assess organizational culture needs and monitor progress in meeting them. In partnership with the Center, EPIC created safe and open spaces and processes to bring Center staff together to: • Get to know each other better • Build positive relationships and trust • Share information • Collectively strategize and agree on key priorities • Establish and practice norms and guidelines for how to work effectively together • Celebrate accomplishments, pause, and reflect • Identify challenges, and co-create solutions to address them The following key interventions were identified in EPIC reporting and by interview respondents as having contributed to improvements in organizational culture. The numbers in parentheses are the percentage of the total of 33 interview respondents (total includes nine EPIC team interviewees) who specifically cited these interventions as having contributed to improved organizational culture. • All-Staff Meetings: Office-wide off-site meetings (45 percent) and all-hands meetings (27 percent) • Organizational Wellness and Staff Surveys (February 2018, May 2018, December 2018, April 2019, October 2019, and May 2020) (45 percent) • Leadership Development Program (39 percent) • Coaching (33 percent) • Consulting/being a thought partner/informal conversations (18 percent) • Support to Executive Management Team and their support to Team Leads (12 percent) • Process and system design (12 percent) In addition to the interventions mentioned above, respondents mentioned a few others less frequently (three or fewer mentions) as having directly or indirectly contributed to improvements in organizational culture as well as having contributed to other IRs (e.g., strategic planning, work planning, cross￾workstream technical collaboration, conferences and large meetings, MEL). These interventions are described in the relevant IR sections below. Under this IR, we will discuss how EPIC implemented off-site and all-hands meetings and the organizational wellness and staff surveys, their outputs and outcomes, and factors that helped and 23 | EPIC EVALUATION REPORT USAID.GOV hindered their success. We will discuss the Leadership Development Program, coaching, and support to the Executive Management Team under IR 2 as these interventions also contribute to strengthening leadership capacity, and we will discuss process and system design under IR 4, which focuses on program cycle systems and processes. Designing & Facilitating Meetings In collaboration with DDI/EDU, EPIC co-designed and facilitated numerous meetings over the life of the project including annual all-staff off-site meetings, 16 all-hands/town hall meetings, numerous team/workstream retreats, and monthly Executive Management Team strategy meetings. Off-Site Meetings “OE was a sticky workstream. People didn’t really understand what it really meant and how they could benefit from it. The idea is to have space where people can intentionally strategize, be inclusive, come up with guidance on the way forward for the Office and the sector.” — DDI/EDU Operations Specialist In collaboration with DDI/EDU leadership and staff, EPIC designed, facilitated, and supported off-site meetings to address the Center’s pressing strategic, managerial, organizational culture, and interpersonal issues as well as engage in work planning. • In 2016, year one of the project, a two-day Center-wide off-site meeting to pause, reflect, and identify a collective way forward led to creation of a new organizational structure and spaces for communication, decision-making, effective meetings, and identification of other improvements in how the Center could work more effectively together. “The first pivotal moment was the first 2016 off-site retreat. There was a lot of reticence about doing it. You could tell at the end of that meeting how relieved they were. They felt more like one Office.” — EPIC Sr. Management • In 2016, EPIC designed and facilitated a series of team-building meetings with the Quality, Equity, Sustainability Team (QUEST) to help them redefine and rebrand their role in the Office, which resulted in improved engagement and relationships with the different technical teams. • In 2017, EPIC designed and facilitated a one-day Education Sector Council (ESC) retreat for representatives from all DDI/EDU teams and Regional Bureaus aimed at more cohesive support for Missions. A working group was formed to guide implementation of decisions made at the retreat (e.g., create a working group to catalogue education sector research activities, and to repurpose ESC meetings). • In 2017, EPIC designed and facilitated five two-hour workshops (with all staff in small groups), which resulted in consensus on the five Office Strategic Priorities. This served as the framework for team work planning and prioritizing the Center’s activities. This workshop series was followed by a Center-wide off-site meeting designed and facilitated by EPIC that resulted in participatory construction and incorporation of Center of Education Strategic Priorities for 2018-2019, follow-up on organizational wellness and Office culture, and reorganization of the new Center for Education as well as other topics relevant to other IRs. Participation of Regional Bureaus and colleagues from the Global Development Lab strengthened relationships and increased communication and collaboration within the education sector. This meeting built on input from key informant interviews that EPIC had carried out earlier with Regional Bureaus to obtain feedback on collaboration with DDI/EDU 24 | EPIC EVALUATION REPORT USAID.GOV and ways in which collaboration could be strengthened and the input from the ESC meeting described above. • In 2018, EPIC co-designed and co-facilitated a Center-wide off-site meeting that followed up and built on the previous year’s off-site meeting. This was the first time Regional Bureaus and colleagues from the Global Development Lab attended a DDI/EDU off-site meeting. The meeting focused on organizational wellness, Center strategic priorities, USG Education Sector Strategy, and USAID Education Policy. Key outcomes of this meeting included: − An Organizational Wellness working group was formed committed to organizing activities to improve organizational culture, climate, and effectiveness based on input from staff reflections on results of the organizational wellness surveys and dashboard. − A task force was established to advance the discussion on a new structure for the Center. − A working group was formed to define a work plan review process and conduct work plan reviews. − The Center agreed to pilot a new Mission Support Form. − Participants identified asset mapping of existing networks to increase thought leadership and partnership, and each team shared who their networks/partnerships were and the composition and value of each. EPIC agreed to support the Center for Education Working Group meetings on a weekly or biweekly basis during the last quarter of 2018 to focus on issues related to reorganization and restructuring. The 29 participants who responded to the meeting evaluation rated the content of this meeting (what was discussed) an average of 7.9 out of 10 and the process (methodology, facilitation, engagement) an 8.5 out of 10. • In 2019, EPIC co-designed and co-facilitated a Center-wide off-site meeting that brought together 45 DDI/EDU staff and colleagues from Regional Bureaus and the Global Development Lab. Day one focused on 1) the new and evolving DDI/EDU Center functions and priorities, 2) an exercise on Education Policy, 3) a presentation of the policy rollout framework, 4) a role play activity on communicating policy and education-related priorities to various audiences, and 5) an “unconference” gallery walk, where staff identified the issues they wanted to address and created groups around new initiatives/ideas for changing the way DDI/EDU works. Day two opened with an update and analysis of existing partnerships, USAID’s role, the strategic value, and recommended next steps for each followed by a mini “book club” discussion around six articles related to the principles and practices of a healthy work environment. The meeting concluded with creating recommendations and considerations for DDI’s new operating model. As a result of the healthy work environment discussions, participants agreed that a healthy work environment relies on individuals, teams, and the Center to creatively overcome challenges around transparency, morale, and collaboration. The ideas that were generated and rated highly by staff included: 1) an extra flex hour each week; 2) improve decision-making by categorizing and making them at the right level; 3) recognize and build on positive moments; and 4) distinguish and be mindful about different types of collaboration. Twenty-five evaluations were completed, resulting in an average score of 3.84/5.0 on 25 | EPIC EVALUATION REPORT USAID.GOV the extent to which meeting objectives were achieved and 3.88/5.0 for overall satisfaction with the meeting. • In January 2020, EPIC co-designed and co-facilitated an Off-Site/Mid-Year Review to focus on better coordination across contract mechanisms and facilitated a supervisor/team lead training/workshop. The first half of the meeting focused on contract mechanism management and coordination, including pain points, recommendations, and best practices for CORs and Alternate Contracting Officer Representatives (AORs); COR/AOR support; and ignite talks with colleagues on key education themes. The second half consisted of a Supervisor and Team Lead Workshop to help participants examine their best moments, pain points, difficult issues, common concerns, frustrations, good practices, real-life and case examples of situations in the Office, and ways forward. A TED Talk on bullying and toxicity was shared and discussed. The post-retreat evaluation from this meeting found that: 76 percent of respondents felt the off-site is a valuable part of creating a healthy Office climate and culture; 74 percent (20) of respondents had a better understanding of pain points and the way forward on collaborating on Office-wide mechanism management (five were neutral and two disagreed). Fifty-two percent (14) of respondents had a better understanding of best practices and the way forward on collaborating on the management of Center-wide mechanisms/ procurements (nine were neutral and four disagreed). And, 74 percent (20) enjoyed participating in the Center Off-Site (11 strongly agreed) (four were neutral, one disagreed, and one strongly disagreed). “Offsites...each one got bigger and better, first was big, second got bigger and better, the third one leadership decided to open it up to the Regional Bureaus and the labs. There had been silos. This was a good way to break those down, hear announcements from the same source, sit at the same table, and discuss the same issues.” — DDI/EDU Operations Specialist “Each of the retreats led to the development of a work plan for the following year...We clarified expectations of staff with a participatory process. Created a set of competencies and behavioral indicators, which were used in a 360 instrument that informed coaching. For a period of time there was some energy around that. Another was the next year or a year later when we looked at the Office’s strategic objectives/priorities. Conversations came up with five priorities to inform work planning and collaboration.” — EPIC Technical Specialist “What I have noticed the most has been we went through all these workshops that were helpful making staff feel comfortable talking about issues and leadership began to know what the issues were. It was a safe space. On the negative side, those who were creating problems didn’t come. Either they were on TDY or didn’t care. That’s not on EPIC, they just didn’t come. They may have benefitted but they had some self￾awareness problems.” — DDI/EDU Technical Specialist All-Hands Meetings EPIC collaborated with DDI/EDU to co-design and co-facilitate 16 all staff meetings including all-hands staff meetings and quarterly town halls. These meetings provided a space for getting to know each other better, providing updates and information sharing, monitoring progress on Center priorities and organizational wellness, gathering staff input, offering short training sessions, pausing and reflecting to celebrate accomplishments, identifying opportunities and challenges, sharing lessons learned, solving problems, reviewing priorities, and planning for the future. As the Agency reorganization evolved, staff meetings also focused on the DDI transformation, matrix management, and plans to transition to the new Center of Education. 26 | EPIC EVALUATION REPORT USAID.GOV During an all-hands meeting in 2018, EPIC requested feedback from DDI/EDU staff on how their support contributes to the Center, what they should continue doing, and what they should stop doing. Comments on EPIC’s contributions to the Center were overwhelmingly positive (e.g., “Supports us in thinking through our most pressing challenges,” “Great facilitation that helps us focus,” “Helps us work better to get the jobs done,” “Coaching enormously helpful,” “Extremely helpful to our work planning,” and “You are the GLUE and leading from behind.” Center staff commented that EPIC should continue: excellent organization and project management, making meetings practical, bringing together the sector and providing feedback from stakeholders, providing direct support to teams, training, leadership coaching, consultant support for key activities, innovative thinking to keep group reflection events interesting, great facilitation, generating user-friendly knowledge products, and being great thought partners. Feedback on what EPIC could do better or differently included: help the Center do better with coordination and follow-up or management actions, balancing thinking with feelings in large groups, improving coherence of PD activities, help make guidance notes and knowledge products look beautiful (as EPIC had already done with some teams), and hold the staff/EMT firm to decisions (roll out). DDI/EDU responses to all-hands meeting evaluations over the years provided EPIC and DDI/EDU leadership with feedback on what staff found most useful and ways to improve these gatherings. Nearly all survey respondents expressed their appreciation for these meeting spaces. Specifically, they appreciated leadership updates, agendas that are not overpacked, the mixing of methods that allow for interaction as well as information sharing, finishing on time, allotting sufficient time for topics on the agenda, skilled facilitation that helps the group focus, integrating team building activities, prepared speakers, and when concrete actions/next steps are agreed upon and followed up on. Constructive feedback to improve all-hands meetings tended to be specific for each meeting (e.g., should have given more time to x topic or put x topic earlier in the agenda). More frequently cited constructive feedback related to ensuring that there were clear actions/steps agreed upon and that there was follow-up to ensure that progress was made on them, that there was not too much time dedicated to staff just listening rather than interacting and providing input (e.g., spend more time in breakout groups), seek staff input before meetings to inform the agenda, and ensure that speakers/presenters are prepared. “I liked when staff meetings were mandatory and everyone had to go. I attended all of them and felt that they were incredibly helpful. A year and a half ago they were not required and attendance dropped off. That was unfortunate. EPIC did everything they could given the mandate.” — DDI/EDU Technical Specialist “If there is one overarching theme, we created spaces and prepared people to have good conversations within those spaces and looked at problems and issues to come up with solutions. We didn’t call it CLA, but that’s what it was. We didn’t just look at deficits and problems, we looked at how to build on strengths and what works. You could see the difference in the toxicity. Out of the meetings came the coaching program and leadership training. We’d put those into action and begin to see improvement within team and across team collaboration.” — EPIC Sr. Management Organizational Wellness & Staff Surveys (FEBRUARY 2018, MAY 2018, DECEMBER 2018, APRIL 2019, OCTOBER 2019, AND MAY 2020) In 2018, EPIC worked with DDI/EDU in focus groups to identify indicators and behaviors that would serve as the basis for assessing progress toward a shared vision of organizational wellness. DDI/EDU agreed on seven organizational wellness domains including: 1. The need to establish an empowering/enabling environment that clarifies roles, values skills and opinions, provides skills, guidance, and support, etc.; 27 | EPIC EVALUATION REPORT USAID.GOV 2. The need to build up the kind of decision-making that incorporates inputs from staff, is transparent, timely, thorough, and communicated; 3. The need to inculcate accountability at all levels, where priorities, expectations, and actions are agreed upon, and fairness, consistency, and feedback are exercised; 4. The need to build a culture of respect and appreciation, where successes are celebrated, well￾being and culture are considered and nourished, and dismissive and disrespectful behavior is a thing of the past; 5. The need to create purposeful collaboration (not just collaboration for collaboration’s sake), where information is shared, expertise and strengths are leveraged, the right people are included at the right time, and collaboration adds value; 6. The need to improve efficiency and effectiveness, where DDI/EDU defines and follows SOPs, and organizes around and makes progress on the Office Priorities; and 7. The need to support personal efficacy and responsibility, where each person takes an active role in making the Office a good place to work, builds effective work relationships, brings issues and problems to the table in constructive ways, takes steps to pursue their professional development, and is civil and respectful to others. EPIC administered organizational wellness surveys to DDI/EDU staff in February (n=35), May (n=29), and December 2018 (n=32). The results informed organizational wellness dashboards that EPIC shared with the whole Center during town halls and staff meetings that EPIC co-designed and co-facilitated. The dashboard provided staff with vocabulary they could use to identify their needs, articulate their expectations, agree on next steps for improvement, and monitor their progress. “At the end of 2017/2018, we started working on the organizational wellness instrument. There were a lot of grumblings in the Office, people talking about toxic relationships, lack of transparency, back door meetings, etc. We proposed the instrument, which was a survey that we would implement periodically to see what progress the Office was making on the various indicators. We had focus group discussions with everyone and based on what we heard, we developed various benchmarks with behavioral indicators that were very specific and measurable.” — DDI/EDU Operations Specialist “For organizational health, we proposed an instrument to help them gauge it periodically. It made a difference for them to clarify expectations of each other. We had themes and behavioral indicators for them that became the subject of discussion in retreats and Office gatherings. It gave them some language and a shared framework that helped them determine how to make improvements.” — EPIC Technical Specialist Figure 3 presents a summary of results by domain of the organizational wellness self-assessment surveys conducted in 2018. 28 | EPIC EVALUATION REPORT USAID.GOV FIGURE 3: ORGANIZATIONAL WELLNESS DASHBOARD (FEBRUARY, MAY, DECEMBER 2018) Overall, DDI/EDU staff rated their organizational wellness for all domains at an average of 3.4/5.0 in 2018. DDI/EDU staff consistently rated their personal efficacy and responsibility the highest of all domains with an increase over time from an average of 3.93 in February to 3.99 in December 2018. Other noteworthy indicators that were highly rated included: “Our collaboration adds value” with 83.87 percent agreeing or strongly agreeing; “My supervisor has the autonomy, skills, guidance, and support to be effective” with 65.63 percent agreeing or strongly agreeing; and “We make progress on the 5 Office Strategic Priorities” and “We develop, refine, and follow Bureau and Office SOPs” both at 61.29 percent agreeing or strongly agreeing. DDI/EDU staff rated the Accountability domain lowest at 3.13 (February), 2.99 (May), and 3.02 (December). Specific noteworthy poorly rated indicators included “We follow our SOPs and tools for onboarding new staff” with 41.93 percent disagreeing with a range of comments from “we’ve fallen behind” to “there has not been new staff for years” to “I’ve heard from new staff that this has improved over the last few years”; “There is fairness and consistency in holding people accountable within and across teams and levels” with 38.71 percent disagreeing with a couple people commenting that there appear to be favorites and that some are held to a higher standard than others; “I have the authority and information I need to make decisions and take action” with 37.51 percent disagreeing, primarily according to their comments that they aren’t delegated the authority by leadership; and “I receive sufficient and effective feedback to better achieve goals” with 32.26 percent disagreeing, commenting that feedback is “not frequently received on work or growth areas,” that one “needs to ask for it and even then I don’t always get it,” and that there are “times when I receive conflicting signals or instructions.” A more detailed breakdown by individual indicator for all three surveys is presented in Annex 5. Equally important, discussions of the results of the organizational wellness surveys and the narrative feedback that staff provided in those led to some actions – including the decision to have monthly all-hands meetings to provide an ongoing space for staff to discuss “their issues,” and the impact of the many changes that were occurring due to the restructuring of the Agency and the transition from E3/ED to the Center for Education. 29 | EPIC EVALUATION REPORT USAID.GOV “I don’t know a lot about what happened in the background of the surveys, but I did appreciate that I could trust that the results of the survey were honest because EPIC was doing it. They were able to be a neutral space to share the findings. It added to the transparency.” — DDI/EDU Technical Specialist “The org wellness dashboard was really helpful. The fact that the needle didn’t go down in the midst of extreme periods of change is a positive indicator. That’s something to be proud of. In other orgs, the needle could have shifted the other way very quickly.” — EPIC Sr. Management In 2019, there was a leadership transition in the Center that resulted in a shift to a modified staff survey that was more focused on giving feedback to leadership. While this yielded useful data, EPIC is unable to track progress against the original indicators and behaviors over a longer period of time. EPIC supported the administration and analysis of a new survey in April 2019 (n=36) that included aspects of organizational wellness and other topics of interest to the new leadership including: what staff considered to be foremost priorities, significant challenges, what the Center was doing well and what needed to be improved, what contributed to staff satisfaction and what would make work more rewarding and satisfying. This survey highlighted priorities of staffing: making sure staff had the right complement of skills to provide ongoing technical leadership and support to Missions and Bureaus and the challenge of providing sustained support when there was such a high number of institutional contractors on shorter-term contracts. Staff called for a strategic staffing plan and creating an operating culture that supports better communication, decision-making, and technical collaboration. Also related to staffing deficiencies, staff expressed their feelings of being overwhelmed due to high workloads and inequitable work distribution, which eroded staff morale. Another priority area identified by staff was the desire for more effective collaboration, which, at the time, was challenging to achieve, given their silo-operating organizational structure, the uncertainty about restructuring to the DDI Bureau and Center for Education, and cumbersome bureaucratic processes. They also prioritized the lack of transparency and respect from leadership as an area in need of attention due to what respondents felt was “burdensome and relentless scrutiny” of the TDY approval process, micromanagement, a lack of flexibility, and lack of trust by leadership in staff professionalism. Staff provided feedback on EMT leadership strengths (technical expertise, caring about individuals, timely sharing of information, transparency, inclusion of division chiefs to form a larger leadership team) and what they should do less or stop doing (provide more transparency and accountability in decision￾making, listen to all voices—not just favorites, get rid of decision-making bottlenecks, reduce layers of approval). Nearly 78 percent of staff stated that they were somewhat satisfied at work (between 3 and 4 on 5-point scale). They shared specific examples that contributed to their satisfaction and what would increase their satisfaction at work. Based on this input, EPIC supported the EMT in taking steps to address areas in need of improvement. For example, attending to staffing issues, changing how Friday staff meetings were structured and handled, adjusting roles and responsibilities of EMT members, adjusting management processes, increasing communication about decision-making, etc. EPIC administered a follow-up staff survey in October 2019 (n= 32) that focused on whether staff observed any positive advances since the last survey, what leadership was doing well that should be continued, what leadership should do differently, what changes had been made since the last survey that had contributed to staff satisfaction, what needs to be “unstuck,” and what else would make their work more satisfying or rewarding. 30 | EPIC EVALUATION REPORT USAID.GOV Seventy-two percent of respondents expressed strong agreement that there had been good advancements/positive actions since the last survey in March 2019, and 3 percent expressed strong disagreement. In order of frequency, advances included increased transparency/communication (eight comments), management changes and more efficient management (seven comments), filling in staffing gaps (six comments), positive reinforcement (four comments), EMT responsiveness and outreach (four comments), and opportunities for collaborative efforts (four comments). EPIC contributed to these advancements by supporting the EMT leadership with coaching, consultation and informal conversations, design, and facilitation support for EMT strategy meetings, all-hands and other meetings. Respondents also identified challenges the Center was facing that needed attention moving forward including low morale due in large part to overwork and negative behaviors that are not reprimanded; more transparency around decision-making and timely communication on how/why decisions are made; and ensuring equities across the Office in terms of equitable workloads, resources, and recognition. Changes that contributed to work satisfaction since the March 2019 survey that participants noted included positive reinforcement given through the kudos activity during Friday meetings, recognition and thanks given on a regular basis, and shout outs to leadership/participation during social activities, which help to build team cohesiveness; management changes that resulted in respondents feeling like leadership listened to their feedback and increased their autonomy; and more relevant meetings as a result of modifying the focus of Friday meetings and all-hands meetings that introduced opportunities to learn about a technical area, Agency initiative, or mechanism. Respondents continued to call attention to the need to staff up so that workloads could become more manageable and to improve staff capacity in various technical areas so that they could meet work demands and increasing support requests. They expressed continued concern about relitigating decisions and stressed the need for enforcing norms and efficient processes for making decisions and who should make them. They also identified the need to address Team Lead and management issues by providing supervisory training and holding managers accountable for positive behavior. Finally, respondents identified what would make work more satisfying, including more autonomy/opportunities to contribute, career advancement, telework flexibility, more manageable workloads, reinforcing Center norms/org health, and prioritizing staffing. In collaboration with DDI/EDU leadership, EPIC included offering supervisory training to Team Leads and managers and increasing responsiveness to the field by implementing a new Mission support process to address some of the concerns raised in this survey. In May 2020, EPIC supported the administration of a more targeted survey (n=27) that focused on how staff were adapting to remote work during the COVID pandemic, including questions related to teleworking, Friday all-staff virtual meetings, and any issues or problems that staff were dealing with that they wanted leadership to be aware of. Over 40 percent of respondents agreed that teleworking and related processes were going well. They credited clear guidance and having the necessary tools, flexible hours, frequent check-ins, and fun competitions as contributing to their positive experience. The remaining respondents did not mention whether telework was working well or not. Aspects of telework that respondents said needed to be improved included long work hours, length and frequency of meetings, email etiquette and volume, refocusing priorities, scheduling meeting times, and lack of transparency in communications regarding COVID-19 decisions. Thirty percent of respondents stated that the Friday virtual staff meetings were working well, 22 percent said they were “fine,” and 11 31 | EPIC EVALUATION REPORT USAID.GOV percent did not feel any changes were needed. The remaining 37 percent did not mention whether the virtual staff meetings were working well or not. Suggestions for improvement of the virtual staff meetings included meeting logistics, having a more focused meeting agenda, and reinforcing Office priorities. Other issues related to COVID and working from home were: work/life balance and work hours, process for EMT decision-making and clearances for COVID-19 related things, anxiety about the phased return process, and appreciating leadership empathy and patience for challenges staff are facing in getting their work done. EPIC produced a synthesis report of these findings and shared them with the leadership team to determine follow-up steps. The organizational wellness and staff surveys provided important input to leadership from staff on the state of the Center. Staff meetings provided space for staff and leadership to discuss the findings, agree on action steps, and hold each other accountable. We recommend that in the future, the Center consider a hybrid survey that combines the organizational wellness questions with additional questions on other aspects of organizational effectiveness such as systems and processes, leadership, etc. to provide a more comprehensive picture of progress moving forward. The survey should reflect a shared vision of the hard and soft qualities, benchmarks, behaviors, and kind of transformational Center that will result in transformational education outcomes. IR 1.2: DDI/EDU LEADERSHIP CAPACITY TO APPLY LEADERSHIP AND MANAGEMENT SKILLS TO THEIR WORK STRENGTHENED. Overall, 33 percent of DDI/EDU interview respondents believed that leadership capacity was strengthened over the life of the EPIC project, 30 percent believed that accomplishments in leadership capacity strengthening were mixed; 30 percent did not feel they had been in the Center long enough to judge or did not address this question; and 8 percent did not believe that leadership capacity had been strengthened. The degree to which DDI/EDU interview respondents perceived that leadership capacity was strengthened varied depending on their position in the Center. Four out of five senior management leaders believed that leadership capacity had been strengthened (the fifth person did not specifically address this question). Of the Team Leads interviewed, one did not see any change, two reported mixed results depending on the level of leadership, and two did not specifically address the question. Of the Technical and Operations Specialists, 29 percent believed leadership capacity had been strengthened, 36 percent believed results were mixed due to factors such as leadership transitions, the lack of interest of some leaders to participate in leadership development activities, pressing demands from higher ups, lack of follow through and accountability, and others. Twenty-nine percent did not respond specifically to this question, and 7 percent did not think leadership capacity had been strengthened but clarified that this was “not due to EPIC’s efforts” and attributed it to the “personalities of the Team Leads.” “Early on, they were able to work on building an understanding of the value of building leaders at different levels and what kinds of skills and soft skills you want leaders to have and how to cultivate that. They used off-site retreats for the Office and others who work in education to come together to set the priorities for the next year. It’s been up and down because of the changes in leadership in the Office. The amount of time that is dedicated to Office culture varies. Also reacting to things that are going on in the world and making those pivots. They may pay attention to things they say are important to them and then shelve them.” — DDI/EDU Technical Specialist “I think overall it’s a dramatic improvement. EPIC deserves a lot of credit for it. We had leadership support for it: coaching, etc. Where we had problems were in conflict management, negative responses. Leadership had a very hard time. Everybody knew about behaviors and people got away with murder. You can complain until you’re blue in the face and nothing is going to change before [a new leader] who brought a 32 | EPIC EVALUATION REPORT USAID.GOV new style and doesn’t have that problem. Once they decide something, it moves forward. Before it was much softer. EPIC frequently was caught between a rock and a hard place. Through coaching, they were privy to a lot of information and they could submit their recommendations, but it was up to leadership to take action, and that didn’t always happen.” — DDI/EDU Technical Specialist In collaboration with DDI/EDU, EPIC developed and implemented interventions to strengthen the leadership and management capacity of DDI/EDU leadership at all levels by assessing training needs, designing and delivering face-to-face and virtual training, and providing coaching to support skills development and application of learning, and to assist with strategizing and problem solving. The following key interventions are presented in order of frequency as identified by interview respondents as having contributed to improvements in leadership capacity. The numbers in parentheses present the percentage of the total of 33 interview respondents (total includes 24 DDI/EDU staff and nine EPIC team interviewees) who specifically cited these interventions as having contributed to strengthened leadership capacity. • Coaching (45 percent) • Consulting/being a thought partner/informal conversations (42 percent) • Leadership Development Program/Professional Enhancement Series (36 percent) • Organizational Wellness and Staff Surveys (February 2018, May 2018, December 2018, April 2019, October 2019, and May 2020) (36 percent) • Support to Executive Management Team and their support to Team Leads (18 percent) • Office-wide off-site meetings (18 percent) • Process and system design (15 percent) • Cross workstream technical meetings (15 percent) • Staff meetings (12 percent) • Work planning (12 percent) • Strategic planning (12 percent) Interview respondents also mentioned other interventions less frequently (three or fewer times) that contributed to strengthened leadership capacity including communications and knowledge management products, leadership opportunities in planning large meetings and conferences, EducationLinks, pause￾and-reflect moments, stakeholder consultations, and webinars. Under this IR, we will discuss how EPIC implemented the Executive Coaching Program, its consultative/thought partner role, the Leadership Development Program, and how EPIC provided support to the EMT. We have discussed the organizational wellness dashboard and staff surveys, the off￾site meetings, and all hands and staff meetings in the previous section under OE IR 1.1. We will discuss 33 | EPIC EVALUATION REPORT USAID.GOV the remaining systems-related interventions (strategic planning, work planning, and cross team technical meetings) under OE IR 1.3 and IR 1.4. Coaching Program EPIC began planning for a coaching program in 2017 and launched the program in FY 2018, providing 64 sessions to 21 DDI/EDU staff that year, 108 sessions to 21 staff in 2019, and 99 sessions to 18 DDI/EDU staff in 2020. Coaching continues in 2021. A total of 51 DDI/EDU staff have participated in the coaching program over the life of the EPIC activity. Figure 4 presents the total number of coaching sessions broken out by year per coachee since the inception of the coaching program in 2018. For those who participated in coaching, the number of coaching sessions per participant ranges from 1 to 28 sessions over the life of EPIC, with an average of 8.76 sessions per participant. FIGURE 4: NUMBER OF COACHING SESSIONS BY COACHEE AND YEAR EPIC conducted coaching evaluation surveys to coachees to monitor progress of this intervention. TABLE 4: COACHING EVALUATION SURVEY RESULTS How satisfied are you with your coaching? August 2018 Average rating (n=9): 4.67/5.0 April 2019 Average rating (n=6): 4.67/5.0 September 2020 Average rating (n=5): 4.6/5.0 How effective has the coaching been for you? August 2018 Average rating (n=9): 4.67/5.0 April 2019 Average rating (n=6): 4.33/5.0 September 2020 Average rating (n=5): 4.2/5.0 34 | EPIC EVALUATION REPORT USAID.GOV Survey respondents found coaching satisfying due to the “top notch experience and skills” of their coaches, the “straightforward practical exercises and feedback” they received to address problems and challenges, and the safe and open environment in which they could candidly express their concerns. They appreciated that coaches challenged their thinking, asked probing questions to deepen their understanding of their challenges and how to address them, and that the coaches were “genuinely invested in [their] personal and professional growth.” Areas for improvement that respondents believed would make coaching more satisfying included extending the coaching period for a longer period of time (staff were generally limited to six sessions), finding more time in their schedules to meet more regularly, having the option to meet for more than an hour when dealing with particularly complex issues, providing coachees that are not clear on their coaching goals with more direction or structure, and asking coaches to relay in an anonymous way common issues that multiple coachees may be dealing with to Office leadership, when appropriate, so that leadership can help to address challenges at the team or Office level. Survey respondents shared the following examples of how coaching has helped them in their work: • Supporting them in making the transition to a new position or role in the Office; • Improved ability to give and receive feedback; • Improved ability to engage in difficult conversations; • Improved strategies to work toward professional and work-related goals; • Improved ability to establish boundaries to improve work/life balance; • Improved ability to negotiate “win-win” solutions; • Improved confidence at work; • Improved management and supervisory skills (e.g., how to have an effective conversation about expectations); • Improved ability to understand other people’s perspectives; and • Improved ability to make more thoughtful decisions. Sixty-seven percent of DDI/EDU interview respondents spontaneously cited coaching as being beneficial both personally for professional development and more broadly to improve organizational effectiveness of the Office. Some staff observed improvements in the performance of others who were being coached. A few staff noted that coaching works best with those who want to be coached and advised against imposing coaching on those who are not interested. Post-coaching surveys administered to coachees in April 2019 found that 83 percent of respondents were satisfied with their coaching and shared that it was an “opportunity for meaningful reflection.” One coachee commented that the coach “has been key to my sanity during difficult times.” As a result of the coaching sessions, 66.7 percent of respondents believed they were more effective in their job. Examples of how the coaching program positively affected the coachees included: 35 | EPIC EVALUATION REPORT USAID.GOV • Being “able to step back in a helpful way, see patterns, see different ways of looking at things, and ultimately how to act in a more thoughtful way versus react.” • “Begin[ning] to have a better understanding of how to work toward goals that are important to me and start asserting myself a bit more.” • “Having someone, other than my poor husband, listen to Office-related issues and professional concerns has helped relieve stress on my family.” Interviewees shared the following reflections on coaching and how this intervention has contributed to strengthening their leadership and management capacity: “One thing that has helped a lot and has ripple effects is the coaching and coaching of the leadership of the Office. They see that as really helpful. Where we have seen substantial changes is where the leaders can make well thought-out decisions for the benefit of the Office. Leadership having the support and then be able to address issues that impact everyone else. They’ve been doing that really, really well, and people know they will be heard, and something will be done. That’s a difference from previous leadership to now. It was the whole EMT.” — DDI/EDU Technical Specialist “Coaching – I myself have benefitted from that. I’ve seen people take advantage of that. I think that it just has created this idea that we are all trying to improve and none of us are perfect. I’ve gotten input on how I react with people I struggle to work with. Creates a culture of us all improving and that’s okay. I’m aware of more people taking advantage of it now.” — DDI/EDU Technical Specialist “A lot of work went into coaching. I haven’t seen as many changes from [some leaders]. I’m not sure why that is. Some people view coaching differently. [The Office Directors] saw it as a means to professional development. Some others saw it as a punishment. Some people didn’t go to their coaching sessions or appeared to not have learned from them. The issues are that there is no accountability for decisions. Avoidance of issues, not addressing things at that second [middle management] level.” — DDI/EDU Technical Specialist Being a Thinking Partner EPIC team members have applied their systems thinking, consulting, leadership and management training, and coaching skills to actively, empathically, and intentionally listen to and communicate with DDI/EDU leadership and staff to better understand them as people, their perspectives, and the dynamics of the context in which they work. A Retrospective Review conducted by Social Solutions International in 2018 stated that “EPIC has demonstrated its ability to be a thought partner to ED. Several staff noted that they would like EPIC to become more of a strategic advisor in certain instances.” The report went on to make recommendations that EPIC and DDI/EDU together clarify expectations for a strategic advisory role for EPIC, and for DDI/EDU leadership, “champions,” and staff to make new Office norms and strategic priorities sustainable by advocating for and institutionalizing new priorities and processes. Forty-two percent of DDI/EDU interview respondents for this evaluation stated that EPIC contributed to strengthening the Center’s leadership capacity by “being a thinking partner,” through informal conversations and its consultative approach. The EPIC team conducted regular assessments, gathered data through key informant interviews, observed team and Office-wide dynamics, and studied lessons learned from after-action reviews and from previous activities. EPIC then jointly with DDI/EDU identified trends, patterns, possible opportunities, pitfalls, and implications that could inform DDI/EDU 36 | EPIC EVALUATION REPORT USAID.GOV decision-making moving forward. “Being a thinking partner” is a more difficult intervention to quantify and describe as it is characterized by ongoing trustful relationships and open, honest conversations between EPIC team members and DDI/EDU staff that often take place informally or in confidential settings such as coaching sessions and one-on-one meetings, as well as in more public meeting spaces. DDI/EDU staff appreciated EPIC’s external perspective and role, while at the same time, EPIC staff was embedded enough in the Center’s day-to-day operations that they had a good understanding of the Center for Education context and the relationships. “They directly supported the development of the USG strategy. Helped us think through the timing, consultations. [EPIC] was phenomenal thinking through with us what the last strategy gains were so that we could meet the six-month deadline. We started from a different point than from zero. Thoughtful approach to how we could do it. Also helped with the policy. That’s a very direct impact that they had.” — DDI/EDU Technical Specialist “One of the things that the OE stream helped out with was the multiple transitions in leadership in the Office. It has been particularly helpful to have that overall view of what is OE. Helped new directors coming in to be partners to let them know about where some of the pain points are. Have an external viewpoint beyond the director talking to staff. EPIC can say what other directors have tried and how that went. Sometimes people have used that, and some haven’t.” — DDI/EDU Sr. Management “[Senior leadership] has taken issues seriously when they have been called to their attention. [EPIC] has been able to do that. [They] looked at the overall health of the Office, opinions of junior staff rather than just their direct reports to take a more critical look at the Office. Taken issues from what was best for the overall Office. I would give credit to the leadership for really addressing some serious personnel issues.” — DDI/EDU Technical Specialist “It made a difference in how we handled really tough issues. Before, we didn’t have a support area for that – just go into our Agency support that was really busy. It was enormously helpful to feel like we had support, somebody to bounce ideas off of and create solutions.” — DDI/EDU Sr. Management Leadership Development Program/Professional Enhancement Series EPIC designed the Leadership Development Program at the end of the first year and launched it in the second year of the project (2017) to strengthen leadership’s capacity to apply leadership and management skills to their work and to improve DDI/EDU’s organizational culture. Training topics included Social and Emotional Intelligence, Conversational Intelligence, Constructive Disagreement, Practicing Constructive Disagreement, Courageous Followership, Effective Teams, Giving and Receiving Feedback, Using MBTI to Enhance Team & Cross Team Functioning, Using MBTI to Overcome Stress, and the Leadership Pipeline. Sessions also were used as focus groups to develop the DDI/EDU organizational wellness model (Years 2-3). EPIC launched the Executive Coaching Program in 2017 (described separately) as an important complement to the Leadership Development Program. Table 5 presents Leadership Development Program courses in chronological order, the number of participants, and the average rating of the course by participants on a five-point Likert Scale (1 indicating that course objectives were not met to 5 indicating that course objectives were successfully met). Participation in the courses ranged from a low of 16 to a high of 31. Average course ratings ranged from 3.98 for the “Effective Teams” session to 4.71 for the Practicing Constructive Disagreement session. Over time, EPIC made adjustments so that courses were offered multiple times for smaller groups to accommodate staff schedules and increase engagement and practice time. 37 | EPIC EVALUATION REPORT USAID.GOV TABLE 5: LEADERSHIP DEVELOPMENT PROGRAM COURSES AND PARTICIPANT RATINGS (1-5 SCALE) NAME OF COURSE DATES NUMBER OF PARTICIPANTS DEGREE TO WHICH COURSE OBJECTIVES MET: 1= NOT MET TO 5 = SUCCESSFULLY MET Using MBTI to Enhance Team & Cross-Team Functioning 1/31/17, 2/2/17 27 4.35 Using MBTI to Overcome Stress 2/21/17, 2/24/17, 3/10/17 22 4.22 Courageous Followership 2/8/17, 2/15/17, 2/22/17 17 4.13 Giving & Receiving Feedback I 3/2/17, 3/8/17 16 4.42 Giving & Receiving Feedback II 3/17/17, 3/24/17 18 4.66 Effective Teams 4/4/2017 22 3.98 Constructive Disagreement 7/18/17, 7/20/17, 8/7/17, 8/14/17 31 4.45 Practicing Constructive Disagreement 8/24/17, 9/11/17, 9/14/17, 9/18/17 17 4.71 According to EPIC’s Annual Report 2017, because of the LDP, “Participants have reported that there are fewer micro-insults, more collegiality, greater cross-team work, and a more positive working environment overall.” Participants valued different aspects of each course depending on the topic, but there were some consistent themes across courses including appreciation for: creating space for honest and open conversations; ability to apply and practice skills; group work that allowed them to work through the concepts; focusing on real issues/topics relevant to their work; and getting to know each other better during these sessions. Themes that emerged in recommendations for future courses included: provide more examples and specifics; include more participant discussion/group work; focus on specific elements of the course in a deeper dive; more time for practice and feedback; more specific tools; adjustment of the session length, with most asking for longer sessions and a few asking for a shorter session in the case of the MBTI Overcoming Stress session; and having more staff present at the sessions. Regarding the last recommendation, while leadership deemed these sessions mandatory, some staff did not attend. 38 | EPIC EVALUATION REPORT USAID.GOV The main reasons cited by interviewees about why they believed there was lack of attendance included conflicting schedules, overload of work/no time, and lack of interest. See Annex 7 for more details including specific course objectives, participant comments on most beneficial aspects of each course, and recommendations for improvement. In 2019, EPIC rebranded the Leadership Development Program as the Professional Enhancement Series to address participation issues and respond to the Center’s needs and operating context. EPIC offered monthly non-mandatory one-hour sessions on topics that built additional skills and competencies to contribute to a healthy and effective workplace and achieve individual, team, and Center goals including Emotional Intelligence, Conversational Intelligence, Effective Feedback, Managing Change, and Managing Up. EPIC offered one-time coaching and follow-up discussions on these topics, developed Tips and Tools Playbooks on Effective Feedback and Emotional Intelligence for these sessions. A summary of the results of participant session evaluations using a Likert 5-point scale (1 low, 5 high) to rate the degree to which session objectives were achieved and facilitator effectiveness is presented in Table 6. TABLE 6: PROFESSIONAL ENHANCEMENT SERIES EVALUATION RESULTS (YEAR 4) PES SESSION NAME DATES NO. OF SURVEY RESPONDENTS/ TOTAL NO. OF PARTICIPANTS FACILITATOR EFFECTIVENESS DEGREE TO WHICH COURSE OBJECTIVES MET: 1=NOT MET TO 5=SUCCESSFULLY MET Emotional Intelligence 2/28/19 4/6 4.0 4.3 Conversational Intelligence 3/19/19, 3/28/19 5/7 4.2 4.1 Effective Feedback 4/25/19 3/3 5.0 4.7 Managing Change 5/30/19 2/4 4.5 4.3 Managing Up 8/7/19, 8/13/19 3/3 5.0 5.0 Three participants engaged in follow-up coaching on Emotional Intelligence, Conversational Intelligence, and Managing Change. Comments included: “I am already applying suggestions my coach made...To listen better, I am implementing other techniques shared such as writing down my thoughts while others are talking…” and “I applied coaching input in a meeting with a principal the following day, to great effect.” Nearly all participants who attended the LDP and PES sessions found them useful. However, the number of participants who attended PES sessions is relatively low. It is not clear from the data why attendance was so low despite EPIC having tried to address expressed barriers to attendance (e.g., reducing session time to one hour and offering training that was identified as important to participants). In subsequent 39 | EPIC EVALUATION REPORT USAID.GOV conversations with the EPIC team, they suggested that this series was not as well marketed or supported by Team Leads. Executive Management Team and their Support to Team Leads In Years 2 and 3, EPIC facilitated EMT retreats every two to three months for a half to full day. These retreats provided a space for the EMT to address strategic and operational issues, roles and responsibilities, Center and Agency transitions, present priorities and emerging challenges, staff and personnel issues, tensions, and good practices, etc. They often were an oasis for the EMT to pause and reflect outside the Office, away from the day-to-day pressures and demands. In key moments, EPIC was invited to EMT meetings to be a thought partner and resource for addressing Center issues. This rhythm slowed down in Year 4 due to leadership transitions but picked up again in Years 5 and 6. “The challenge that group [EMT] always had was that they were always managing up and didn’t have time to interact with the rest of the Office. That led to dissatisfaction. Feeling that decisions weren’t being made. They rose to the occasion at times. The changes in the group didn’t help…. They lost a lot of people that they brought on board. Younger staff who brought a lot of energy and willingness to work hard. A lot of them left. They had contract positions.” — EPIC Technical Specialist “We used to have periodic EMT meetings and that fell off. Then LeAnna came in and relationships got built and then LeAnna decided they would have EMT strategy meetings once a month on Friday afternoons to talk about the big issues and have outside facilitation to help them keep on track and stay focused, and to not forget what they have done and to follow up on the commitments they have made in the past. That has been huge. It’s been a big pivot for the EMT to speak with one voice and come together to make the big decisions.” — DDI/EDU Operations Specialist In October 2019, EPIC co-designed and facilitated an EMT workshop to review key moments, challenges, and changes in the Center over the preceding years; talk about visions, aspirations, and concerns; and examine the division of roles and responsibilities across the team. In January 2020, EPIC co-designed and facilitated another EMT workshop as a step in the co-creation of a shared and customized system of support by, with, and for Team Leads/Supervisors in alignment with 1) their individual areas of need and interest in professional development; 2) the organizational effectiveness strategy of the Center; 3) DDI; 4) the implementation of the new education policy; and 5) aid effectiveness as a sector. Six out of ten participants completed a workshop evaluation survey. Five out of six respondents thought the session identified practical and relevant ideas on how they can better lead their team/division. They also intended to apply the session information to their managerial work within the next three months. Half of the respondents felt more confident in their abilities to lead their team/division after the session. In June, July, and August 2020, the EMT strategy meetings provided the space for leadership to virtually convene and plan for the upcoming all-hands meetings in July, August, and September, respectively. “They [EPIC] have been instrumental in supporting the EMT. We couldn't do this work without them. They help us think through messages we want to present, prepare for all-hands meetings, pull together how to best use that time with staff.” — DDI/EDU Sr. Management As mentioned above under IR 1.1, EPIC administered, analyzed, and shared results of the organizational wellness and staff surveys with the EMT, which provided valuable input from staff to inform the EMT’s planning, decision-making, and actions. 40 | EPIC EVALUATION REPORT USAID.GOV “We did leadership training and coaching. For some of the individuals, it had an impact and it helped them, both the supervisors and supervisees. But there was a lack of somebody keeping their eyes on the target and reminding people to do that. Certainly, the health and culture improved. It’s not like it was at the beginning.” — EPIC Technical Specialist IR 1.3: DDI/EDU’S PURPOSE, OFFICE-WIDE PRIORITIES, OBJECTIVES, RESOURCES, PLANS, AND ACTIVITIES ALIGNED. Sixty-six percent of DDI/EDU interview respondents stated that the Center made notable progress on this intermediate result with EPIC’s support, 21 percent did not specifically address this question, and 13 percent believed that results were mixed, with progress in some but not all areas. “Things have come a huge, long way in making planning processes more clear and transparent and linked to strategy. The feedback that I’ve heard is appreciation of incorporating budget work in our work planning. EPIC support with work planning is much appreciated and has made a huge difference.” — DDI/EDU Sr. Management At the beginning of the EPIC project, the USAID Education Strategy and structure were organized around three clear strategic objectives, and there was outright competition among teams for attention, staff, and resources. DDI/EDU staff complained of siloing and insufficient resources to do their jobs. Following the end of the hiring freeze in March 2018, the Office grew from 28 to more than 40 staff. Later that year, USAID launched its reorganization process, and a new USG Education Strategy and USAID Education Policy were put in place, resulting in a shift in priorities and resources. The current USAID Education Policy is broader and more flexible than the previous strategy, with increased opportunities for cross-team collaboration, but also requires greater attention to setting Office priorities within that broader scope. Most DDI/EDU interview respondents saw the potential for more effective Center-wide programming under the new Education Policy, while some preferred the earlier Education Strategy’s objectives, which they believed were clearer. In light of USAID’s Journey to Self-Reliance, the shifting roles of country partners and Missions, the transformation of the Agency to its new organizational structure, and the broader Education Policy framework, the Center will need to continue to pay attention to how it is aligning its purpose, priorities, objectives, resources, plans, and activities. The Center also must ensure that short-term, medium-term, and longer-term goals and desired results are clear. “Making space for strategic dialogue. Daily work with work plans and making sure we’re linking all of that up. Coaching of the management team and trying to get us to look more strategically. Even helping us think through what questions we need to be asking as we go through this transformation about how we’re going to set ourselves up under DDI and linking that to higher Agency goals.” — DDI/EDU Sr. Management Within this dynamic and evolving context, EPIC contributed to the alignment of the Center’s purpose, priorities, objectives, resources, plans, and activities by collaborating with the Center on designing and facilitating interventions to: • Develop the Education Strategy and Policy • Streamline Center processes • Establish Center priorities (what was the Office of Education, pre DDI) 41 | EPIC EVALUATION REPORT USAID.GOV • Support transformation of the E3/ED Office to the DDI/EDU Center for Education We will now describe what EPIC did to contribute to these outcomes and the extent to which progress was made. Support the Development of the Education Strategy and Policy in Two Distinct Periods In its first year (2016), EPIC supported the development of the Education Sector strategy (as a follow-up to the 2011-2015 Education Strategy) by administering a survey to collect input from 225 participants from Missions, Bureaus, and other units. More than 80 Foreign Service Officers and more than 80 Foreign Service Nationals responded and provided specific feedback and requests for support. The EPIC team designed and facilitated an intensive, week-long Education Sector Strategy feedback and drafting session in May 2016 with DDI/EDU, Regional Bureaus, and other stakeholders. As it was reaching its final stage, this strategy was shelved, assuming the new administration would have its own priorities Later, the READ Act and the USAID Transformation efforts launched in the new administration led to a new USG Education Strategy and USAID Education Policy. In 2017, EPIC supported the inclusive development of the USG Education Strategy by providing knowledge capture, facilitation, logistics, and coordination with USG and USAID stakeholders. EPIC provided the Education Strategy team a synthesis of stakeholder feedback from more than 35 intra￾agency, inter-agency, and external consultations, webinars with more than 25 Missions, responses from the public comment period, written input from other stakeholders, and feedback from the USG Technical Working Group and USG Advisory group meetings—all to inform the development of the USG Strategy on International Basic Education. In 2018, EPIC supported the inclusive development of the USAID Education Policy by producing a synthesis of the feedback (October 18, 2018) from various consultations and a public comment period. “Developing the new strategy was a huge lift. From EPIC and management’s point of view, there was a lot of attention paid to this.” — DDI/EDU Technical Specialist “I know they [EPIC] contributed. A deep, long process to develop the new strategy. I was only involved in parts of it for my area of expertise. There was definitely a pivot that EPIC helped us go from one strategy to another.” — DDI/EDU Technical Specialist “We had to move toward what is our strategic vision, and that happened in 2018 with our new strategy. I think that was the biggest achievement. We had successful-enough management to feel good about our strategy and reposition the education work in the sector to reflect more of what’s going on now and a vision for the future. A broader vision than we had before.” — DDI/EDU Team Lead Streamline Center Processes Office-Wide Systems When EPIC first started, DDI/EDU identified the need to streamline some of its Office-wide systems. EPIC worked with DDI/EDU leadership and staff to map, develop, and document onboarding, travel, and clearance processes. As part of improving the onboarding process, EPIC identified with staff what was seen to be working in DDI/EDU and drew on other USAID sectors (Global Health) and private sector good practices to pull together a comprehensive on-boarding process for newly hired Education Officers and their supervisors. 42 | EPIC EVALUATION REPORT USAID.GOV They also developed a standardized template and process for team work plans. Some teams did no planning, and those that did, had their own templates. EPIC worked with the teams and the EMT to come up with a process and template that could be commonly understood and shared across teams and the Office. In 2017, EPIC produced and disseminated three playbooks in a “Tips and Tools Playbook” series that relate to improved Center systems and processes, including Effective Meetings, Email Etiquette, and Productivity in Any Workspace. (Other Tips and Tools Playbooks are mentioned under the Leadership Development Program/Professional Enhancement Series.) Work Planning & Teamwork To improve teamwork and the work planning processes that teams used to align their work to Center purpose, priorities, and objectives, EPIC designed and facilitated multiple team meetings and workshops over the life of the project. In Year 1, EPIC designed and facilitated work planning sessions with eight teams and facilitated meetings with the QUEST team to redefine their roles and integration into and influence in the other work teams. In Year 3, EPIC supported work planning retreats for five teams including the Reading Team; Program Team; QUEST; Education in Conflict and Crisis (EiCC); and Higher Education, Workforce Development/Participant Training and Exchange (HEWFD/PTE). In Year 4, EPIC designed and facilitated a two-day retreat for the Basic Education Division Team leadership, which helped inform the development of a draft Basic Education Division Vision and provided follow-up facilitation support during the Basic Education Division Quarterly Meeting. EPIC also supported work planning retreats for four teams including the PTE Team; Evidence Team; QUEST; and EiCC Team. This support included retreat design and planning, facilitation, knowledge capture, and logistics coordination. In Year 5, EPIC designed and facilitated a Higher Education/Youth Workforce Development (YWFD) Division Retreat to support work planning and other team effectiveness/performance priorities, including issues related to work/life balance, team building, roles in Mission support, points of contact/back-ups to countries and regions, and key events and partnerships to engage with moving forward. “We have put together systems for quarterly reviews and work planning. Before work planning was done using different forms and approaches. Now it’s better, we have a system for doing those annually – doing them and getting them approved. What we don’t have yet is really using those work plans, at least in my division. We sometimes have quarterly check-ins. Not sure if the work plans are being used to check in. That’s a shortfall or a gap. We have priorities but we don’t really come back to them to see how we’re doing on our priorities. COVID plays into that but even before then, we didn’t really use them – other things came up. It’s something we can definitely work on.” — DDI/EDU Technical Specialist “We, the leadership, didn’t hold people accountable. We should have had reviews to compare where people were and needed to be. The tools were in place but weren’t followed up enough. That is on us, not an EPIC thing.” — DDI/EDU Sr. Management “Get the work planning right. Need to be transparent about how and when you give input to the process. Especially when you’re looking at these cross-cutting areas. Need to have the technical experts weigh in and guide what the functions produce and what they do.” — DDI/EDU Sr. Management Support to Missions In Year 5, EPIC provided a strategy consultant to work with the Mission Support Working group to design and develop a new process and collection of tools to field and respond to Mission technical 43 | EPIC EVALUATION REPORT USAID.GOV assistance requests more efficiently and effectively (Educational Technical Assistance Support - EDTA). The consultant designed and implemented an EDTA pilot, developed a final report with findings and analysis from the pilot, and developed and presented recommendations for a global implementation of EDTA. The Center launched a six-month EDTA pilot with the DRC, Ethiopia, Mali, Rwanda, and Senegal Missions. The onset of COVID-19 in 2020 limited options for technical assistance and altered Mission needs, leading to an extension of the pilot timeline by two months. Three of the five Missions submitted four requests and of these, two were completed as intended and two remained paused due to the pandemic at the conclusion of the pilot period. Even so, in both cases, DDI/EDU staff have provided ongoing virtual support to these Missions. The pilot findings informed recommendations to improve the process, which include providing regular reminders to Missions about the mechanisms available to request support; identifying optimal meeting cadence for the EDTA Core Team; and that the EDTA Coordinator should maintain a request tracker. Both Missions that completed EDTA assignments rated 5/5 (completely) to the questions, “To what extent did Center staff deliver on the agreed upon statement of work?” and “To what extent did this EDTA process help you receive the requested services?” A total of 11 Mission staff were positively affected by the work. Organizational Learning To support organizational learning and adaptation, over the life of the project, EPIC designed and facilitated periodic reflection sessions including town hall meetings, all-hands meetings, and the annual two-day off-site meeting (described in more detail above under OE 1.1). The January 2020 off-site evaluation found that 77 percent of respondents felt that overall, the off-site fulfilled its objective of bringing DDI/EDU staff together on mutually important issues. When asked during the off-site retreat to share recent positive changes to their work, one DDI/EDU staff member wrote, “Strategic alignment and decision-making with EMT on how to move forward a complex issue.” EPIC also conducted evaluation surveys to gather participant feedback on most events that it supported and facilitated after-action reviews for most major DDI/EDU activities to learn what worked and what did not work to inform the design and delivery of similar future events. Establish Center Priorities In June 2017, EPIC designed and facilitated a two-day off-site meeting, which is described in more detail under OE 1.1. At this meeting, participants agreed that the Center and teams needed a shared set of priorities to guide their work. EPIC facilitated focus groups for all staff to participate and come to agreement on the Office’s shared priorities. Five were agreed upon: the Education Strategy, a shared learning agenda, Mission support, engaging and linking strategic partnerships and alliances, and organizational wellness/culture. In 2019, the series of all-hands meetings, annual off-site meeting, and EMT strategy meetings co-designed and co-facilitated by EPIC contributed to the creation of a new set of priorities related to the transformation to the new Center for Education. These priority initiatives and milestones were communicated to the Center in July 2019. “We’ve set up some good processes for how we can communicate with each other: schedules, checklists, etc. More processes that are structured that help us understand how to move forward. Those are easy things. The other things like what to prioritize is harder and needs more work.” — DDI/EDU Technical Specialist 44 | EPIC EVALUATION REPORT USAID.GOV “I think our leadership has worked hard to engage staff on what the priorities and focus should be and trying to align activities. Overall, this policy we have now is very broad. We are not directing the work from DC. The priority and objective of supporting the Missions has been made very clear, demand driven from the field has come through very clearly. My team is still wanting to have a little more control, more to say like we had under the previous strategy.” — DDI/EDU Technical Specialist Support Transformation of the E3/ED Office to the DDI/EDU Center for Education In Year 4, EPIC supported eight Center for Education Working Group meetings, which resulted in an informed consensus on the recommended mission, vision, mandate, comparative advantages, outcomes, and theory of change. In June 2019, EPIC designed and facilitated the annual off-site meeting. A takeaway from this session was the acknowledgement that the Center and sector needed a shared, strategic analysis of the partnerships necessary to carry out the new Policy and needed to rethink what a new Center of Education might look like and do differently to align with the new direction of the Agency. The guidance from the Agency and DDI indicated that conversations around structure, priorities, and staff roles and positions would have to be put on hold until other conversations and legal agreements between human resources and the labor union took place. In Year 5, EPIC designed and facilitated a virtual annual off-site meeting and five all-hands meetings that focused on the DDI transformation, matrix management, and plans to transition to the new Center for Education. EPIC worked closely with the EMT (which became the SMG), the Transformation Change Management Support Team (TCMST), DDI coordinator for transformation, and other key stakeholders to inform the development of each session’s agenda. The SMG and EPIC tried to keep pace with what permitted agenda items were and share that information with staff to have discussions. “There are now systems in place to plan. A question is whether people take that seriously. We went through months of planning but then people didn’t go back to follow up on the plans.” — DDI/EDU Technical Specialist “What EPIC has done a really good job of is put in place the right tools to help us execute. We are still trying to learn how to execute them.” — DDI/EDU Technical Specialist “The program functions so much better than it did four years ago. Roles are clearer. Operational plans, funding management processes are much better. I’ve really appreciated that.” — DDI/EDU Technical Specialist “When COVID-19 happened, they had to pivot. Over the last few years, it has been easier for the Office to do that as they began joint work planning and they learned the value of collaborative work planning to agree on priorities. It hasn’t been even, with some picking up things and then dropping them.” — EPIC Technical Specialist IR 1.4: CLA APPROACH STRENGTHENED, DOCUMENTED AND SHARED. Only 38 percent of interview respondents specifically commented on this intermediate result. Of those who did weigh in, 44 percent stated that a CLA approach was strengthened, documented, and shared, 44 percent said results were mixed, and one person (12 percent) said this IR was not achieved. While EPIC integrated CLA principles and practice into nearly everything it did in its work with DDI/EDU and intentionally learned with DDI/EDU from their shared experiences, it did not explicitly work with DDI/EDU to develop CLA objectives nor did they document a CLA approach, per se. Though it was the way the project worked, it was not always labeled or named as such. Respondents 45 | EPIC EVALUATION REPORT USAID.GOV noted that EPIC did thoroughly document all new Office work processes and practices, although these documents were not always used by DDI/EDU staff once they were completed. The EPIC team also documented each event that they supported, conducted after-action reviews (AARs) with their DDI/EDU counterparts following most events, used the results of the AARs to inform future similar activities, and were intentional in ensuring that the processes that they co-designed with DDI/EDU were inclusive and supported the participation of all involved. It is clear from the interventions described in the other OE intermediate results that CLA principles and processes were thoughtfully integrated into the many spaces that EPIC supported such as the annual off-site meetings, all-hands meetings, team retreats, and EMT strategy meetings as well as in the co-design of Office systems and processes such as team and cross-team work planning, strategic planning, priority setting, and others. “I do think the Office does CLA. We sometimes collaborate to an extreme. This goes back to leadership. The Deputy Director decisions couldn’t be made without consensus. It’s a great quality to want to achieve that, but not all decisions need to be made collaboratively. Fine line between wanting to be transparent and giving up authority. Some can abuse that.” — DDI/EDU Technical Specialist “There is a lot of adapting and sharing knowledge. There is a tendency to reinvent the wheel. I worked on the new education policy. It was helpful to have the institutional knowledge to build on. EPIC had worked on the strategy from a comms and engagement perspective. There was an intent to not recreate something that has already been done.” — DDI/EDU Technical Specialist “The word that describes the essence of this is the ‘relationship’ that we built together. That’s the key of it. At the very beginning, the project was conceived to be a co-collaboration, a co-creation. That was the brilliance of it. They got that right. Gave it a while before we came up with the MEL plan and what products we needed to deliver. They were insightful. Had a COR and back-up COR and Office Director who saw this as a real resource that we could build together. It’s about co-design and co-creation. They have to own this because it’s their change project. We’d put forth for discussion models like Kotter around change and evidence-based approaches ...McKinsey’ research on organizational culture, Harvard Business Review articles, the Leadership Pipeline, provide a book, trainings, and exercises on socio-emotional intelligence and articles, books, and individual and group exercises on polarity management... but it was less about that than entering into a dialogue of what they wanted and needed to do, and support that. Had two-day annual off￾site meetings to talk about how they were working together and what they were doing. Each of these events would generate action steps. Created a space where we could learn and have CLA going on. It gets into the culture…. When we came up with a survey and dashboard on org culture, they stepped up and attempted to have a CLA process on priorities, Office culture, decision-making. If there is one overarching theme, we created spaces and prepared people to have good conversations within those spaces and looked at problems and issues to come up with solutions. We didn’t call it CLA, but that’s what it was. We didn’t just look at deficits and problems, we looked at how to build on strengths. You could see the difference in the toxicity. Out of the annual meetings came the coaching program, leadership training in the first year, Office priorities in the second, USAID transformation in the third, etc.. We’d put those into action and begin to see improvement within the team and across team collaboration. Everyone, especially leaders, are responsible for the ‘commons,’ the Office culture and climate that in the end, is the greatest contributing factor in organizational performance.” — EPIC Sr. Management RECOMMENDATIONS AND LESSONS LEARNED BE PATIENT, PERSISTENT, REFLECTIVE, AND INTENTIONAL-STRENGTHENING ORGANIZATIONAL EFFECTIVENESS IS COMPLEX, REQUIRES COMMITMENT AND TRUST, AND TAKES TIME. As we have seen in this OE section, EPIC implemented many complex interventions to strengthen DDI/EDU’s culture, structure, systems, and processes within a complex, dynamic, and rapidly evolving 46 | EPIC EVALUATION REPORT USAID.GOV context. Strengthening DDI/EDU’s ability to successfully achieve its OE goals requires commitment by leadership, staff, and the EPIC team to try new ways of doing things and holding themselves and others accountable for following through, learning from experience, and adapting as needed. It requires a trusting relationship to have open and honest dialogue about what is needed, what is working, what needs to be improved and how. EPIC entered at a time when the Center’s culture was described by many interview respondents as “toxic” and “dysfunctional.” The EPIC team needed to take the time to build relationships with staff and demonstrate that they listened and were competent, reliable, and trustworthy. It takes time to plan for and make desired changes. Over the life of the EPIC project, DDI/EDU experienced multiple major challenges that had an impact on its effectiveness, and to varying degrees, EPIC’s ability to implement interventions. Progress was not linear. There were successes and setbacks. Adjustments in approach were sometimes needed to balance competing needs or perspectives that were, in fact, both valid and needed (e.g., attention to inclusive participation and efficient decision￾making). Overall, DDI/EDU’s organizational effectiveness has made good progress; there is more to do; and it takes commitment, trust, and time. When designing future mechanisms such as EPIC, it is important to commit to a realistic timeframe that takes these considerations into account. “There was a tendency to start from distrust rather than positivity. There were significant barriers to overcome to get to where we are now, which is more a place of starting from optimism and trust. We’ve seen them do this before. Can trust them to deliver, they are quality providers...There is that tendency to assume that if x person is managing it [as the COR], it will go well. Tendency to think if it didn’t go well one time, you have a mountain to climb to change that narrative. First impressions are really important. For EPIC, it is consistent demonstration of value that did it. There were some personality changes in the Office, which helped as well. They left, plus EPIC’s demonstration of what they can do over time did it. First impressions are still a big factor in where you start from in a position of trust.” — DDI/EDU Technical Specialist “We started with trying to create a common vision, what would success look like. Used OD best practices. Kotter, etc. Some people slowly began to see this. Others were only interested in their technical work, not interested in Office culture. There was a tension, paradox. One overall lesson learned is about our ability and inability to deal with these paradoxes, how to navigate them. Polarity is fascinating. Paradox is two polarities that are not independent – they are both/and. Working with the Office on balance – our ability to work with paradoxes is important.” — EPIC Sr. Management “Org health and culture is slower and more complex [to change]. It is a success story because we opened spaces for people to come together to talk about problems, solutions. Environment where nothing is off the table. Two-day annual retreats. Town hall meetings every quarter. Team retreats. If you look at the timeline and look at the first retreat to get to know each other, lots of new people to get more on the same page… It really is a day-to-day, week-to-week, month-to-month process. You’re part of the community but not from it. How you work with stakeholders to build up meaningful relationships and redefine how you work in this space. It’s an emergent strategy that requires relationships and active diplomacy with team leads and the Education Sector Council, rather than event focused. Retrospectively, maybe I should have done more of that.” — EPIC Sr. Management “In order to have gains in org culture, you have to have sustained commitment from leadership. The Agency has put so many things in front of the leaders that they weren’t always able to sustain their attention. How do you maintain a positive environment with more people and more ways of being when they grow? I think this was right in TRG’s wheelhouse, but they [DDI/EDU leaders] weren’t always ready for these ideas.” — EPIC Sr. Management 47 | EPIC EVALUATION REPORT USAID.GOV “My key take-away from the project was that support of the Office Director and Management Team is important. It was a slow start, which isn’t unusual with these big contracts with everyone trying to get their feet on the ground and figure out what they are doing. Once management was vocal about their support, it helped. There was a shift when Evelyn had moved on and there wasn’t support especially for OE, which was essential. Management didn’t prioritize it at the time, not to say that progress wasn’t made. It just took us a couple of steps back. There has to be real support and accountability. Without that, things crumble a little bit.” — DDI/EDU Technical Specialist “Be invested in change. If you’re invested in status quo, don’t do the work. Be willing to take risks. When it goes south, be willing to clean it up.” — DDI/EDU Sr. Management CREATE A SHARED VISION AND ALIGN TO IT, ON AN ONGOING BASIS. Creating a shared vision and aligning to it is a known best practice, as confirmed by EPIC’s experience with DDI/EDU, and is worth remembering and paying attention to throughout all current and future OE strengthening efforts. One area of improvement mentioned by an interview respondent was to have leadership ensure that work plans reflected DDI/EDU priorities. Another area of improvement highlighted by interview respondents is that once work plans are completed, it is important to regularly use them as a tool to determine progress made, to hold each other accountable for implementing the plans, and to modify plans, when necessary. This practice was evolving over the years and was affected by the COVID-19 pandemic. Now with the launch of the new Center, the new work planning process seems to be bringing together and addressing the lessons learned. “Vision sharing – the one thing we did that was most effective was taking the time out to do the vision sharing and planning. That was critical and effective. We ended up in E3 being considered the most effective unit due to structure.” — DDI/EDU Sr. Management “We were big on evaluation. Once vision was clearer, we set up an ongoing evaluation process that was a lot easier.” — DDI/EDU Sr. Management “What never solidified around them was enforcement of the priorities in work planning conversations. It was improving, but leadership to look across the work plans would have been good to say yes or no, this is a priority or not. People are overworked and that continued. It affects the process side of things. They aren’t able to keep up with all the things they said they will do. We work with them to set a timeline and it gets thrown out a lot because they spread themselves too thin. It affects morale and the quality of their work because they don’t have the time to provide input in all the places you would like to.” — EPIC Team Lead “Lately over the last year, year and a half, we were asked to do a work plan but then we never follow up on it. It sits on the shelf. We don’t know if there will be follow up on the work plans we put together last year and whether there will be regular portfolio reviews. We used to have them, but with COVID-19 everyone is in survival mode. It was starting to be a regular practice, but now it’s fallen off. There was a working group that looked across work plans across the Office. Those were good meetings, but we haven’t had them in a while.” — DDI/EDU Technical Specialist INCLUDE COACHING IN FUTURE PROGRAMS. The evidence from EPIC’s experience with DDI/EDU summarized above in Section OE IR 1.2 supports coaching as an effective intervention to strengthen leadership and management capacity. We recommend that coaching be included in similar programs in the future. “Coaching is invaluable. I’ve been doing it for 10 years. As you make org decisions, you need to have a coach to tell you about wrong moves and how to do better.” — DDI/EDU Sr. Management 48 | EPIC EVALUATION REPORT USAID.GOV “It really depended on leadership. With coaching, leadership could get coaching, but if leadership wasn’t open to change, why are they being coached in that area?” — DDI/EDU Technical Specialist “It was hugely helpful to have a coach. Having been in the field, few of us had to manage a team this big. Coaching has been hugely helpful.” — DDI/EDU Sr. Management “ALWAYS REMEMBER YOU ARE WORKING WITH REAL PEOPLE.” EPIC’s “human- and system-centered” approach to organizational effectiveness was evident in how they worked with DDI/EDU staff and the interventions they designed, facilitated, and supported. They took time to get to know DDI/EDU staff to understand them, their roles, perspectives, interests, skills, and aspirations. They sought staff input to custom design interventions and gathered feedback to learn how staff experienced them to inform future interventions (e.g., organizational wellness dashboard, off-site meetings, training courses, etc.). “Always remember you are working with real people. It’s all about people. And that’s what EPIC got. They didn’t see it as widgets, 20 manuals, processes, and procedures. It was an in-depth look at the humans as they come in and flow. I’m doing this now in my current job and will continue to do this. Got more out of this than my college degree. And I’m still hearing from people who have left. That is pretty cool.” — DDI/EDU Sr. Management MAKE LEADERSHIP DEVELOPMENT TRAINING RELEVANT AND REAL. Participants of both LDP and PES courses indicated in evaluation surveys that courses that included real case examples or real challenges that participants brought to the course and small group interaction with their peers were more helpful and relevant to their work. We recommend that future interventions consider these findings, as well as EPIC’s positive experience integrating “soft skills” into technical training so that participants learn how to apply leadership, management, and other “soft skills” to real challenges they are facing at work. EPIC has been moving in the direction of providing more microlearning products in other workstreams to support just-in-time learning and as job aids. This approach may be further explored for leadership and management capacity development. Real challenges that participants bring to training sessions can also form the basis for the development of a case study library that could be useful to others in the future. CO-CREATE, BE A THOUGHT PARTNER AND PARTNER IN PLANNING. Throughout the interviews and data reporting, DDI/EDU staff expressed the value of working with EPIC to co-create the changes that DDI/EDU desired. This collaborative approach ensured that DDI/EDU’s needs and perspectives were addressed, and that DDI/EDU could take advantage of EPIC’s organizational development, facilitation, training, and coaching skills and external perspective in developing solutions customized for the Center. It also served as an experiential way through modeling for DDI/EDU staff to develop their skills in facilitating co-creation processes. Before COVID-19, the fact that some EPIC staff had regular access to work alongside DDI/EDU staff in the same Office facilitated informal conversations as well as more formal meetings, during which EPIC staff could listen to DDI/EDU staff challenges and be a thinking partner in helping them to address them. EPIC also advocated for greater input from voices in the field to be co-creators with DDI/EDU to provide their perspectives on how DDI/EDU can best support their efforts. We recommend that DDI/EDU continue to work with similar mechanisms in this way and that they intentionally bring more field voices into their planning efforts, as appropriate. 49 | EPIC EVALUATION REPORT USAID.GOV “I think being part of planning and not just ‘here are your marching orders’ has really contributed to the effectiveness of this Office. Having the confidence to allow that second group of the contractor to come in and be integrated into the planning in partnership is pivotal. It allows space for better collaboration, more effective communication between the USAID Office and their contractors. Clear roles and responsibilities. Taking the time to create processes that work for everyone. I created processes and trackers. One thing I wish is that I had a counterpart to collaborate with to determine who needs to be involved, who talks with whom. There’s a tendency to procrastinate. The more you collaborate and plan at the front end, the less you are overwhelmed at the other end.” — EPIC Operations Specialist “A lesson learned that I’m coming to understand is the value that OE provided in always advocating for more voices from the field. That is something that is an Office culture issue that is bigger than any one portfolio. It’s an important message to be coming from OE. I’m only now realizing that it needed to be coming from them. I would want to be explicit about that being the case. More input from the field would be better. That’s the link that I wish we could have had. It’s a lesson learned.” — EPIC Team Lead “I worked with EPIC as an extension of my team. They weren’t just service providers. They knew what was going on in the Office, what the challenges are, co-creating solutions, having really open conversations about people and strategic directions, identifying where things were untenable. It was huge for EPIC to understand what we needed, and they were pretty understanding of why things were hard sometimes. They were happy to provide solutions instead of questioning us...In some ways, the OE workstream was situated to allow the project to have a deeper insight into relationships and how the workstreams operated. That made a difference. The number of coordination meetings and reflection meetings at all kinds of levels, all-hands meetings, retreats, workstream meetings, weekly meetings with Tom and Sharon solely dedicated to the management and partnership part of it.” — DDI/EDU Technical Specialist “Agree on norms for how engagement should happen with the rest of the Office and how adaptive management is going to happen, from budgeting to assessment of deliverables. Things happened organically over time, but if you don’t have a COR who knows how to do that, you need to have an explicit process. Put in resources for strategy and reflection monthly at a minimum.” — DDI/EDU Technical Specialist “Something like this is only as good as you manage it to be. It’s an opportunity. It takes management intention and attention to use it well. The will was there. Our ability to manage it well rose and fell at times. It was kind of incestuous as people air dirty laundry and contentious issues. When you look at the tradeoffs, it was worth including, and made a difference in how we handled really tough issues. Before, we didn’t have a support area for that — just go into our Agency support that was really busy. It was enormously helpful to feel like we had support, somebody to bounce ideas off of and create solutions.” — DDI/EDU Sr. Management INTEGRATE COLLABORATING, LEARNING, AND ADAPTING PRINCIPLES AND PRACTICES TO IMPROVE ORGANIZATIONAL EFFECTIVENESS. It is clear from the many interventions that EPIC implemented to strengthen organizational effectiveness and how EPIC implemented them that CLA principles and practices were fully integrated and modeled in its work with DDI/EDU, and that these principles and practices helped the EPIC team and DDI/EDU collaborate, learn, and adapt (see OE IR 1.4). EPIC and DDI/EDU did not co-create a specific CLA plan, with CLA objectives and explicit descriptions of their approach with a MEL plan to support measuring and documenting progress in these areas. It may be helpful in the future to be more explicit about the aims related to CLA itself in terms of DDI/EDU’s organizational effectiveness, which would provide the Center with a clearer vision of what successful CLA looks like in their context and provide a foundation upon which to further develop these practices. 50 | EPIC EVALUATION REPORT USAID.GOV PROFESSIONAL DEVELOPMENT SO 2: IMPROVE USAID EDUCATION STAFF’S TECHNICAL CAPABILITY AND SKILLS TO ACHIEVE THE AGENCY’S EDUCATION SECTOR GOALS The strategic objective of this workstream was to improve USAID education staff’s technical capability and skills to achieve the Agency’s education sector goals through a quality continuum of learning (COL) reaching all actors in the education ecosystem, improving core competencies of DDI/EDU and education field staff, and strengthening, documenting, and sharing a CLA approach. Notable functions of this workstream included identifying professional development and technical capability needs of USAID education staff, designing courses in collaboration with the Professional Development Working Group (PDWG) to address them, and providing technical and design assistance on PD products and events to offer education staff more holistic training opportunities. While many of the products from this workstream were well-received by the target audience, its continued success depends on the workstream’s ability to reflect emerging thinking around individual, team, and Mission core competencies required for successful implementation of the USAID Education Policy, as well as commitment to a COL strategy that is accessible to the widest possible audience. Through EPIC’s performance period, this workstream produced ten deliveries of core courses, three face-to-face learning opportunities, two distance learning opportunities, thirty-three webinars, and seven programming videos. A full breakdown of these products is shown in Annex 8. Other key interventions enacted by EPIC in this workstream included a meta-analysis of the education sector’s professional development needs, the development of a work plan that supports a COL (focusing on various modalities for learning), participation in the PDWG, a learning priorities survey, planning and implementation of the Global Education Learning Series, and a pivot from face-to-face trainings to virtual events. “The PD workstream is finally reaching the point where it’s fulfilling the vision... In 2016 we had the two [courses] and they went really well. But that is only 15-20 people two times a year. People came in and left. This wasn’t a PD strategy. We needed to provide a myriad of opportunities for people to learn. The Road to Learning. They agreed in principle to a myriad of options: podcasts, trainings, articles, etc. What is the range of topics and levels of skill...Then the new policy and strategy came up. Then COVID happened, which blew the GEC out of the water... We went from a 600 people conference to a global online education learning series over several months. We’re still not at a coherent well thought-out curriculum moving away from individual competencies to team and Office competencies. There are knowledge and skills that we need to get better at, like systems thinking and tools. COVID pushed people to say that we can’t do face-to-face. We will come up with a better array through which we reach hundreds of people over time. One accomplishment [in the] summer of 2019: the PD working group began to come together again. We worked with them on a PD strategy...I’m hoping that as a legacy that we can leave them with a PD strategy rather than a series of events.” — EPIC Sr. Management The following sections describe the interventions that EPIC co-designed and implemented with DDI/EDU, the outcomes they produced that contributed to each PD IR, and the factors that facilitated and hindered progress. Key events over the life of the project in this workstream are presented in Annex 4, Table 2. A summary of the factors that facilitated and hindered the success of EPIC’s PD efforts is presented in Annex 5. 51 | EPIC EVALUATION REPORT USAID.GOV IR 2.1: QUALITY CONTINUUM OF LEARNING STRATEGY THAT REACHES ALL ACTORS IN THE USAID EDUCATION ECOSYSTEM INSTITUTIONALIZED. Achieving results in this IR proved to be challenging due to the Office not having up-to-date core competencies articulated and agreed upon. EPIC tried to clarify which competencies the Office wanted to use, but the Office had more pressing priorities. EPIC staff commented that it is hard to develop an effective learning strategy without clear guidance on the knowledge, skills, and abilities that Education Officers need to do their jobs effectively. Many interview respondents noted that it was difficult to achieve these results without strong leadership and a champion for PD that would prioritize their achievements. Nevertheless, the PD workstream did benefit from strong leadership in Year 5 and continued to make significant progress toward institutionalizing a quality COL strategy. Most of the individual, team, and Center/Sector core competencies can be found and derived from the USG Education Strategy and Education Policy priorities and principles. “On the PD stuff, I think what’s happened in the last six months has been progress. The working group has better engagement, clearer tasks and asks in large part thanks to the work of Shira, Joanna, and Daryn.” — DDI/EDU Technical Specialist User-Focused COL Strategy To create a comprehensive COL that was relevant and effective for all 250+ worldwide actors in the USAID ecosystem, the PD workstream undertook continuous assessments of the needs and perspectives of staff (U.S. and overseas, across all points in their careers) and incorporated data on adult learning strategies. In Year 1, 26 staff representing Africa, Asia, Latin America and the Caribbean, and Washington participated in focus groups to support the development and creation of PD materials. In Year 2, the PD workstream produced a meta-analysis report of education sector needs, which analyzed ten resource documents from 2016. These documents ranged from surveys of 225 education stakeholders to Year 1 course evaluations. The PD Workstream also produced a booklet on Effective Learning through Different Modalities, which identified the most current research and thinking on how adults learn. This report was the blueprint for the COL strategy and offered critical guidance to the PDWG and EPIC staff about which methods were best to develop competencies needed by Education Officers, especially modalities that could more effectively reach the 250+ staff through timely, relevant, and cost-efficient means. In Year 4, the PD workstream conducted an Education Sector Learning Priorities survey to ensure that all learning opportunities were tailored to respond to these findings. The responses (68 respondents, the majority from Missions) were analyzed and summarized in a report that was presented to the Education Sector Council and used to inform the PDWG strategic plan and Global Education Conference (GEC) learning. In Year 5, the PD workstream supported the DDI/EDU Office in substantially redesigning the in-person, COVID-postponed GEC to a fully remote Global Education Learning Series (GELS). This was done by supporting the 12 member PDWG Session Design and Development committee in five monthly meetings and designing PD materials that would fit a redesigned COL that more closely aligned with the 2018 Education Policy principles and priorities. As a result of these activities, the EPIC PD team has made significant progress in developing and implementing a COL approach rooted in the principles of adult learning; translating learning into practice and change; and implementing collaborative, Mission-centered course design and delivery. PD products were designed to include a variety of modalities, resulting in a blended learning approach that featured face-to-face courses, virtual distance learning courses, interactive webinars, and on-demand videos. The PD workstream collaboratively created these materials with Missions and Regional Bureaus. Field staff were included in design and/or served as subject matter experts (SMEs). Core course topics 52 | EPIC EVALUATION REPORT USAID.GOV were based on results gathered from comprehensive survey activities as well as feedback specific to each training. In particular, participants registered in training courses were interviewed to gauge levels of experience and interest, which shaped course facilitation. Webinar topics were selected and designed based on questions posed by the field and/or Regional Bureaus during training courses, thus ensuring that materials are responsive to the expressed needs of a more global audience. Core courses were refined and updated using participant feedback after each iteration, ensuring that the content and presentation was as relevant and useful as possible. Learning Opportunities Over the lifetime of the project, this workstream produced ten deliveries of core courses, three face￾to-face learning opportunities, two distance learning opportunities, and thirty-three webinars. (See Annex 8 for a complete list.) In addition, the workstream engaged in ongoing collaboration efforts with the KM workstream to produce seven programming videos and support creation of learning materials through the EducationLinks website, which supported the COL. Core Courses and Face-to-Face Learning Opportunities Education Overview Course (EOC) The PD workstream used feedback from the Year 1 focus group to design the Education Overview Course. The course focused on orienting Education Officers to the new USAID Education Policy and help them learn to shift their work, where necessary, around its principles, priority technical areas, and new practices. It also integrated the desire of field staff to share experiences and network. The first course was offered in September 2016 in Washington, DC, to 29 participants. EPIC then partnered with the DDI/EDU PDWG to design and deliver a second iteration of the course in May 2017, which was completed by 19 participants. The third iteration of the course was designed as a distance learning opportunity to accompany the virtual GELS in 2020. This design more explicitly included the course as a part of the updated COL strategy that included pre-learning modules and conference activities, which were then built upon in the course. Participation varied in this iteration: 96 participants completed at least one module, but only 14 completed the entire course. Results from the course are further discussed in IR 2.2. Effective Education Portfolio Management Course (EEPM) In Years 1 and 2, EPIC partnered with the PDWG to design and deliver the EEPM course, which had last been offered a decade previously. This course focused on USAID’s Program Cycle guidance and tools and was delivered eight times over the lifetime of the project (four times in Washington, once in Bangkok, once in Frankfurt, once in Accra, and once virtually). The first iteration, offered twice in Year 2, included feedback from the Year 1 focus group, and included opportunities for participants to hear directly from ED staff working in the field. The course design was then revised and updated based on course evaluations from participants and input from DDI/EDU, Mission, and Regional Bureau staff. This version was offered three times in Year 3, in Washington as well as in Bangkok and Accra. These updates included the creation of new sessions focusing on leadership skills, emotional and conversational intelligence, and strategic influence as well as relevant aspects and components of the Education Strategy and Policy and how it influences and guides the work of Education Officers. The course was again updated in Year 4, reflecting feedback and suggestions from previous course evaluations such as a greater focus on adaptive management and flexible mechanisms, incorporating more participant-led case examples, and less time spent on project appraisal documents. Starting in Year 4, EEPM participants began receiving follow-up surveys between four to nine months post-training to assess the extent to 53 | EPIC EVALUATION REPORT USAID.GOV which the course learnings had affected their work. The seventh offering of the course was held in Washington in Year 4 (15 participants). The final redesign of the EEPM course occurred in Year 5, when the course was reconfigured as a virtual distance learning course due to COVID-19. The course shifted from a one week, in-person training to a two-week, virtual, instructor-led course. This update also included pre-recorded sessions from technical SMEs. Twenty-nine staff participated, indicating that the virtual course and facilitation allows for a higher number of participants. Results are further discussed in IR 2.2. Education in Countering Violent Extremism The Education and Countering Violent Extremism (ED-CVE) course was designed in response to a special request from the Asia and Middle East Regions and the Office of Afghanistan and Pakistan Affairs (OAPA) to address the global challenge of violent extremism through an education lens. The first course was delivered in December 2018 (Year 3), after the EEPM Asia training. The second course was delivered in March 2018, the week before the EEPM Middle East training. The ED-CVE course benefited greatly from the positive relationships built during the design and delivery process across sectors (Democracy and Governance and Education), across agencies (USAID and Department of Defense) and within the Education Sector Council (ED and Regional Bureaus). Engagement across these stakeholders included discussions around current USG and USAID policies, strategies, and research on CVE. The field voice was well represented and integrated in the course design as field staff led various course sessions and gave input into future regional approaches to education and CVE. During the course, participants examined critical evidence, experiences, and analytic gaps related to the drivers of violent extremism and the role education can play in mitigating it. The ED-CVE course was scheduled to be offered again in December of Year 4 in Ghana but was cancelled due to low attendance. It was instead offered in May of Year 4 in Washington, DC. Adjustments were made to the course using participant feedback collected during the previous courses as well as in six-month follow-up surveys, tailoring the case examples to reflect issues and dynamics most relevant to the participants. Results are further discussed in IR 2.2. OUTCOMES Core courses had 290 unique participants over the lifetime of the project, and about a third of these participants took more than one course (n=85). All core courses received an over 85 percent satisfaction rate from participants, and participants had an average knowledge increase between 20-30 percent. See IR 2.2 for a more in-depth discussion of course results. Distance Learning Opportunities Gender Distance Learning Course In Year 3, the PD workstream designed and built a gender distance learning course in collaboration with USAID, as well as accompanying products such as Gender How-To Notes and a facilitator's guide. The course was intended to provide USAID education staff with the skills they need to design and implement gender-responsive education programs, by conducting a focused gender and education analysis incorporating the three principles of equity, safety, and empowerment. The course was approved and posted on USAID University in August 2018. Unfortunately, EPIC staff were informed that due to changes in priorities because of the transition to the new administration, the course would no longer be accessible. Positive Youth Development The PD team also initiated the development of an Agency-wide Positive Youth Development (PYD) online course in the last quarter of Year 3. The course was a two-module distance learning format. The 54 | EPIC EVALUATION REPORT USAID.GOV first module, PYD 101, focused on defining positive youth development and why it is important and necessary to integrate in development programming. The second module, PYD 201, presented tools and strategies to apply PYD principles and concepts to evaluate the effectiveness of a range of program and activity design scenarios. This course was offered on USAID University, which meant that EPIC did not have access to information on course participants, completion rate, and effectiveness. Online Learning Modules In Year 5, the PD Working Group designed the first virtual distance learning format of the EOC (described above). Of the 20 learning modules, nine foundational modules and four soft skills modules were specifically developed for the EOC, to be offered to registered USAID staff. EPIC co-developed the remaining seven online learning modules focused on the technical areas prioritized in the 2018 Education Policy: Disability Inclusive Education, Social and Emotional Learning and Soft Skills, Reading and Literacy, Youth Workforce Development, Pre-Primary Education, Higher Education, and Education in Crisis and Conflict. These modules contributed to the education sector’s central repository of resources and served as a quick orientation to those new to the sector. The modules are self-paced, guided learning experiences that provide Education Officers with foundational information on a specific technical area—terms and definitions, research, case examples, and best practices. Five of the seven modules were released and shared as pre-learning for the 2020 GELS, allowing participants to have a shared, foundational understanding and vocabulary around these technical areas. Finally, the seven modules also served as the content building blocks for the technical unit of the new online EOC. The modules had very high engagement levels: Disability Inclusive Education (244 views), Social and Emotional Learning and Soft Skills (277 views), Reading and Literacy (221 views), Youth Workforce Development (206 views), Pre-Primary Education (372 views), and Higher Education (135 views). An online evaluation form was also offered to participants at the end of each module. At the time of this report, 82 total evaluations were completed, with the following breakdown: Pre-Primary Education (n=22), Youth Workforce Development (n=18), Social and Emotional Learning and Soft Skills (n=15), Disability Inclusive Education (n=13), Reading and Literacy (n=10), Higher Education (n=4), and Education in Crisis and Conflict (n=0). Across modules, participants overwhelmingly found the information to be presented in clear, effective, and engaging formats (84 percent), found the content informative and relevant (88 percent), and planned to apply or share the knowledge and skills in the near future (94 percent). Multimedia PD Activities Webinars Throughout the lifetime of the project, the PD workstream produced and delivered 29 webinars to meet education staff needs, generating a total of more than 950 views. The number of unique participants increased each year, demonstrating that the webinars were increasing their reach and contributing to the COL. The topics for these webinars were generated in response to feedback gathered from education staff through the PDWG, in focus groups and interviews, during face-to-face courses, and in post-course evaluations. The first webinar, Flexible Instruments, was designed in response to PDWG participants conveying the great deal of interest in this topic and had 24 participants. In Year 2, learning from the meta-analysis of PD needs informed the design of two webinars, Evidence in Early Grade Reading (nine participants) and Host Country Engagement and Sustainability (20+ participants around the globe, including two Mission viewing parties). In Year 3, 105 unique participants from five regions joined at least one PD webinar. The PD workstream co-designed 55 | EPIC EVALUATION REPORT USAID.GOV and facilitated eight webinars on the topics of host country engagement and sustainability, evidence in various education-related topics such as disabilities and early grade reading, education financing such as costing models, and government-to-government agreements. Topics were derived from the needs and interests that participants expressed in the EEPM courses. Participant breakdown is shown in Table 7. In Year 4, the PD team worked with DDI/EDU staff to design and facilitate webinars that engage participants, particularly Mission staff, to share questions, best-practices, and approaches pertaining to a given thematic area. This marked an increased focus on aligning topics to current needs of staff, and less focus on themes of evidence and sustainability in education. This focus included ensuring that presenters and participants draw from different Missions to facilitate cross-Mission exchange and learning around various education-related topics. The PD workstream collaborated with the KM workstream to maximize promotion and attendance, coordinate messaging, co-design, and facilitate 11 interactive webinars, on topics such as Strengthening the Service Delivery System for Inclusive Education and Performance Plan and Report to the Field, LGT Partnership Presentation, and Social and Emotional Learning. Year 4 webinars were attended by 118 unique participants. Year 5 webinars fell into two categories: multimedia products created as part of the GELS (described below), and COVID-19 related webinars. The PD workstream responded to requests from the COVID-19 Working Group to support design, production, and knowledge capture for seven internal webinars related to COVID and education. Notably, the PD workstream also designed a standard feedback tool for these webinars to capture impact on participants. Two hundred eighty unique participants from five regions joined Year 5 webinars. On-Demand Videos EPIC designed and completed seven on-demand videos on thematic areas of education programming. The first video, Inclusive Education (129 views), highlights best practices in inclusive education programming. The second video, Integrating Gender into Education Programs: Best Practices from Honduras (62 views), showcased best practices from three implementing partners working with USAID/Honduras. The third video, Disability Inclusive Education under USAID’s Education Policy (192 views), provides relevant resources, such as the Disability How-To Note and the Literacy for All Toolkit. The fourth and fifth videos on the Rapid Evaluation and Risk Analysis tool (59 views) and Social and Emotional Learning (197 views) were filmed during the ED-CVE course. DDI/EDU technical SMEs in these areas were interviewed and videos were shared in the Education newsletter. PD also collaborated with the KM/Comms workstream to produce two videos: one on the principles of the Education Policy and one on the priorities in the Education Policy. In Year 5, the PD workstream continued this multimedia PD learning activity by developing the Online Learning Modules (mentioned above), and repurposing input from the Year 4 training courses into blended learning modalities in concert with the GELS. More information on the GELS is provided below. Global Education Learning Series The PD workstream put significant effort into planning for the 2020 Global Education Conference. The PD Team worked closely with the GEC Session Design and Development (SDD) committee by facilitating weekly SDD committee meetings, designing processes, facilitating learning opportunities, developing knowledge products (e.g., toolkits, graphics, agendas, scoring rubric, webinar recordings, and PowerPoint presentations), and designing and structuring the final GEC program. When the decision to cancel the in-person conference was made due to COVID-19, an opportunity emerged for EPIC to help the Office pivot from the GEC to an online GELS. The PD team worked through the summer to transform what was supposed to be an in-person conference into an online Global Education Learning 56 | EPIC EVALUATION REPORT USAID.GOV Series that offered a greater diversity of learning modalities and knowledge products to fit different and emerging professional needs, preferences, and learning styles. In total, the workstream designed, facilitated, and coordinated 13 learning opportunities: four panel discussion webinars (457-1,008 participants), seven short narrative videos (66-216 participants), and two Tea with a SME events (up to 160 participants). Overall, these learning opportunities were well received: 83-90 percent of participants found the sessions relevant (according to a participant survey). However, the after-action report noted that participant engagement was a big challenge as compared to an in-person meeting. The virtual format came closer to demonstrating a continuum of learning strategy that is accessible to a larger audience and provides more timely resources that staff and partners can learn from and refer to outside of synchronous courses. Interview respondents who were aware of recent developments in the PD workstream were optimistic about the positive momentum of the GELS as a blueprint for future PD materials and saw potential for this to grow into a more comprehensive COL strategy. “I’ll speak generally, the development of the GEC. TRG played an instrumental role. We were extremely impressed with everyone who played a role. Everyone was extremely upset when we had to cancel it due to COVID. The quick pivot to the online learning series was good. Had a launch two weeks ago. I was reviewing the questions from all the partners that came in from that. We had a training this week online. All these are ways to help pivot from a face-to-face conference for PD opportunities to virtual support in the field.” — DDI/EDU Sr. Management “Now for the GELS and EOC online, those are the key characters who come into play in the key activities. We’ve set a good standard for the continuum of learning. Not just deliver a course and goodbye. Get feedback, continuous tweaking to get better, relates to CLA. Collaboration with the KM folks to do photo booth, video booth to capture an impact story and good things we hear from the participants. How do we hear from field folks rather than being Washington-centric. How to capture that information as well to reach a wider audience. That’s been going well. In the pivot to virtual, with the EOC online course, how can we make this asynchronous course as interesting and beneficial. How to keep in touch with participants to see how they applied what they learned and support them in continuing learning...Be responsive to the field demands.” — EPIC Team Member OUTCOMES The various modalities and consistent adaptation of PD materials to provide more relevant, accessible, and high-quality learning opportunities to education staff resulted in a high number of engagement points throughout the life of the project. Overall, core courses were taken 260 unique times, advanced courses were taken 76 times, distance learning courses were accessed 1,514 times, webinars were accessed 957 times, and on-demand videos were viewed at least 639 times. A more user-focused approach to monitoring would allow the project to track the number of times that unique education staff were exposed to products, as opposed to simply tracking the engagement numbers for products in silos. Though the COL approach was highly effective in reaching individual actors, survey results show that it falls short of addressing system-level change. While most participants saw changes in their personal work, only 30 percent saw changes in team/Office performance, and only 15 percent felt that it resulted in broader programmatic changes. These sentiments were echoed in qualitative interviews, where participants noted that the workstream experienced several fits, starts, and missed opportunities due to multiple staff transitions as well as lack of agreement on a core set of competencies for the sector and comprehensive PD strategy. 57 | EPIC EVALUATION REPORT USAID.GOV TABLE 7: IN-PERSON TRAINING COURSES MONTH/YEAR COURSE LOCATION NUMBER OF PARTICIPANTS September 2016 EOC DC 29 October 2016 EEPM DC 24 May 2017 EOC DC 19 June 2017 EEPM DC 10 November 2017 EEPM Bangkok 11 December 2017 ED-CVE Bangkok 33 April 2018 ED-CVE Frankfurt (MENA) 26 June 2018 EEPM DC 14 August 2018 EEPM Accra (Africa) 13 May 2019 ED-CVE DC 17 June 2019 EEPM DC 15 October 2020 EOC Virtual 96 November 2020 EEPM Virtual 29 IR 2.2: CORE COMPETENCIES OF DDI/EDU AND EDUCATION FIELD STAFF TO DESIGN, MANAGE, AND EVALUATE PROGRAMS THAT ACHIEVE THE AGENCY’S EDUCATION SECTOR GOALS STRENGTHENED. Over the lifetime of the project, the PD workstream undertook several interventions (described above) to build the core competencies required for USAID education staff to build capacity, influence policy design, and support systems more effectively in host countries to achieve their educational goals. In addition to the webinars, knowledge products, and on-demand videos described in IR 2.1, the PD workstream created two core courses and one specialized course that were designed, delivered, and updated in a manner that consistently responded to staff core competency needs. A final survey of the effectiveness of PD learning materials showed that most participants found the materials from the blended learning approach to be relevant to their work (87 percent) and were able to competently 58 | EPIC EVALUATION REPORT USAID.GOV apply the knowledge and skills they gained (83 percent). Seventy-five percent of the final learning survey respondents felt that they could expertly apply the learning in their job as well as mentor/coach others. Across trainings, participants frequently indicated utilizing both the hard and soft skills that they learned. The Education Overview Course was designed to orient Education Officers to the new USAID Education Policy and help them learn to shift their work, where necessary, around its principles, priority technical areas, and new practices. It was offered in a face-to-face format two times (in 2016 and 2017) and was redesigned and offered as an asynchronous distance learning course in concert with the GELS in 2020. The EEPM course covers technical and leadership/soft skills that equip participants to be more effective in developing buy-in, commitment, scale, and sustainability in their technical areas. This course was offered in a face-to-face format seven times from 2016 to 2019 and was redesigned as a distance learning course in 2020. The ED-CVE course was a more specialized face-to-face course designed to cover a global perspective of the Education Policy and how CVE work is framed across varying agencies. This course was offered three times between 2017 and 2019. At the end of Year 5, the EOC and EEPM courses were offered virtually for the first time. EEPM was delivered as a one-time, virtual instructor-led course that had both synchronous and asynchronous components. The EOC modules were available in an asynchronous format whereby participants could elect to take any of 20 modules over the course of three months. Staff were hopeful that the transition to online courses would increase the number of people that the course reached and would more effectively support their competencies. As expected, the virtual format resulted in a larger number of participants having access to the learning. The EEPM virtual course had 29 participants, while all in￾person deliveries of the course had an average of 15 participants per course (between 10 and 24). The asynchronous EOC course had 96 participants complete at least one module, though only 14 completed the entire course. By comparison, in-person deliveries of the EOC averaged 24 participants (between 19 and 29 participants). A breakdown of the participants that attended each course is shown in Table 7 above. Overall Descriptive Statistics In total, 266 unique participants completed at least one course. Of these participants, 32 percent (n=85) took more than one course. At the time of their training, 37 percent of participants were affiliated with USAID/Washington (n=100), and 63 percent were affiliated with a Regional Bureau or Mission (n=164). Tenure at USAID ranged from less than one year to more than 10 years, though the most frequently indicated tenure was 1-3 years (41 percent of respondents). Overall Satisfaction To assess the quality of courses and the satisfaction of those who participated, each course included a feedback survey, which captured participants’ immediate reactions, satisfaction levels, changes in knowledge, and anticipated changes in behavior. These surveys were completed by a total of 164 participants. Across all courses, survey analysis showed that participants had high levels of satisfaction with course content and structure (at least 84 percent, with an average of 94 percent), their knowledge and skills increased (average increase of 25 percent), and their intention to incorporate new learning into specific aspects of their work increased (an average of 80 percent of participants agreed). Awareness of available resources also increased in all trainings (on average, awareness increased 25 percent). 59 | EPIC EVALUATION REPORT USAID.GOV Participant satisfaction increased over time. Courses were rated lower (84 percent satisfaction) in 2016, then improved to average 94 percent in all other years. When negative comments were made, participants most frequently expressed lack of satisfaction with support materials, requested further information on technical aspects of the modules, and requested more practical examples of concepts. Virtual course participants also commented that the long audio sessions without visuals were more difficult, that they wanted more engagement, and that they felt overwhelmed by the number of additional materials that they were expected to review. Participants had varying levels of satisfaction with the length, pace, and complexity of courses. The overall average was 84 percent. ED-CVE had the highest scores: (86.5 percent), followed by EOC (84 percent), and EEPM (82 percent), however, some received scores as low as 60 percent (EEPM Asia’s complexity score). Notably, virtual courses had the lowest scores in terms of satisfaction with length: 63 percent of participants felt that it was too long. Across courses, participants provided very positive feedback on the quality of the facilitators. In general, and particularly in EEPM courses, participants enjoyed the mix of technical and soft (leadership/emotional intelligence) skills and recognized that both were important to their work. Higher satisfaction scores tended to correlate with higher increases of knowledge and skills. Participants overwhelmingly preferred engaging activities (top activity in 100 percent of trainings), group discussion (small and large) (top activity in 81 percent of trainings), and interactive lectures (top activity in 72 percent of trainings). This feedback demonstrates the importance of convening field and Washington staff and allowing them to learn from one another’s experience. This was true in both virtual and in-person courses. Findings from EOC Courses (In-Person and Virtual) As described in IR 2.1, the EOC was originally developed as a response to staff needs voiced in Year 1 focus groups. The learning objectives for the first two deliveries of the course were as follows: 1. Apply understanding of USAID Education Sector strategy and goals to the development and implementation of education programs. 2. Apply understanding of global, regional, and country contexts and USAID practices to communicating, collaborating, and building strong relationships with stakeholders. 3. Advocate for the critical importance of the education sector and be able to summon resources to address challenges across sectors and with host governments, local implementers, donors, and other partners. 4. Leverage Missions, EDU/DDI, Regional Bureau, and other Washington USAID offices for effective programming, given their respective roles and resources. The objectives were slightly redesigned in the virtual delivery to focus more on the USAID Education Policy: 1. Frame the USAID Education Policy, particularly the Journey to Self-Reliance, the principles, and priorities, and how they relate to other Agency-wide policies, key actors, and offices for the budgeting process for education programming. 2. Describe the breadth and scope of USAID education programming, from pre-primary to higher education, considering equity and inclusion, such as gender, disability, poverty, and conflict and crisis contexts. 60 | EPIC EVALUATION REPORT USAID.GOV 3. Implement the USAID Education Policy with the correct guidance, tools, and resources for different technical areas. 4. Make decisions about how to invest the Mission’s resources based on the USAID Education Policy, relevant USAID funding guidelines, and evidence from the country. 5. Explain your role in supporting children and youth to achieve learning gains in partner countries. 6. Establish your relationship with the Office of Education and the technical teams for consultation, guidance, and shared learning. For the face-to-face course deliveries, participants’ knowledge/skills increased nearly 25 percent on average across course objectives, moving from low scores to moderate high scores. Awareness of resources also increased 27 percent. Though average change in knowledge/skills were very similar, the objective with the highest change was “Advocate for the importance of the Ed sector and be able to summon resources to address challenges across sectors” (0.97 change), while the objectives with the lowest change were “Understand global, regional, and country contexts as well as USAID processes and apply that understanding to communicating, collaborating, and building strong relationships with stakeholders” and “Identify roles and responsibilities of USAID Ed actors and how to leverage them for effective programming,” each with 0.92 change. The modules that had the greatest change in knowledge across course iterations were understanding the Education Sector strategy and goals and education outcomes in conflict-affected areas. For the virtual delivery, participants’ knowledge/skills also increased an average of nearly 25 percent across objectives, from low-moderate scores to moderate-high. The area with the greatest increase in knowledge and skills was the ability to describe the breadth and scope of USAID Education programming across stages and contexts (Objective 2). The average knowledge change for this Objective was 0.81, with 10 of the 17 participants reporting moderate knowledge and seven reporting high knowledge. The lowest average change in knowledge was found in Objective 5 (explain your role), with an average change of 0.5. Nearly half (n=8) of the participants did not report a change in knowledge; this Objective had the highest pre-course knowledge scores (average 1.87), which indicates that participants already had sufficient knowledge, and as such did not have as much to learn. All modules received comments on the high quality of the module content and teaching methods. Nearly every module received open-ended feedback that participants learned everything that they expected to, and that the topic was well covered. Across modules, participants frequently mentioned that key concepts and definitions were clear, appropriate resources were helpful, and practical examples were highly appreciated. Participants greatly appreciated when modules provided a good mix of audio, visual, and reading learning. However, participants expressed some dissatisfaction with some virtual engagement components of the course, namely an online forum that was not relevant or helpful. This indicates that the EPIC team still needs to develop methods to generate meaningful interaction in distance learning settings. “This was the highest quality training I have taken thus far at AID. It made such a difference (to individual & group learning comprehension & retention) to have skilled educators & facilitators delivering the course material. I was so impressed by the diversity of content, teaching styles, activities, materials (table toys!), and structured learning modules. Truly an exceptional course!” — Participant, Pilot EOC (2016) 61 | EPIC EVALUATION REPORT USAID.GOV “The course was very interesting and interactive. The learning experiences were very practical. I think this was a great approach.” — Participant, Virtual EOC (2020) Findings from EEPM Courses (In-Person and Virtual) As described in IR 2.1, the PD workstream partnered with the PDWG to design and deliver the EEPM course, which had last been offered a decade previously. The course objectives for the face-to-face courses were as follows: 1. Understand and use the Collaboration Learning and Adapting (CLA) Paradigm in education programming. 2. Design evidence-based, flexible projects and supporting implementing instruments that support USAIDs education strategy. 3. Develop skills to more easily complete required project design and implementation documents (PADs, SOWs, PDs). 4. Create flexibility for adaptive management through use of flexible implementing instruments. 5. Address challenges that arise during design/implementation with host government, donors, implementers, and other partners. 6. Understand changes to ADS 200 and 201 and ensure compliance through product design and implementation. The learning objectives were expanded in the Year 5 virtual course, and are as follows: 1. Identify how CLA can be integrated in education programming. 2. Design evidence-based, flexible projects and supporting implementing instruments that support USAID’s Education Policy. 3. Practice analysis, communication, and advocacy skills to design country strategies, projects, and activities more effectively. 4. Assess your emotional intelligence skills and how they can make you more effective in your work. 5. Create flexibility for adaptive management through flexible implementing instruments. 6. Discuss solutions and approaches to addressing challenges that arise during design and/or implementation with host government, implementers, donors, and other partners. 7. Explain the changes to ADS 201 to ensure compliance throughout strategy, project, and activity design and implementation. 8. Create and access learning resources to support your work in the education sector. 62 | EPIC EVALUATION REPORT USAID.GOV Overall, participants were highly satisfied with the EEPM trainings, and the satisfaction increased over time. 84 percent of participants from the first training were satisfied, which rose to 90 percent for all following deliveries, and 100 percent for the final face-to-face delivery. However, some participants were less satisfied with the course pace and complexity. The EEPM Asia course received only 60 percent satisfaction with complexity, the Year 2 Washington EEPM received only 75 percent, and the EEPM Accra received only 78 percent. Comments from the feedback after these sessions show that participants wanted more time to focus on key materials and opportunities to apply their learning and network with other staff. Participants’ changes in knowledge/skills in the face-to-face EEPM deliveries were also a bit more varied than the EOC course. Scores increased between 20 percent and 32 percent, moving from low/moderate to moderate/high. The second DC course iteration in Year 2 had the lowest change in knowledge/skills across objectives, with an average change of 14 percent. However, scores increased in later iterations of the course, with the highest average changes shown in the final Washington EEPM and an average of 30 percent increase in knowledge/skills. The objective with the highest change was “Develop skills to more easily complete required project design and implementation with purpose and efficiency,” and the lowest was “Address challenges that arise during design/implementation with host governments, implementers, and other stakeholders.” The modules that had the greatest change in knowledge across course iterations were understanding and using CLA, awareness of resources, designing projects and instruments that support the Education Strategy, and leadership skills: emotional intelligence and leadership styles. In the Year 5 virtual EEPM, participant knowledge/skills also increased across learning objectives. The two areas with the greatest increase in knowledge and skills were the ability to explain changes to the ADS 201 to ensure compliance (Objective 7) and the ability to create and access learning resources (Objective 8) (average change of 28 percent and 27 percent, respectively). In contrast, Objectives 3 (analysis, communication, and advocacy skills for design), 4 (emotional intelligence), and 6 (solutions and approaches to addressing design and implementation challenges) had relatively lower average change in knowledge and skills (about 10 percent increase) but among the highest pre-course knowledge scores. All but one participant either Agreed or Strongly Agreed that the course increased their understanding of how the development hypothesis and theory of change underlie the USAID country strategy, that they have the tools and skills needed to apply the CLA paradigm in their work, and that they have identified areas in their work to apply the CLA paradigm. Throughout the course, participants greatly enjoyed the combination of videos, group exercises, and breakout discussions. The facilitation received several positive comments as well. Popular training methods included Q&A sessions, real-life examples and case studies, and role play. Learning materials such as slides and graphs also were highly rated. That said, some participants requested additional learning resources to better clarify and use these materials and noted that some sessions were too fast (especially PDD and Evidence for Decision-Making). Less popular training methods included share-backs, which some participants found repetitive. “The thoughtfulness and flow of the training design was outstanding. Daryn has an exceptionally effective human touch that made the process engaging and effective. A+!” — Participant, EEPM DC (2019) “Anytime dialogue is encouraged in small groups with our colleagues from around the world is time well spent!” — Participant, EEPM Virtual (2020) 63 | EPIC EVALUATION REPORT USAID.GOV Findings from ED-CVE Course As ED-CVE was a more technically specialized course, participant knowledge scores were higher than EOC or EEPM scores, both before and after the training. In the first two deliveries of the course, participant knowledge scores increased 25 percent, moving from moderate to moderate/high. The course was then redesigned, which led to scores increasing significantly (41 percent average increase in knowledge). Scores were moderate/high before the training and high after the training. In particular, participants’ ability to “Design, implement, monitor, and evaluate education interventions within a CVE program” increased by 50 percent in the third training. One of the objectives that saw the greatest improvement between iterations of the course was “Apply CVE policies to design CVE education programs for children and youth.” Participants scored over 25 percent higher on this objective in the second iteration of the course. The modules that had the greatest change in knowledge across course iterations were CVE analysis data and tools and MEL Principles, and the modules on PYD were incredibly popular across course deliveries. Across course deliveries, respondents provided more concrete, detailed explanations of the role of education in CVE after the training. Participants also identified many more monitoring, evaluating, and learning (MEL) principles, approaches, and methods that could be applied to education and CVE programs in the post-test. Specifically, participants listed more systems-based and adaptive methodologies, such as complexity-aware approaches and developmental evaluation in the post-test. One notable best practice was the inclusion of non-USAID class participants. “This is the first education training that I've taken that included other technical Office representation and other Agency/dept representation. Those colleagues’ presence immensely contributed to my learning and to the course.” — Participant, ED-CVE Bangkok (2017) Outcomes from All Courses Responses from follow-up surveys sent to all education staff who had participated in any of the EEPM, EOC, or ED-CVE trainings provided evidence that nearly all participants found the course learnings to be relevant to their work (87 percent) and were able to competently apply the knowledge and skills they gained (83 percent). Three-quarters of the final learning survey respondents felt that they could expertly apply the learning in their job as well as mentor/coach others. Across trainings, participants frequently indicated utilizing both the hard and soft skills that they learned. Over half of EEPM participants described how the trainings were beneficial to them in applying CLA, at different stages of the program cycles, in designing new programs and activities, and better understanding the education strategy/policy and funding parameters. Likewise, participants indicated that the training sharpened their program management skills at similar rates; primarily in leadership, communication, advocacy, and emotional intelligence. Of the six final learning survey participants that took the ED-CVE training, most applied the sessions on PYD (n=6), ED￾CVE and Guidance (n=4), the theory of change (n=4), ED-CVE Policy (n=3), and data and tools (n=3). Participants who took ED-CVE trainings most frequently reported applying their deeper understanding of CVE drivers and approaches into their design and implementation of activities as well as sharing resources and improving coordination within Missions, among their teams and with implementing partners. Notably, while most participants (over 75 percent) saw significant applications of the trainings in their personal work (as described above), only 30 percent saw changes in team/Office performance, and 15 percent felt that it resulted in broader programmatic changes. Team/Office changes included technical 64 | EPIC EVALUATION REPORT USAID.GOV improvements such as sharing resources to incorporate into activities as well as organizational improvements such as better teamwork for improved results. IR 2.3: CLA APPROACH STRENGTHENED, DOCUMENTED, AND SHARED. The PD workstream had mixed levels of success in this IR. Interview respondents noted that one key strength of the contract by design and in its implementation, specifically in relation to PD, has been the ability to flexibly adapt in response to urgent, important, and ad hoc requests by DDI/EDU as they emerge. “For the most part, the CLA was very much on point. The continual learning was a trial and error. Part of it was to be more collaborative, you have to work with people’s schedules. We were working with people who were stretched thin and PD wasn’t their top priority.” — DDI/EDU Technical Specialist “Able to increase learning opportunities in different ways beyond face-to-face, which was the priority. Collaborated with DOD, BG Office, education across the entire spectrum not just E3ED. In terms of challenges, I think that everyone thought… that they would do the training. It became at times an excuse for people to go to the field. Trying to come up with a cohesive plan was not always respected by others.” — DDI/EDU Technical Specialist CLA in Creating PD Products and Activities Much of the success of IR 2.1 and 2.2 was made possible through applying CLA practices to the creation of new products. Interview respondents noted that this workstream modeled a lot of great CLA practices. Consistent and intentional collaboration with DDI/EDU, Regional Bureaus, Missions, and SMEs included gathering feedback from staff in Washington and oversees, conducting periodic sector-wide assessments of needs and priorities (including field staff in design and subject matter choices), and selecting core course topics based on results gathered from comprehensive survey activities as well as feedback specific to each training. Since the workstream offered repeated versions of the same courses, the participant evaluations provided a built-in opportunity to immediately assess and incorporate feedback into future refined and updated iterations, ensuring that the content and presentation was as relevant and useful as possible. Surveys, such as the learning priorities survey in Year 3, were used to inform strategic planning and learning priorities for the GELS. In the retrospective review completed in Year 3 of the project, a key recommendation for the PD workstream was to measure the application of knowledge gained during training. This suggestion was applied three times, in the form of follow-up surveys four to nine months after trainings. While this provided helpful information in adapting the trainings, it should have been more widespread. The PD workstream also demonstrated successful collaboration across EPIC workstreams, particularly KM and ECP. In particular, the PD workstream collaborated with the KM workstream to repurpose content from training courses into content for EducationLinks, drawing from the expertise of Education Officers in the field. The workstreams also collaborated in the planning of the GELS, described below. In Year 5, the PD workstream worked closely with the KM and OE Teams to support the organization, design, and development of the 2020 Global Education Conference. This involved everything from helping the Office establish goals and objectives for the conference to supporting individual sessions that were accepted for the conference. Due to the COVID-19 pandemic, the PD Team made a sharp pivot to adapt and transform what was supposed to be an in-person conference into an online Global Education Learning Series. EPIC staff was very encouraged by the success of this event. It established the continued need for knowledge and professional development products with which the sector could 65 | EPIC EVALUATION REPORT USAID.GOV engage. It also demonstrated the success of using virtual modalities as a platform for learning and collaboration within the education ecosystem. The PDWG was also pleased with the support provided by the PD workstream; weekly SDD committee meetings enabled EPIC and the committee to pivot and adjust processes as needed. An after-action review session and survey was conducted to collect lessons and best practices, as well as data on the utility and quality of EPIC’s role in the event planning. Overall, the SDD team found the resources very sufficient (96 percent), and agreed that the workload was reasonable, team members were highly committed, and communication amongst members was open (80 percent). They were more neutral in response to whether roles and responsibilities were well-defined (64 percent) and the clarity of the decision-making process (60 percent). “The EPIC PD team is phenomenal. Great to work with. Really responsive. I appreciate that they can be thought partners. They’ve contributed a ton to helping design and develop different ways of engaging learners throughout the education ecosystem. They’ve been really agile in adapting to this changing environment due to COVID-19. Example is shifting from the conference to the online learning series. Thinking with me on what is achievable. Objectives for this series given fewer hours. They’ve been instrumental in the redesign of the Education Overview Course. We now have a design that is dramatically different from what we had in the past. It will be a huge contribution toward how the Office develops new Education Officers. Very strategic, innovative, creative, and accommodating with all these changes like the EEPM course adaptation to a live online instructor-led format will make it much more accessible to Education Officers. They are able to be agile and flexible and creative to help us adapt to this new environment.” — DDI/EDU Technical Specialist “I think we’re making progress on the continuum of learning. More collaboration between KM and PD teams, that’s been expanded after I came on and particularly when Daryn came on. Forced moments to get together to see what everyone was doing. We continue to improve that. The GEC was a time when we had to work together… we have been able to share, and that’s been fun and challenging and a little bit stressful to figure out what we want to do and share with each other. Over the last year, there has been a lot more collaboration among PD, KM, and Comms. PD handled what happened in the sessions, substance of the events. KM steps in and posts and promotes and captures and shares knowledge. It’s been fun. Something different that EPIC hadn’t had the opportunity to do before those major events.” — EPIC Team Lead “From the PD side, in the development of training materials, it was bringing in people from different parts of the Office. KM monthly content meetings that they had was a great way to draw people in and work together. Have multiple touchpoints, which help to inculcate new behavior. It’s much less stove-piped, much more collaborative now. The ‘how’ was some of the processes and systems that forced them to collaborate.” — EPIC Sr. Management CLA as a PD Topic This IR was also addressed through the redesign of courses to align with the 2018 USAID Education Policy and further integrate the CLA approach. The USAID Education Policy has shifted from inclusion of goals and targets to principles and priorities that guide education programming. This shift increases flexibility in how education programs are integrated into country development and cooperation strategies as well as increases emphasis on collaboration and partnership. CLA principles are reflected in the overall objectives of the EOC, including increasing knowledge and skills to implement the Policy in various technical areas and establish relationships in the Office of Education for shared learning. The 20 EOC modules integrate CLA principles as well. For example, Education in Crisis and Conflict addresses programmatic adaptive management and learning in crisis and conflict environments. The soft skills modules include the importance of conversational intelligence for developing transformative partnerships. The overall objectives of the EEPM course include understanding and using the CLA 66 | EPIC EVALUATION REPORT USAID.GOV paradigm in education programming and implementing adaptive management tools and strategies. EEPM units were designed to increase incorporation of CLA into daily work, including project design and activity implementation. In all face-to-face iterations of the EEPM, participants showed success in strengthening skills/knowledge of CLA. All courses resulted in average knowledge/skill increases in CLA objectives, with an average increase of 1.12 points on a five-point scale (22 percent increase). Confidence in applying CLA also improved during the course (range of 0.58 to 1.63 point change on a 5- point scale). After the Year 5 EEPM course, all but one participant indicated that the course increased their understanding of how the development hypothesis and theory of change underlie the USAID country strategy, that they have the tools and skills needed to apply the CLA paradigm in their work, and that they have identified areas in their work in which to apply the CLA paradigm. CLA Difficulties This workstream experienced some confusion about roles and responsibilities as they relate to collaborating on PD products. Most DDI/EDU staff interviewed understood EPIC’s role as complementary to USAID, where USAID staff provide the technical expertise, technical content, and deliver technical training. They felt EPIC’s role was to provide expertise in adult learning and instructional design, help trainers develop their training and presentation skills, create better conditions for learning, and provide support for learning technologies and platforms and the development of learning materials. On the other hand, two DDI/EDU respondents had the expectation that EPIC staff should have education sector technical expertise and should be able to produce content, develop training curricula, and deliver the training—in essence, supplementing their technical team. While some EPIC core team members do have many years of education sector experience and EPIC has brought in technical experts when requested, many EPIC staff and consultants are not education technical experts. Rather, they bring more specialized expertise in designing and facilitating learning and organizational development from a systems perspective to create more optimal conditions for learning. “It seemed like EPIC did a lot of work to help us set things up and organize PD, and in some cases, they would put together the training, but a lot of the EPIC team members were not technical specialists, so it seemed like our team needed to go back and rework things. I always found them very open to feedback. They knew that we were so busy that it would be hard for us to put together things ourselves. The online series that is rolling out now has staff members working in consultations and would then work together to put together modules. The LOE was great. Because they weren’t technical specialists, our teams would need to do a lot of editing. It was great to have an organization that was helping to organize PD, but we need to figure out a better way to do this. It is true, we don’t have a lot of time to put together these trainings. Definitely making sure that the technical teams are looped in from the beginning and throughout. TRG has worked really hard on these things. The online module series, we wouldn’t have had the time to put these together.” — DDI/EDU Technical Specialist RECOMMENDATIONS AND LESSONS LEARNED DEVELOP A COMPREHENSIVE PD STRATEGY. Lack of a core professional development strategy meant that there were varying perceptions on the goals of the PD workstream in terms of who it aims to reach and for what purpose (e.g., professional development for Washington-based staff to keep them up to date on technical topics; bringing together staff from the field and Washington to learn from one another; or attending leadership and management training workshops and individual coaching under the OE workstream). Interview respondents described 67 | EPIC EVALUATION REPORT USAID.GOV the importance of better connection to the education ecosystem, in terms of reaching more Education Officers worldwide, connecting with the education ecosystem writ large, and in ensuring that the PD needs of the Center for Education are met. The PD strategy should be built on to include: • A five-year results-based vision of success. • How to set up a global system of organizational/sector learning. • The universe of people to engage in PD activities. • The core competencies staff need to be effective in their work. • The range of modalities that will be offered. • The specific steps to reach individuals, teams, Missions, and working groups. • How regional workshops and conferences can be used to develop capacity to support Agency transformation and education sector goals. • How all workstreams will collaborate to make this vision a reality. “This was the workstream that we hoped we would reach more Mission staff other than just participants. Get Mission staff involved in design. We started to make some inroads but didn’t get as far as we would have liked. EPIC has been more DC focused but for various reasons we weren’t able to get there... Bureaucratic levels were also an obstacle, as everything was done indirectly through multiple levels.” — EPIC Sr. Management “Core competencies involved HR – long process to get changes approved. It took them years to develop them. They needed learning opportunities to get stuff done now. We didn’t have a cohesive plan, place where people could get what they need, etc. It was just orienting the Office because everyone in the Office had their own ideas.” — DDI/EDU Technical Specialist “We’re still not at a coherent well thought-out curriculum moving away from individual competencies to team and Office competencies. There are knowledge and skills that we need to get better at, like systems thinking and tools.” — EPIC Sr. Management “Core competencies: don’t know that I could list them. Having consistent communication about these things would help. I remember there was discussion about the list of core competencies. Don’t know if that was resolved. Had a PD working group, but folks didn’t think it was that effective. Is it check the box? Make decisions? Change things. We made operational progress but can’t say if the quality improved.” — DDI/EDU Team Lead STRONG LEADERSHIP AND A CHAMPION FOR PD ARE NECESSARY FOR WORKSTREAM SUCCESS. Many interview respondents noted that while the PD workstream was an essential piece of enacting the Education Strategy worldwide, gaps in leadership and personnel turnover meant that the workstream experienced periods of being undervalued and non-prioritized. They felt that transitions in activity leads meant that there was no consistent champion to shepherd important PD processes. They noted that when these human resource needs were fulfilled, the workstream experienced more buy-in and was more effective. 68 | EPIC EVALUATION REPORT USAID.GOV “There is a pivotal moment, which is the changes in leadership of the Office. I don’t think that as an Office, we anticipated the changes in leadership to impact the outputs of the project as much as I think they did. A shock or stressor is definitely changing leadership. If the structure and governance are not well followed, then new leadership can cause some level of stress and change. I would explore that. EPIC, because it touched so many pieces of the Office, it was more vulnerable to it… I don’t know if this is pivotal, but there was a lot of turnover in the PD work. That was a constant challenge.” — DDI/EDU Team Lead “I think that when Courtney came on, they hadn’t had anyone with a PD background to manage this WS. She was able to professionalize the subsector. She’s a strong personality. Knows what she’s doing and moved things forward. It’s difficult to do because everyone is involved in this. It had stalled a bit, but with her leadership she was able to move it forward. The issues were more on our end.” — DDI/EDU Technical Specialist “We started by trying to define the competencies and focus but we never got to an agreement...I think there wasn’t leadership that said they needed to come up with their plan for PD. No one was calling for it. It was left for the worker bees to continue, and nobody was paying attention.” — EPIC Team Lead BE FLEXIBLE AND ADAPTABLE TO RESPOND QUICKLY TO ARISING NEEDS. As was the case in other workstreams, one of the key strengths of the EPIC contract was the flexibility in responding to needs and pivoting in reaction to changing circumstances. For example, the PD workstream attempted to host the Global Education Conference twice during the lifetime of the contract and was met with significant extenuating circumstances on both occasions. Respondents noted that the EPIC team responded with flexibility and provided valuable solutions, which greatly advanced the objectives of the contract. In particular, respondents celebrated the pivot from the in-person GEC to the virtual GELS, and were hopeful that the shift to virtual modalities would continue to increase the reach and impact of their work. “With COVID-19, our Office created a task force to look at education in the pandemic. Shoutout to Daryn and Shira, who has been so responsive and organized to support the COVID 19 webinar series for education staff in the Missions. That’s not something we planned for. EPIC has been flexible and open to feedback. AdobeConnect didn’t work well and EPIC proposed other platforms and ways that have been really successful. Their customer service and innovation and creativity has been top notch. I think they’re all great.” — DDI/EDU Technical Specialist “They’ve come a long way. There were starts and stops with changes in the lead on the USAID side. That has impacted EPIC in terms of how they are valued and what they brought in and what kind of progress is made. EPIC has tried to be flexible and stepped in when there has been an opportunity to be a value add in this workstream. The overall vision now is clearer.” — EPIC Technical Specialist THE CONTRACTOR AND DDI/EDU SHOULD WORK AS STRATEGIC PARTNERS, WITH CLEAR ROLES AND RESPONSIBILITIES, TO DEVELOP PD MATERIALS. This workstream was most effective when roles and responsibilities were clarified, allowing for strategic input from EPIC and DDI/EDU to be engaged when it was most useful. Specifically, use other mechanisms for surge technical support, and continue to rely on EPIC staff for adult learning and communications, with content input generated from consultations with DDI/EDU staff. Additionally, many interview respondents noted that the workstream was much more successful when EPIC was included in the PD working group. This engagement should continue. “At one point, we were really involved in a PD working group when I started with EPIC. It was a rocky relationship... The new activity manager on the USAID side joined. There was a point when we were 69 | EPIC EVALUATION REPORT USAID.GOV excluded, which was a big hit for us. It comprised [of] Office of Ed folks and bureaus. Would have informed us of the direction they intended to head in. Now that there is a new Activity Manager from USAID who is very collaborative and seeks out our knowledge, she has restarted the Working Group. We work really well with her. We use that space to really discuss things like the GEC.” — EPIC Team Member “The PD point is crucial in having experts trained in adult learning and course instruction. We underutilized them. It was an opportunity to develop really strong courses. It’s a skill that a lot of the people in the Office don’t have. Think about the skills that we don’t have that we want to learn from and harness. We are technical experts, but we really needed EPIC for the training side. Rather than delegating USAID work to EPIC, use EPIC to help with the branding, presentation, higher-level comms work that we can take and implement day-to-day.” — DDI/EDU Technical Specialist ENSURE THAT ALL BUY-IN AND PERMISSIONS REQUIRED FROM ALL LEVELS OF USAID’S BUREAUCRACY ARE CORRECTLY ACCOUNTED FOR AT THE KEY POINTS WHEN DESIGNING PD ACTIVITIES. To ensure the greatest possible success for PD activities, it is crucial to engage key stakeholders at the correct time in the design and development process. This practice also involves allotting the required amount of time for deliverables, recognizing that the bureaucratic structure of USAID requires additional time. This practice will reduce the chances that a deliverable may run into bottlenecks or barriers later in the process, potentially leading to it being thrown out. “Knowing how and when to engage key people at certain points in the design of something. Engage could mean bring them in in the initial ideation part, be involved in the decision-making on what it is going to look like…. A couple of times it has happened that we thought we had the blessing of the decision maker and found out later we didn’t. I hadn’t worked with USAID before. Stuff takes time in a bureaucracy. It can take many miles for it to make a 90-degree turn. In other places I’ve worked, it is faster. In the Office of Education, there are so many levels, contractual regulations. Requires patience. For those who have worked in start-up places, that is not how it works here. Clearance: build in time to clear content. For KM or PD stuff. Not down to the minutia of how we are going to facilitate an activity, but if there is something that is supposed to represent USAID, be sure that whoever you are working with is putting it through the right levels of clearance...Worked really well with the SMEs to develop it, but at the end as it was ready to go up onto USAID University, the plug was pulled. At the time that the course was originally developed, it was under a different administration and then there was a shift in the policy and the higher ups didn’t sign off on it. Now no one can access it.” — EPIC Team Lead CONTINUE TO USE VIRTUAL MODALITIES AND ASYNCHRONOUS LEARNING OPPORTUNITIES AS EXTREMELY EFFECTIVE COMPONENTS OF THE COL. With the decision to cancel the GEC and instead create a series of modules for the GELS, the PD workstream demonstrated the need to connect to the education ecosystem with more frequency than face-to-face trainings. Though participants espoused the value of diverse modalities, often the face-to￾face trainings received the most attention. COVID-19 catalyzed the production of all virtual learning events and allowed education staff the opportunity to immediately benefit from specific resources, instead of waiting to take a multiple-day training, which is both resource and time intensive. In four of the course feedback surveys, a top suggestion from participants was to adjust the course format to ensure that topics are covered but the amount of information is manageable, and include virtual opportunities (e.g., webinars, coaching, mentoring, networking). “So much of this knowledge and skills that were seen as needing an in-person course are now available to everyone whenever they want it. You can take an hour of it. If you have no experience with social and emotional learning, you now have accessible all this information presented in a guided and sequenced way. They don’t have to figure out where to find and what to do with the information. Two things are 70 | EPIC EVALUATION REPORT USAID.GOV accomplished with that. Will get more FSOs and FSNs and partners more on the same page in language, policy, all those things. If you work in E3ED, you can tell someone to go and take this course rather than having to wait for months to take a course. Technical staff had battles about who would be teaching courses to get in front of USAID staff to do their PPT presentation. When they come together in person, they can look at how to apply things, build skills, do things that are harder to do online. It has freed up time to do the other things that warrant being in person.” — EPIC Team Lead MAKE TRAINING INTERACTIVE AND PRACTICAL AND PROVIDE OPPORTUNITIES TO LEARN FROM PEERS. According to course evaluations, high satisfaction rates and knowledge and skills improvement in courses were related to three factors: 1) incorporating an interactive and engaging facilitation approach; 2) using practical and real-life examples; and 3) providing the opportunity to learn from peers. INCORPORATE ONGOING EVALUATIONS AS PART OF THE COL. Delivering multiple iterations of the same courses provided an excellent opportunity for CLA practices. Administration of post-course surveys allowed for feedback to be immediately incorporated into future iterations of the course. Though surveying course participants immediately after course completion is useful for assessing satisfaction, knowledge gained, and intent to change behavior, it is limited in its ability to measure changes in participants’ behavior as well as impact on their individual work, technical teams, Offices, Missions, and the sector writ large. Additionally, post-course survey participants frequently showed signs of evaluation fatigue, leaving more questions blank or providing short, non-descriptive responses. Many course evaluations suggested adjusting data collection to avoid such fatigue and improve accuracy and richness of the data. EPIC addressed this through sending six- to nine-month follow-up surveys to participants in three courses, as well as sending a final learning application survey to all USAID staff who took at least one training. These surveys were useful in capturing behavior change and impact, but the engagement rate was rather low (36 participants). This challenge should be addressed by incorporating ongoing brief and in-depth evaluations as part of the continuum of learning. “EPIC having done multiple trainings of the same type over the last four to five years has allowed us to feed into improvements, and serve as institutional knowledge to be able to do CLA with PD. It looks like us being able to draw on documentation of past trainings to post-evaluation phase. It’s there for DDI/EDU to look at as well. We can look at evaluations from past trainings to see what participants liked and didn’t like and to strengthen our offerings and the Office’s own trainings separate from EPIC support.” — EPIC Team Member CREATE AN INTEGRATED PARTICIPANT TRACKING SYSTEM TO BETTER UNDERSTAND HOW USAID STAFF ARE EXPOSED TO MULTIPLE PD LEARNING OPPORTUNITIES. A more user-focused approach to monitoring would allow the project to track the number of times that unique education staff were exposed to products, as opposed to simply tracking the engagement numbers for products in silos. This would generate more data on how the COL is experienced. PROVIDE FOLLOW-UP LEARNING OPPORTUNITIES FOR TRAINING PARTICIPANTS BASED ON THEIR INPUT ON TOPICS AND LEARNING MODALITIES, PAYING PARTICULAR ATTENTION TO ASYNCHRONOUS METHODS. Most final learning application survey respondents (83 percent) were interested in participating in follow-up learning activities such as blogs, podcasts, or webinars. The most frequently requested additional learning topics were skills for project design and implementation documents; designing evidence-based projects; sharing of case studies, lessons learned, and successes from countries with similar contexts; and specific topics such as education and gender empowerment, focus on 71 | EPIC EVALUATION REPORT USAID.GOV migrant/refugee children, and government priorities. Similarly, specific topics that were mentioned by participants in post-course evaluations were: program design, management, and implementation; workforce development; early grade reading; M&E; USAID’s budget and education-related budgets; scale and sustainability; understanding Automated Directives System (ADS) 200 and 201; and Project Appraisal Document (PAD) application. Emphasizing and creating more asynchronous modalities (such as pre-recorded videos) would allow for staff to immediately access the training that they need without a delay in time or a large expenditure of resources on the part of the training delivery team. “I was observing and managing, and it occurred to me that this course was supposed to reach all Ed Officers, and it was immediately clear to me that one delivery was not going to do that. It was outdated, a way to provide PD that had not caught up with the options to train large numbers of people in more accessible ways. It was an expensive and inefficient way. Understandable that people like in person training, but it wasn’t the most efficient way to skill people up, train them, and inform them to be able to do their job in Missions…More input from the field would be better. That’s the link that I wish we could have had. It’s a lesson learned.” — EPIC Team Lead KNOWLEDGE MANAGEMENT SO 3: CREATE A SUSTAINED, GLOBAL SYSTEM OF SECTOR-WIDE LEARNING AND KNOWLEDGE EXCHANGE The strategic objective of the knowledge management workstream was to create a sustained, global system of sector-wide learning and knowledge exchange drawing from and connecting relevant information across all stakeholders. Learning and knowledge exchange occurred through education platforms containing relevant research, evidence, and CLA approaches to provide content that enabled individuals, teams, and Missions to continuously improve education program design, delivery, and outcomes. Workstream activities included development of a shared content strategy, launch, and management of the EducationLinks website, dissemination of products for the Education in Crisis and Conflict Network (ECCN) and the Global Reading Network (GRN), an internal newsletter, the Mobiles for Education Alliance Symposium and Sustainability Report, and gathering of education practitioner learning stories. Key events over the life of the project are shown in Annex 4, Table 3. The KM workstream continually refined a strategy to develop systems, processes, and content to create a robust learning environment and approach for the education sector. Through an experienced team and collaboration with other workstreams such as PD and ECP, EPIC clarified and organized information relevant for the field, redesigned resources to maximize engagement and navigation, and increasingly disseminated products over time through multiple media. Cross-workstream collaboration helped expand the reach and utility of relevant education content and resources to USAID staff and partners. Continued input from stakeholders in the education sector; collection of metrics on usage, application, and results of learning resources; and practice of CLA are important to increase the reach and impact of knowledge management tools to advance global education. A summary of key KM events is presented in Annex 4, Table 3, and a summary of the factors that contributed to and hindered the success of EPIC’s KM efforts are presented in Annex 6. “This is the most successful area that we’ve had as an Office. Our KM has improved exponentially. We went from zero to 75. Literally there was no structure, expectation, priorities. We’ve really made great strides here. Is it the latest and greatest information? Hard to say. This is where we see CLA working more, 72 | EPIC EVALUATION REPORT USAID.GOV too. The success of this can be linked to Heather, who has taken this on to improve these things. Her team with EPIC has methodically and in a sequenced way made improvements in key areas. There was a vision, the vision was communicated, periodically recommunicated, steps described, and led by somebody to let us know what was coming next. This is an area that we should be celebrating. Maybe the conditions for the success for KM could be reflected in some of the other areas.” — DDI/EDU Team Lead The following sections describe the interventions that EPIC co-designed and implemented with DDI/EDU, the outcomes they produced that contributed to each KM IR, and the factors that facilitated and hindered progress. Key events over the life of the project in this workstream are presented in Annex 4, Table 3. A summary of the factors that facilitated and hindered the success of EPIC’s KM efforts is presented in Annex 5. IR 3.1: DDI/EDU STAFF AND MISSIONS HAVE REGULAR ACCESS TO THE LATEST AND MOST RELEVANT EDUCATION INFORMATION, EVIDENCE, RESEARCH, COMMUNICATION, AND UPDATES. DDI/EDU and Missions have improved access to the latest and most relevant education resources because of the joint efforts of the EPIC team and DDI/EDU staff. Critical to access was the creation, launch, and maintenance of the EducationLinks external platform under the leadership of the project COR and KM Portfolio Manager. Since EducationLinks was migrated from the Education Toolkit in Year 3 (September 2018), users have praised the user-friendly platform that enables them to easily find information of interest via blogs, resources, announcements, events, and toolkits. One half of follow-up application survey respondents who reported accessing EducationLinks used the platform at least monthly (i.e., monthly to weekly). Several interview respondents expressed their appreciation for the monthly cross-team content review meetings that foster a more comprehensive and coordinated approach to content creation and sharing. Interviewed users commented that EducationLinks was a significant step up from the previous disparate websites, shared drives, and materials that were accessible only within work teams. Starting in Year 4, the ECCN and GRN spoke sites were launched, increasing access to information in specific areas in education. ProgramNet housed sensitive internal content that was not appropriate to post on EducationLinks. More than 3,100 products have been posted on EducationLinks, spoke sites, and ProgramNet since their launches. The content development strategy and events such as the government shutdown and pandemic affected the volume of products developed. Many interviewees expected EducationLinks to continue to offer new ways to engage USAID staff and partners as a main hub for the education sector. “There is a huge difference between the website when I first joined when I struggled to find things to now, which is very professional. Heather was involved a lot. Worked with EPIC to make a stimulating website with easy to find tools and resources for the field. Outside of this activity, there has been a development of a huge number of resources that are housed on this site that are extremely valuable... I know this is true for me – the availability of resources for us to be able to do our work has increased dramatically. They are easy FIGURE 5: SAMPLING OF DDI/EDU KNOWLEDGE PRODUCTS • COVID-19 Curated Resource Collection • Disability Inclusive Education and Education Finance How￾to-Notes • Higher Education Landscape Product Package • Rapid Education and Risk Analysis video • Return to Learning Toolkit • Social and Emotional Learning video and policy brief • USAID Education Policy video • Young Women Transform podcast and videos 73 | EPIC EVALUATION REPORT USAID.GOV to locate through a user-friendly website. None of these resources were available to me when I first joined. Helpful when I go to the field.” — DDI/EDU Senior Management “I think for Edulinks, it has a lot of functionality and complementary services that some in the Office don’t know about. We aren’t disciplined about learning about where Missions go to get information at USAID and understanding that. The things that I have done with EPIC have been super easy and I’ve thought ‘oh my gosh, I didn’t realize that is something you could do.’... Our Office doesn’t seem to have a coordination function for the sector... EPIC has been super helpful at less glamorous things like developing a style guide, polishing things. We had a workshop for lessons learned and couldn’t hire someone to write something. It was nice to work with EPIC to create a process to collect wisdom from the sector to contribute to this piece. That’s a nice way that EPIC can be used. Don’t know how common that is, but it’s a fundamentally different way that helps us capture learning and wisdom.” — DDI/EDU Team Lead USAID staff and partners also noted improvements in access to information through newsletters, communities of practice, and published knowledge products such as blogs, toolkits, and case studies. In Year 2, the KM workstream launched a monthly internal newsletter to keep staff updated on new education resources and information. Interviewed users described the newsletter as an incredibly important step in filling an information need and maintaining DDI/EDU’s consistent presence in KM. More than 650 education staff received the newsletter each quarter, with a peak of 756 per quarter in Year 5. The open rate of the newsletters was 50 percent for Years 4 and 5, indicating access to updated, relevant education information. Twenty-two percent of recipients clicked on a link in the newsletter. Additionally, PD courses such as EOC and EEPM raised participants’ awareness of learning resources such as USAID Education Sector Communities of Practice. Comments from PD course participants indicated an interest in further support to access information. For example, training suggestions from 2016 and 2018 EEPM courses included how to better use web resources and guidance in education. A summary of information access in terms of product dissemination and usage based on available data is presented in Figure 6. (EducationLinks usage is further described under IR3.3.) FIGURE 6: PRODUCT USAGE, YEARS 3-5 Note: Newsletter users = clicks on a link in the newsletter. Education Toolkit data are for Year 3 only as information was migrated to EducationLinks in Year 4. “This has been remarkable. Started with a toolkit that no one knew how to navigate to EducationLinks that our COR said was pivotal for people to access knowledge in the sector. This is one of the biggest changes. I 74 | EPIC EVALUATION REPORT USAID.GOV would attribute this to Heather’s vision and having Lisa share that vision and work with it at the helm. Also, Cheryl’s assistance was valuable... I would feel comfortable saying we hit this one out of the park. Not just the products and how they are developed, but there is also a process to think through the full KM cycle of how these products will be used. It’s been great.” — EPIC Sr. Management “Really appreciative of the monthly newsletters to the internal staff. Before there wasn’t anything like that. EPIC tends to make adjustments to make everything look perfect.” — DDI/EDU Team Lead “…design and launch of Education Links was a game changer. That was primarily managed out of this workstream. The platform enabled the access piece under IR1. In addition to regularized communications through the internal newsletter. And then engaging practitioners, which is more recent.” — DDI/EDU Technical Specialist “We’ve done a really good job working with the Office to create the structures, processes, and tools to continue their KM work. Governance that oversees how Edulinks and microsites are run. Created guidance around how knowledge products are developed... Outreach guidance for comms success stories and KM sharing of technical knowledge and learning.” — EPIC Team Lead The process to develop content was noted as a challenge initially but improved over time. Interview respondents described limited staff resources and an unclear strategy and process to consolidate global education information for easy access at the beginning of EPIC. As the project progressed, the KM team collaboratively developed a content strategy process to help DDI/EDU and EPIC prioritize and coordinate the development of DDI/EDU knowledge products. The process included cross-team planning meetings and editorial calendars to guide the entire cycle from content development to editing to product design to monitoring and evaluation of disseminated materials. “I think EduLinks is a good start. Every time I’ve asked to add something, people do it quickly. It’s been more of a struggle to get new knowledge produced. There are a few reasons. Heather, who is in charge of KM, was wearing more than one hat. Had limited time. Part of it has been my time, my availability to dedicate to moving things forward. I think we don’t look at things comprehensively... We are encouraged to not work beyond our immediate team. It goes back to time – people are stretched. No team is fully staffed.” — DDI/EDU Team Lead “In the beginning of the project, we were definitely in a place of trying to define what the bucket of KM was for this Office and how that translated to the work areas. In the first days, there was very little infrastructure in DDI/EDU to support KM- systems and processes. Lots of content but scattered and not uniform. At the outset, a realization for me and the project was that where KM added value for this Office was in being a content aggregator and packager and less on the knowledge generation... This workstream has tried to be proactive where possible and reactive when things change.” — DDI/EDU Technical Specialist “...there is a much more coherent approach to how knowledge is accessible. There is the push side with newsletters, and the pull side where people are going to the platform to get information themselves. Power of bigger picture thinking of what the Agency is doing in this space… Strategic packaging of information has come an incredible way.” — DDI/EDU Sr. Management IR 3.2: RELEVANT EVIDENCE AND LEARNING CAPTURED AND SYNTHESIZED. Progress was made in the capture and synthesis of relevant evidence and learning regarding gender, disability, information, and communication technology for education (ICT4E), and education finance programs. EPIC completed a landscape review for each QUEST area to identify needs and give guidance 75 | EPIC EVALUATION REPORT USAID.GOV around the program cycle. According to EPIC and DDI/EDU staff interviewed, EPIC support was crucial in identifying and addressing activity needs. The Gender How-to-Note, for example, was developed following the QUEST landscape mapping. The KM team continued to facilitate access to knowledge and good practice in these quality, equity, and sustainability areas in Years 4 and 5. Original KM products centered around five engagement campaigns: Disability Inclusive Education, Young Women Transform, Social and Emotional Learning, Education Finance, and Higher Education. EPIC supported DDI/EDU in increasing knowledge product engagement among staff through content and website management (e.g., EducationLinks, GRN). Top 10 Principles for Developing ICT in Education Programs, Importance of Accessible Learning Materials, disability inclusive education, youth workforce development, reading and literacy, higher education, and systems strengthening were among the most accessed blogs and top content searched on EducationLinks. Additionally, networking and collaboration events were opportunities to share evidence and learning. As the Secretariat of Building Evidence for Education (BE2), EPIC coordinated and facilitated teleconferences and meetings with the Steering Committee to share the group’s work with the larger education community (e.g., the Comparative International Education Society). BE2 included representatives from USAID, World Bank, United Nations International Children’s Emergency Fund (UNICEF), and the United Kingdom’s Department for International Development (DFID). Participation in the BE2 Annual Conference and Steering Committee meeting in fiscal year 2019 nearly doubled to more than 80 participants compared to previous years. The Mobiles for Education Annual Symposium convened more than 350 practitioners, thought leaders, policymakers, and funders from 46 countries to explore topics including building digital communities to support education, the central role of the teacher and community educator and technology to support their professional development, and instruction in traditional and non-traditional educational settings. Most 2019 participants surveyed (94 percent) were satisfied with the Symposium, and attendees appreciated the opportunity to network with colleagues to learn about emerging technology. As one Symposium Steering Committee member stated: “The feedback we have received from everyone on this year’s Symposium has been overwhelmingly positive. The number of times that colleagues remarked to us that ‘I never knew that anyone was doing ___’ or ‘this is like a crash course in all things edtech’ … is testimony to how practical and useful this year’s event was to our staff.” — Symposium Steering Committee Member Despite some progress, DDI/EDU and EPIC team members felt that capturing and synthesizing evidence and learning deserves more attention. Some DDI/EDU staff had higher expectations that EPIC would provide deeper technical analysis and reporting. Other DDI/EDU respondents noted that mechanisms such as Data and Evidence for Education Programs (DEEP), which DDI/EDU uses to generate and increase use of evidence, should provide the content that EPIC could then organize or transform in different ways to increase access. EPIC has worked closely with DEEP and found that further work was needed to establish a more reliable data and evidence base that could then be packaged and published. Interview respondents suggested the following to strengthen results in this area in the future: 1) conduct in-depth stakeholder analysis to better frame what evidence is captured and how it is synthesized and shared; 2) listen to the field when prioritizing which evidence is generated, captured, synthesized, and shared; and 3) clarify across DDI/EDU staff the role of EPIC and other mechanisms in generating, capturing, and synthesizing evidence and learning as well as the role of DDI/EDU staff in ensuring technical quality. 76 | EPIC EVALUATION REPORT USAID.GOV “What are we learning and feeding that back into our programming, we don’t do that enough yet. We’re still pretty siloed even though we have a new policy. Ways that we can break them down and share knowledge – it’s not just about sharing knowledge but using it to inform our work and see how it can benefit what we do... My interest is more feeding knowledge into practice. I’d like to strengthen that.” — DDI/EDU Technical Specialist “EducationLinks has been successful. As much as we can get important information that is simplified and synthesized. DDI/EDU putting themselves into the shoes of Education Officers/the field. Sensitize things with the field. Know how they can be involved, what the entry points are. Could play a really important role in letting the Office know how to work more effectively with the field, what people are saying they need and want. There’s an opportunity to do more there.” — DDI/EDU Technical Specialist IR 3.3: OFFICE OF EDUCATION VISIBILITY IS ENHANCED THROUGH THE DESIGN AND LAUNCH OF AN EXTERNAL EDUCATION SECTOR PLATFORM. Both the volume of DDI/EDU knowledge products disseminated (as described under IR 3.1) and the usage of the products suggest an increased visibility of the Office through EducationLinks. One DDI/EDU Team Lead described the establishment of EducationLinks as “a massive effort and improvement in communication … a great tool to have!” An EPIC representative described the visibility that EducationLinks offers as “pivotal because it was also the face of the sector; a robust platform with great information.” EPIC employed strategies such as an EducationLinks Roadshow and content submission forms to increase awareness of EducationLinks and engage users. The EducationLinks Roadshow included presenting to Mission staff during EEPM and ED-CVE courses; sharing materials with DDI/EDU staff at conferences; and meeting with partners (e.g., Basic Education Council, Global Partnership for Education, Mobiles for Education Alliance) to discuss learning products and effective sharing practices. Since its September 2018 launch, EducationLinks’ site visit and engagement metrics have grown steadily. During its first year (FY 2019), for instance, the site had 25,304 sessions, 52,993 pageviews, and 17,821 users. To date, engagement has increased significantly (up to six-fold), with more than 163,800 sessions, 280,300 pageviews, and 108,200 users. Users read approximately 31 percent of all the pages viewed, and about 30 percent of users who visit a Resource page download the resource. The average time on page is 2.6 minutes. More than 210 countries have accessed products posted on EducationLinks, and more than 100 countries had at least 100 unique pageviews. Organic search engine and social media referral traffic increased monthly as EducationLinks pages were indexed and accessed via search engines and shared on social media accounts. As shown in Figure 7, except for the period during the government shutdown, the number of engaged users, or those who visited one or more blog, resource, or event page, increased over time. Engaged site users increased an average of four-fold from 3,642 per quarter in FY 2019 to 14,550 per quarter in FY 2020. 77 | EPIC EVALUATION REPORT USAID.GOV FIGURE 7: EDUCATIONLINKS USER ENGAGEMENT “The website is great. EducationLinks is a huge improvement from what we had before. It looks great. Was really great to have something external facing to share information with external partners to be on the same page. That is a huge improvement on what we had.” — DDI/EDU Technical Specialist “[Knowledge management] has gotten better. EduLinks is really great… Great to see how that has taken shape nicely. It’s a good resource. EPIC has played a really valuable role in this… We did a lot of work to get the [progress report] out in 2017 that coincided with the Education Strategy. We did another one that wasn’t as flashy and high visibility. I think this goes along with a process that is missing. Haven’t been doing more outward-facing products, that is a gap.” — DDI/EDU Technical Specialist “The shift away from siloed teams is something EPIC has supported in a lot of ways, including EducationLinks. Instead of having multiple websites managed in different ways with different pots of money, we combined them all and worked with education in conflict and crisis to bring that under Edulinks. Bringing together all the content into one place and the operational elements to get there forces conversations to happen in the Office about what everyone’s creating.” — EPIC Team Lead “There have been a lot of changes. There were a couple of significant developments in this area. Getting to a common spoke and hub web presence that would be lasting and give us the ability to give metrics and so that we don’t have our central mechanisms close and have dead websites. It’s a living resource that people are going to remember and use. That was a lot of work that Heather led with EPIC… That had been a vision for 15 years and we had never been able to pull it off. The kinds of things that were funded served a useful purpose in the Office for things that wouldn’t have been funded otherwise... EPIC was novel in this sense... There was a categorization of the knowledge products so there wasn’t everything under the sun. That was positive. It hasn’t totally stuck, but that was huge.” — DDI/EDU Sr. Management “One huge lift that EPIC was involved with was the EducationLinks website. People in the field found it really helpful. It wasn’t marketed to the internal Office that well for DDI/EDU staff. I know a lot of work went into that... Definitely did contribute to achievement of the goals. Without them, it would have been a sad scene, I have to say. They did EducationLinks and a lot of resources that didn’t exist before…” — DDI/EDU Technical Specialist 78 | EPIC EVALUATION REPORT USAID.GOV IR 3.4: CLA APPROACH STRENGTHENED, DOCUMENTED, AND SHARED. The KM workstream has actively collaborated with other workstreams and mechanisms. As a result of this collaborative approach, information was shared across teams and products were developed in an iterative process that focused on feedback and effectiveness. In addition, the KM team was responsible for developing templates and guidance on how to create KM materials, which retrospective interview participants credited with contributing to a unified, coordinated Office. EPIC staff have leveraged each other’s work when they engaged collaboratively and iteratively, as demonstrated by the design of products, tools, templates, and content that were developed through KM, ECP, and PD cross-team efforts. Additionally, EPIC worked with other mechanisms (ECCN, GRN, Young Women Transform, and DEEP) to create and share content and develop websites. EPIC intentionally built trusting relationships with these mechanisms, which resulted in a successful launch of ECCN’s microsite and the socialization of DEEP’s Indicator Report Guidance. This collaboration created a more practical, engaging, and modern learning approach to support the effective work of USAID Mission and implementing partner staff. With respect to learning and adapting, the KM workstream has designed and maintained the EducationLinks website as a central learning point for USAID’s internal and external global education sector partners. The KM Team met weekly to ensure responsiveness to requests and make informed decisions around its purpose and objectives. The KM workstream collaborated with the PD and ECP workstreams to review data such as monthly website analytics and outreach survey responses to understand resource usage, inform content development and dissemination strategies, and adapt strategies as needed. The EPIC team used the results to guide knowledge sharing and learning, and collaboratively develop strategies to refine and expand EducationLinks (e.g., via Facebook, LinkedIn, and newsletters). EPIC also led bi-annual website strategy meetings, which prioritized content for the next six months, facilitated reflection on achievements and challenges, and discussed strategies and processes to adapt. Follow-up survey responses about application of EPIC resources also reflected a CLA approach. The most frequently cited reasons DDI/EDU staff used EPIC-supported knowledge products included to implement a practice (n=7 of 11), monitor a program (n=5), strengthen service delivery (n=5), share with others (n=4), develop or revise a policy (n=3), and change a practice (n=3). Key informant interviews revealed opportunities to strengthen the CLA approaches applied throughout the EPIC project. Collaboration with other DDI/EDU mechanisms can be enhanced, particularly regarding evidence generation, capture, and synthesis. Collaboration with the field also can be strengthened to ensure that KM products and processes address field needs. “The Office does a pretty good job of using EPIC strategically and being clear about who the expected collaboration is with – staff, another mechanism, etc. They’ve been clear about that. I think the work that PD, KM, and comms has done through a monthly collaboration meeting – it took us a while to figure out the purpose of the meeting. Decided to look at content collaboration and now everyone understands the value. We’ve made headway in that.” — EPIC Technical Specialist Something that hindered it is the nature of the mechanism. For procurement, we would like to have a good CLA approach. Shutdown last year, accelerated procurement timelines this year were challenging. Didn’t have a good CLA process in place. Direct hires were also factors, having enough and the right people to do that work. We don’t often use AARs when things go well. Our approach to them is very punitive. When stuff goes well, we don’t want to capture it and we miss the opportunity to capture people’s expertise... What EPIC has done a really good job of is put in place the right tools to help us execute. We are still trying to learn how to execute them.” — DDI/EDU Team Lead 79 | EPIC EVALUATION REPORT USAID.GOV “For EPIC, the challenge was that there were a lot of things in it. In some ways, it’s almost five different things. Those could have worked together really well if they all had champions that worked together and provided a significant amount of resources for efforts to really manage this well. Bundling these things together was more powerful than not. Training, KM, OE, comms all work together to support the knowledge base, come together in a video, a training course, etc. That is a good practice. Good results combining them.” — DDI/EDU Sr. Management “On the CLA IR, I haven’t seen the Office reference an Office-wide CLA approach. It’s not that they aren’t working toward aligning that approach, but I don’t think it’s an intentional CLA approach like in some other parts of the Agency. EPIC is organized differently. The workstreams support CLA but I don’t think it was designed with the Agency CLA approach...That’s a place that could be strengthened. To what extent does the Office want to align and intentionally work towards the broader Agency CLA approach?” — EPIC Technical Specialist RECOMMENDATIONS AND LESSONS LEARNED DEVELOP A COMPREHENSIVE STRATEGY AND ENSURE COMMITMENT TO KM FOR EFFICIENCY AND EFFECTIVENESS. A lack of a clear KM strategy, understanding of priorities, and direction from DDI/EDU to guide areas of focus were noted as primary challenges at the beginning of EPIC. Some understanding of the importance of capturing and disseminating knowledge to make information broadly accessible and advance global education existed, but there was less attention on the KM implementation approach. As the project evolved, the EPIC team worked with DDI/EDU to develop a KM and communication strategy that determined content areas of interest, identified effective tools and resources, and considered coordination and outreach approaches to develop and disseminate relevant products. To maximize effectiveness and efficiency, a comprehensive KM strategy should be aligned with priority content areas of DDI/EDU’s technical strategy. This would strengthen KM systems, particularly the ability to monitor and evaluate use. The KM strategy should be based on clear direction from leadership, buy-in about the utility of a strong KM process, an assessment of audience information and accessibility needs, and clearly outlined content design with ongoing assessment of KM needs, reach, and impact. Recommended components include a clear articulation of priority content areas; expertise needed to develop products; communication approaches; education and outreach methods; content development schedules; editing and quality assurance methods to ensure consistent, uniform messaging; and M&E methods. “There’s still a gap when E3 says they want something to happen and then EPIC gets excited and then E3 backs out of it or says that wasn’t what we want. May be an issue of EPIC explaining it clearly or may be the Office that doesn’t understand or prioritize the value add. Don’t always ask how things are sustainable or not. Common across workstreams…” — EPIC Technical Specialist “...I think a lot of it is again the receptiveness of people in the Office and their willingness to engage with the topics. It goes back to my point about there being a lot of talk but no action. The Office had EPIC do a lot of guides and quick tips and efficiency-improving documents. The team spent a lot of time doing those and they were beautiful, but the Office didn’t use them. There was no clear plan about how to use them. That hindered some of the success... Can’t fault EPIC for that. The Office should have supported uptake of it.” — DDI/EDU Technical Specialist 80 | EPIC EVALUATION REPORT USAID.GOV “There has been a lot of progress made in trying to be more strategic about what knowledge products are made and knowing what knowledge exists. Process for having a standard for creating knowledge products. Not just making a PDF. How to create knowledge products. Know what you can create to help people digest information and know what to do with it. The Office has made a lot of strides in getting their staff to think about these things. EPIC helps to create companion products. They’ve gotten a better handle on coordinating the creation of products.” — EPIC Technical Specialist BALANCE BREADTH AND DEPTH OF PRODUCTS AND ATTEND TO OUTPUTS AND OUTCOMES TO ENSURE KM MATERIALS ARE USEFUL AND HIGH IMPACT. The variety of KM products developed was a strength of EPIC. Consistent with a comprehensive strategy, some concerns were raised about the balance of the breadth and depth of developed content. Some respondents recommended developing fewer, more in-depth products. Recognizing the potential to focus on outputs, respondents also recommended thinking critically about the desired outcome. “Edulinks has been hugely helpful… We need to spend more time on implementing and socializing what we have rather than produce more products. Do fewer things and do them more in depth.” — DDI/EDU Sr. Management “In the future, could maybe conceptualize KM to be more active… KM is more an output. What is the actual desired outcome of KM? … Because it’s been so successful, this could be a moment to determine why it was successful. Could pit the other areas around what they did in KM to see if they can do similar things to what KM did. Ask ourselves those questions.” — DDI/EDU Team Lead DETERMINE AND OBTAIN THE EXPERTISE NECESSARY TO DEVELOP PRODUCTS THAT INFORM DIVERSE AUDIENCES ABOUT VARIOUS ASPECTS OF GLOBAL EDUCATION. Several interviewees discussed the technical and content expertise necessary to develop quality KM products. Respondents recognized the unique communications and design skills that the EPIC team used to produce engaging materials that are accessible to technical and lay audiences. Some respondents also wanted the KM team to have more education subject matter expertise to assist with writing. Additionally, other respondents suggested that more SMEs were needed in particular areas such as secondary education, so all aspects of education are well represented. Careful consideration of the team composition and requisite content area and technical design expertise are important to ensure that the most informative KM products reach and resonate with the broad audiences. “The technical work is prized over the communications work. A challenge I have is to talk with people about why it’s important to talk with anyone about what we do, not just technical people. It’s not ‘dumbed’ down. You are talking with people at a high level but it’s a broader group of people that benefits from not talking in highly technical terms. Staff was keen on mastering the messaging when we had a new strategy, and I don’t think we devoted the same energy to it as we did with the new messaging around transformation... For the new education and strategy and policy, I still think we need good messaging.” — DDI/EDU Team Lead For me, what EPIC has done most is laying out and designing knowledge products. Content was developed elsewhere... Haven’t had much experience developing content with EPIC except for blog posts. I’m very happy with the layout and design work... For the blog posts, I felt like the writers weren’t always able to pick out the really pertinent key points if I fed them information. We have shifted to where SMEs are the main writers. I get that EPIC staff was never meant to be SMEs in our technical areas. It would have been beneficial to find people who had education backgrounds along with communication background.” — DDI/EDU Team Lead 81 | EPIC EVALUATION REPORT USAID.GOV “When I came in, we were very much focused on learning and knowledge for reading programming. Higher ed – because there wasn’t anyone here, we didn’t do anything – but you’re here now and can do things. Youth Ed did a lot of work on learning. It’s been hard to find the right SMEs to help with this... We haven’t been equally supportive of the various levels in the continuum…” — DDI/EDU Team Lead ENSURE CROSS-TEAM COLLABORATION AND INTEGRATION OF KM THROUGHOUT WORKSTREAMS. Numerous EPIC and DDI/EDU staff agreed that the Office benefited from cross-workstream collaboration. The workstreams supported each other and aligned organically. For instance, an EPIC team member felt that “all courses rely on KM.” EducationLinks informed the PD-supported online courses, and the courses informed participants about available knowledge and learning resources to increase access and use. Messaging is an important consideration in OE, PD, and ECP, and can assist with advocacy and stakeholder engagement. Future efforts should take advantage of the synergies of the workstreams to maximize their strengths and address challenges experienced around clarifying the roles and responsibilities of the workstreams. “...I think staff were more committed to messaging around the new strategy because they had to talk with people in the field about it and they weren’t as interested in the high-level messaging. We still have trainings coming up and can work some high-level messaging into that and advocate for advocacy in those trainings. That’s a place where PD and communications can overlap.” — DDI/EDU Team Lead “...I have appreciated that there is always a KM lens whenever we are developing anything training related. Heather and Lisa are always attentive to how we can capture learning...They’ve been open to suggestions from our team on accessibility for all types of learners… If there is a video, the interpreter gives a description of the graphic. I’ve appreciated that.” — DDI/EDU Technical Specialist “Over the last year, there has been a lot more collaboration among PD, KM, and comms. PD handled what happened in the sessions, substance of the events. KM steps in and posts and promotes and captures and shares knowledge. It’s been fun. Something different that EPIC hadn’t had the opportunity to do before those major events…” — EPIC Team Lead USE CLA INTENTIONALLY, STRENGTHEN EVIDENCE CAPTURE AND SYNTHESIS, AND EVALUATE KM ACTIVITIES TO MAXIMIZE IMPACT. Respondents suggested that CLA was critical to strengthen the Agency and advance the field. EPIC strategically integrated CLA in its work with DDI/EDU. Beyond development and dissemination of knowledge and learning products, DDI/EDU staff discussed the necessity of applying knowledge and understanding the results of the Office’s efforts. Some respondents emphasized the need to capture and synthesize evidence that reflects the diverse voices in global education and represents the needs and interests of the field. DDI/EDU should increasingly use CLA intentionally, strengthen evidence capture and synthesis, and evaluate KM activities to maximize impact. The metrics and systems that the KM team established are particularly useful in this analysis. Product dashboards, for instance, capture information about the location, search tools, and topical interests of users. It will be helpful to build upon these data as well as co-create products with stakeholders in the field when possible for the most relevant and usable knowledge and learning resources. “…it’s not just about sharing knowledge but using it to inform our work and see how it can benefit what we do... My interest is more feeding knowledge into practice. I’d like to strengthen that.” — DDI/EDU Technical Specialist 82 | EPIC EVALUATION REPORT USAID.GOV “Part of this is that there is bias in terms of what is being captured. The Office of Ed’s priorities may not be reflective of the field priorities. It was a very top-down approach rather than bringing in the field’s voice to determine what is produced.” — DDI/EDU Technical Specialist “Created a lot of value with tools, policies, platforms to facilitate knowledge sharing. Where we didn’t get was using all that to really hear from the field. I think DDI/EDU would tell you that they always hear from the field and that’s why they are doing what they are doing, but EPIC is not at all involved in those conversations if they are happening, and they probably are. We are one step removed, which makes it harder to tailor those products to the people we serve.” — EPIC Team Lead ENGAGEMENT, COMMUNICATIONS, AND PARTNERSHIP SO 4: COMMUNICATE DDI/EDU’S WORK AND SUPPORT MORE EFFECTIVE ENGAGEMENT WITH USAID/WASHINGTON, MISSIONS, OTHER USG OFFICES, GLOBAL STAKEHOLDERS, AND PARTNERS The strategic objective of the Engagement, Communications, and Partnership (ECP) workstream was to communicate DDI/EDU’s work and support more effective engagement with USAID/Washington, Missions, other USG offices, global stakeholders, and partners by: • Increasing understanding of DDI/EDU’s brand message, and goals amongst internal and external actors. • Strengthening partnerships that achieve USAID’s education strategic goals. • Strengthening the capacity of DDI/EDU staff to advocate for USAID’s education work. • Strengthening, documenting, and sharing DDI/EDU’s approach to CLA. Thirteen percent of the total of 24 DDI/EDU interview respondents believed that this strategic objective was achieved, 50 percent believed that results were mixed with advances in some but not all areas, one person (4 percent) stated that this strategic objective was not achieved, and one-third of respondents (33 percent) stated that they did not have enough knowledge of the advances made toward this objective or did not specifically address this question. While respondents noted significant advances in some areas (especially in increasing the understanding of DDI/EDU’s brand message and goals), and they appreciated the high quality of the work that EPIC was tasked to do, several respondents stated that this workstream did not appear to be a DDI/EDU priority, that DDI/EDU’s Engagement Team wasn’t sufficiently staffed to lead and manage this workstream, that EPIC took more than a year to find the right staff to support this workstream, and that there was no coherent strategy to guide this workstream’s work. “Absolutely. The reason I say yes is because across the board, we stepped up to take a leadership role in professional orgs. Put USAID’s work up front and center. Gave staff visibility. We lost some staff because of that, which is supposed to happen. We joined conversations and brought them into ours.” — DDI/EDU Sr. Management “Some of it was a success and some not. Branding and messaging has definitely been improved. There is a lot of talk there but the results aren’t concrete. A missed opportunity was when the Ed Policy came out, there was supposed to be a huge rollout with Missions and the Office and right when all of this was 83 | EPIC EVALUATION REPORT USAID.GOV happening, the person who was supposed to be in charge of this left and the person who was supposed to pick it up didn’t do it and it didn’t happen. I still am in disbelief. It happened in piecemeal but there wasn’t a full implementation. That was not EPIC’s fault. They tried as much as they could but there wasn’t appetite for it. I don’t know what happened. There were a lot of conflicting opinions and there wasn’t leadership directing it. I also think that the fact that because it’s the Education Policy, there aren’t consequences if it isn’t communicated. That takes away motivation. Compared to other sectors that have strict reporting requirements to Congress, if that was the case here, maybe there would have been a bigger push. The Education Office does need to report to Congress, but the requirements weren’t as strict.” — DDI/EDU Technical Specialist EPIC’s support came through the creation of standard templates, newsletters, and communications products that DDI/EDU staff could use during temporary duty (TDY) and engaging with multiple stakeholders (e.g., in the development of the GEC and GELS). Many DDI/EDU and EPIC respondents felt that engagement and communications activities would have benefited from a more in-depth stakeholder analysis and comprehensive engagement and communications strategy. Some mentioned that an important opportunity was lost when the planned launch of the new education policy was not implemented. A number of respondents commented that responsibility for not having a more coherent strategy and the lack of follow through on the policy launch was more on the USAID side. EPIC tried to promote development of a more comprehensive communications strategy, but the comms team on the USAID side was small, had limited time and resources, and prioritized responding to urgent requests from the front Office, Congress, and others. Additionally, several respondents mentioned barriers to implementing the Education Policy launch, which resulted in delays (e.g., leadership transition, Agency transformation process). “The comms training and coaching that we have done has been targeted at making a dent on this. We haven’t gotten feedback on how effective the coaching has been. In addition to that, the management of the external newsletter and FB and Twitter, we’ve gotten props from LPA after we took over managing social media. [The Team Lead] mentioned that LPA told her that education had really stepped up their game. In many cases, because of processes they are used to, we haven’t been able to be very social on social media. We are only allowed to talk about ourselves. It limits our effectiveness when we aren’t allowed to apply best practices. We at least professionalize how the Office comes off to the world.” — EPIC Team Lead “There are certain products we put out with EPIC support (webinars, fact sheets, etc.), and our ability to engage with partners has been really strong. I can’t accurately say how it was before. They didn’t have much staff before...Before, the Office was trying to do it all on their own with internal resources that led to jargon and things that were hard to read. With Forum One and TRG, there was an external perspective to help people translate the messages from technical jargon to messages and visuals that people could understand. The strategy and policy are so much more clearly written this time. Easy to read and understand for anyone. [EPIC] was phenomenal to work with. Put us on a timeline. Lisa (EPIC’s Comms Team Lead) is one of the strongest comms professionals I’ve worked with. I’ve never seen her lose her calm. She’s always thoughtful and kind and consistently puts out really strong products. Daryn with recent webinars has really helped the Office think through how we communicate externally in the context of COVID. Gina helped keep us on track – provided strong support, too.” — DDI/EDU Technical Specialist “We were pretty successful in helping the Office move from the traditional, default [GEC] conference thinking where people come and stand and talk about a paper. It’s boring, no adherence to adult learning principles. In 2018, EPIC wanted to make sure that that shift occurred. We did, given the proposals for topics, the format. TED talks, fail fast stories, discussion panels that are facilitated discussions (not PPT, PPT, PPT, Q&A). There was some pushback, but we brought people around to doing this differently based on 84 | EPIC EVALUATION REPORT USAID.GOV adult learning. The GEC ended up being cancelled. But that was an opportunity – it was a big win for the Office. They were going to postpone it, but we said we didn’t think that was wise. We didn’t know a lot about the virus. We could have been in the same position in October. Why not try to do this online. There were lots of concerns. Infrastructure, costs, etc. Let’s do this online. We will reach a lot more people and we’ll get better at doing online education. Two weeks ago, we started it. Transformed proposals for the in￾person conference into online delivery. I hope this will be remembered as a pivotal moment to shift the sector to online without having to have anyone leave their offices. Shouldn’t limit ourselves to doing just in￾person conferences.” — EPIC Team Lead The following sections describe the interventions that EPIC co-designed and implemented with DDI/EDU, the outcomes they produced that contributed to each ECP IR, and the factors that facilitated and hindered progress. Key events over the life of the project in this workstream are presented in Annex 4, Table 4. A summary of the factors that facilitated and hindered the success of EPIC’s ECP efforts is presented in Annex 5. IR 4.1: UNDERSTANDING OF DDI/EDU’S BRAND MESSAGE, AND GOALS AMONGST INTERNAL AND EXTERNAL SECTORS INCREASED. Of the 24 DDI/EDU interview respondents, 67 percent stated that EPIC had contributed to the achievement of this IR and that significant progress had been made, 8 percent stated that results were mixed, one person (4 percent) stated that this IR was not achieved, and 21 percent did not feel they had enough knowledge of this IR to judge or did not address this question. Interview respondents spontaneously cited the following support that EPIC provided, which they believed contributed to the achievement of this IR. The numbers in parentheses present the percentage of the total of 33 interview respondents (total includes 24 DDI/EDU and 9 EPIC team interviewees) who specifically cited these interventions. • Communications and knowledge management products (55 percent) • Conferences and large meetings (45 percent) • Face-to-face and online training (24 percent) • Newsletter (24 percent) • Social media (15 percent) • EducationLinks platform (12 percent) Other EPIC support mentioned less frequently (three or less times) included: communications coaching, consulting/being a thought partner, cross workstream collaboration meetings, leadership development training, learning agenda, off-site meetings, support to the Global Book Alliance, process and systems design, and strategic planning. These types of products and interventions contributed to various initiatives and outcomes, which are described below. Communications Strategy Support In 2017, EPIC conducted a brand audit for DDI/EDU including survey input, data analysis, and report write up on a communications survey of external and internal stakeholders. The report identified clear trends to inform DDI/EDU’s messaging and collateral development and offered recommendations to 85 | EPIC EVALUATION REPORT USAID.GOV strengthen DDI/EDU’s brand and messaging. These analyses helped the Communications Team formulate a strengthened communication strategy, which was presented to stakeholders within USAID, including the Bureau of Legislative and Public Affairs (LPA). Global Book Fund EPIC provided support for the Books for Every Child campaign of the Global Book Fund, including developing a website to disseminate materials, educate key stakeholders, engage strategic partners, show impact and results, and brand the program. Education Policy Support Beginning in Year 2, EPIC collaborated with DDI/EDU to develop an Education Policy Facilitator’s Guide. In Year 3, EPIC supported DDI/EDU to develop and implement a plan to socialize the Education Strategy and Policy. Additionally, EPIC provided support in the design and copy editing of the 2018 Progress Report, Education Strategy, Education Policy, and Education Implementation Guidance. EPIC also worked with DDI/EDU to develop three portable banners. In Year 4, EPIC supported the dissemination and socialization of the USG Education Strategy and USAID Education Policy to USAID staff and external stakeholders. Support included the design of French and Spanish versions of the Education Policy, the development of new fact sheets that align with the Education Strategy and Education Policy, and an update to the DDI/EDU internal Facts and Stats sheet. The translated policies were posted on USAID.gov in late March 2020. Eight fact sheets were completed and posted to the usaid.gov/education page. The Education Policy Facilitator’s Guide will help USAID staff communicate effectively about their work under larger Agency messaging. The completed guide and related materials were shared from USAID’s Google Drive with the sector through the internal newsletter. Communications Systems & Processes In Year 3, EPIC began to provide more formalized, sustained, and strategic communications support to DDI/EDU. EPIC collaborated with the DDI/EDU Engagement Team to define, structure, and set up systems to support events, campaigns, and social media management. Print and Online Engagement Material EPIC supported the DDI/EDU Engagement Team when needs arose for print or digital engagement materials. In Year 1, EPIC supported the development of the Books for Every Child website, designed a Book-a-thon booklet and bookmark, and facilitated a brand analysis kick-off meeting to scope this activity. In Year 2, EPIC developed an Office of Education Booklet as the first piece of communications material that aligned with the new brand and messaging. This booklet was used with a range of stakeholders to communicate the work of the Office (prior to its transformation to the Center for Education). In Year 3, EPIC provided support in the design and copy editing of the 2018 Progress Report, Education Strategy, Education Policy, and Education Policy Implementation Guidance, as well as the development of three portable banners. EPIC helped DDI/EDU manage and disseminate its external newsletter and social media platform (Facebook and Twitter) including weekly social media posts and social media toolkits for the launch of the Education Strategy and the Global Youth Economic Opportunities Summit. EPIC also updated the gender in education page on usaid.gov. In Year 3, there were an average of 485 Twitter engagements per month compared to 312 per month before EPIC managed the account; 316 engaged Facebook users 86 | EPIC EVALUATION REPORT USAID.GOV per month: and 328 newsletter clicks per newsletter compared to an average of 260 clicks per newsletter before EPIC managed the newsletter. Following EPIC’s management of the social media channels, LPA praised DDI/EDU for “taking their social media to the next level.” “The professional newsletters, internal and external, are both appreciated by the field and sets up the Office as being a leader in this sector.” — EPIC Management “We’re so tangled up with what we can get through LPA and do. We have good materials coming out. I’ve liked the videos. I wish we can do more, which isn’t a reflection on EPIC, it’s Agency guidelines.” — DDI/EDU Sr. Management In Year 4, EPIC developed a Creative Brief to shape the messaging to roll out USAID’s Education Policy to external audiences and managed the external newsletter and social media channels. The Year 4 Annual Report indicated that USAID’s external newsletter and social media traffic increased that year compared to the previous year: 697 Twitter engagements per month in Quarter 4 (vs. 485 per month in Year 3), 372 engaged Facebook users per month (vs. 316 per month), 357 clicks per newsletter in the first half of Year 4 (vs. 328 clicks in Year 3) but dropped to 63 clicks in the second half of the year. In Year 5, EPIC continued to manage the external newsletter and social media channels. Traffic increased that year: 1,293 Twitter engagements per month (267 percent increase from Year 4); 642 engaged Facebook users per month (203 percent increase from Year 4), and the newsletter open rate was 50 percent, which was similar to Year 4. EPIC created the following promotional and social media toolkits: Education Reporting Guidance; ICT4E How-To Note Toolkit; Young Women Transform GYEO Toolkit; Higher Education Retrospective and Landscape Analysis Toolkit; Global Education Learning Series; USAID-LEGO Partnership; Education Response to COVID-19; International Literacy Day; International Day of the Girl; and Return to Learning. EPIC created 302 Facebook posts, receiving an average of 641 engagements per month, up from 240 in Year 4 and 102 in Year 3, and 1,020 tweets receiving an average of 1,293 engagements per month, up from 504 in Year 4 and 485 in Year 3. Content for social media posts often was aligned to relevant international days and included promotion of significant knowledge products created by DDI/EDU such as the Education Reporting Guidance and Higher Education Retrospective and Landscape Analysis. One interview respondent commented that the newsletter development process has evolved over time and that it may be helpful to continue to update any changes in the process and socialize these changes so that staff are aware of the overall communications strategy for these products and how and when they can contribute. Education Today, Opportunity Tomorrow Video and Brochure In Year 5, EPIC completed and shared a promotional video and brochure across social media channels as part of the Education Today, Opportunity Tomorrow Creative Brief. It was also used to launch the first Global Education Learning Series (GELS) event. The video received 176 views, 75 Twitter engagements, and 40 Facebook engagements. EducationLinks Engagement Strategy EPIC and Forum One worked with DDI/EDU to implement a strategy to support engagement with internal and external EducationLinks audiences to deepen use of site content, increase representation across policy areas, increase traffic from the DDI/EDU internal newsletter and RSS feed referrals, and increase site engagement from social media referrals. Under this engagement strategy, EPIC tested 87 | EPIC EVALUATION REPORT USAID.GOV different locations for the call-to-action to submit content to EducationLinks in internal and external newsletters, added sign-up forms for RSS to email newsletters, and promoted topic email sign-up forms through internal newsletters. Additionally, EPIC created a new email list through the GELS registration that can be used as the basis to start an EducationLinks-specific newsletter. Figure 8 presents social media and newsletter engagement during Years 3 to 5 which indicates an upward trend in average monthly social media engagement and a decrease in the average number of newsletter clicks, although newsletter open rates remained about 50 percent in Years 4 and 5. Based on these results, EPIC planned to review the newsletters and recommend changes to improve newsletter engagement in the future. FIGURE 8: SOCIAL MEDIA AND NEWSLETTER ENGAGEMENT IR 4.2: PARTNERSHIPS THAT ACHIEVE USAID’S EDUCATION STRATEGIC GOALS STRENGTHENED. Seventeen percent of DDI/EDU interview respondents said that this IR was achieved with EPIC support, 25 percent believed that it was partially achieved, 8 percent believed that it was not achieved, and half of respondents (50 percent) did not feel they had enough knowledge about this IR to comment. Respondents described a situation when the EPIC project began in which individuals within the Center each had their own personal relationships with other organizations, individuals, and alliances. There was no Office-wide partnering strategy. EPIC hoped to collaborate with USAID on developing a more strategic approach to partnerships; however, this did not emerge as a priority of the Office. “I think that in this area EPIC has been very good in responding to things that we’ve asked for and making sure that the products are as strong as possible. They follow our lead on this, which is what they are supposed to do. Wouldn’t necessarily say that we have stronger partnerships because of it. I’d see that more in other contracts where we engage with external partners. EPIC’s focus is more on telling our story better and getting it out through various channels like social media. I don’t know that it was their mandate, and it was what USAID wanted.” — DDI/EDU Technical Specialist “The internal Office didn’t feel like we needed partners because they were the experts. That has gotten better. [E]CCN and GRN are externally focused on technical issues. They bring a lot to the table.” — DDI/EDU Technical Specialist “With public-private partnerships, we’ve made good strides. Having a relationship manager has been good. We still have a lot of partnerships that are personality based. Someone wants to keep this project or activity going rather than being linked to the strategy. Want to link partnerships more to the field. Why we’re supporting the global partnership – why we’re doing it and what it contributes – that’s been an area of 88 | EPIC EVALUATION REPORT USAID.GOV improvement. Overall, I’m not sure of all the partnerships we have across the Office. Not sure how strategic they are. Probably need to look at that.” — DDI/EDU Sr. Management “While the Agency has been talking about doing business differently, that wasn’t integrated into the Office and how they did things. If they were doing business differently – use vehicles and partnerships outside of our traditional funding. What are other ways we can affect change beyond our traditional programming. We are only one donor in that space. Activities that bring in other stakeholders. Recognizing other networks.” — DDI/EDU Technical Specialist “This is an area where we tried to work with the Center to have conversations about the value of the partnerships that they are in, what synergies, exchange of evidence and experiences could be had across partnerships, and with the whole Center and not just the POCs for that alliance, network, or partnership. It’s taking up a lot of their time and the added value they get from each is unclear. Year after year, investments of time and resources are made in each, without seeing what else might be needed for each to be more effective. What would be the value added of greater sharing across these partnerships and alliances? We ended up working at one time or another with the different partnerships, and most of the decisions on those partnerships were made from year to year, and not looking at what would it take for them to have a higher strategic value. What would it take for them to strengthen their impact, and what is the Center’s role in that?” — EPIC Sr. Management EPIC recommended that DDI/EDU develop a Center-wide partnership strategy that articulates how the Center will leverage its relationships with partners to support achievement of USAID’s education strategic goals. DDI/EDU staff reported that they valued EPIC’s professional support in design, facilitation, and logistics for several successful partner events and conferences and the key role EPIC played in supporting communications efforts of the Global Book Alliance as well as providing Secretariat support. Global Book Alliance Support In Year 1, EPIC designed and facilitated a one-day retreat of key USAID Agency staff and members of the Center involved in the Global Book Alliance (GBA). This retreat resulted in a decision to recommit support to the GBA. In Year 2, an EPIC consultant collaborated with GBA to develop a GBA website and collateral material, which resulted in a visually engaging site and print materials to communicate GBA’s work. Collateral material included a GBA brochure and one-pager to complement the GBA website. In Year 3, EPIC provided technical communications, production, and design assistance to the GBA. EPIC hired a part-time communication specialist to provide dedicated GBA support. EPIC supported GBA to create and disseminate 48 products including a video, key messages, presentations, one-pagers, weekly tweets, emails, blogs, social media toolkits, and reports. In Year 4, EPIC facilitated a two-day GBA Secretariat Workshop, which enabled the team to develop a strategic framework for future planning, communications, and reporting on GBA’s work. EPIC developed 56 communication products including a new squarespace website and accompanying content management plan; a new series of illustrations on the book chain; presentations for CIES (3), handouts (2), a Steering Committee memo email; Tweet batches (45); a blog on Textbook Requirements; and a partner briefer on the upcoming MOU with GBA, Global Digital Library, and Google alongside a draft blog and press release for USAID. These materials were shared via email and made available on the GBA Gdrive. From its initial launch in January 2018 to September 2020 when EPIC reported on progress in its Year 4 Annual Report, GBA’s Twitter account had 605 followers, up by more than 100 from the previous reporting period and had garnered an average of 597 engagements per quarter. Engagement was 89 | EPIC EVALUATION REPORT USAID.GOV particularly high in Q3, with 1,047 engagements. Twitter content included webinar reminders and promotional content, partner or steering committee member news, and articles relevant to the GBA’s target audience. On average, the GBA Twitter account was gaining 20,000 engagements per quarter, defined as how many times a GBA tweet appeared on any Twitter user’s timeline. “That work [GBA] is less around USAID’s brand and more about the alliance that USAID hosts. That was tricky for everyone. A lot of us were used to using USAID’s messaging and we needed to switch over to the alliance’s messaging. Really helpful to get that support through EPIC. We’re a barebones group.” — DDI/EDU Technical Specialist “Great to be able to tap into EPIC for additional technical support. We needed a website developed and materials branded. It was nice to be able to tap into somebody who had those skills, which were not skills that we had on our team.” — DDI/EDU Technical Specialist Global Education Conference (GEC) This set of activities has been reported by EPIC as having contributed to both ECP IR 4.2 and 4.4 due to their contributions to strengthening relationships with partners as well as strengthening internal CLA related to how DDI/EDU staff work together across teams and with the field to plan these events. In Year 1, EPIC conducted an AAR for the external and internal days of the 2015 Education Summit. The results of this AAR informed the development of a timeline and key milestones for organizing a large￾scale event in the future. EPIC also researched and presented viable venues for the 2017 State of the Art (SOTA) conference. In Year 2, EPIC provided support for logistics and planning of the 2017 SOTA conference that was anticipated to take place in November 2017. The vision was to transcend a moment in time event to one that supported a continuum of learning and engagement before, during, and after the event and that the agenda would be field-driven. EPIC researched good practices in designing and delivering engaging, high-impact SOTAs and identified six potential venues with associated budget comparisons resulting in the selection of a venue and a vendor to assist with event planning. However, due to the failure of Congress to approve a budget in October and a looming government shutdown, the decision was made to postpone the event indefinitely. In Year 4, EPIC supported DDI/EDU to secure approval for a 2020 Global Education Conference (GEC), select the conference venue, and launch the Call for Proposal process. Additionally, EPIC supported the development of: Committee Toolkits with guidance for committee members and presenters; GEC Governance guidelines that defined roles and responsibilities of all involved and laid out how everyone would work together to support the event; a learning plan; and other associated conference materials. In Year 5, EPIC worked with the KM and PD workstreams as well as DDI/EDU’s GEC Core Team and Committees to coordinate and then cancel the GEC due to the COVID-19 pandemic. EPIC’s ECP team pivoted from its coordination role for the GEC to provide support to promoting the GELS, to ensure a shared understanding, harmonized approach, and effective and efficient implementation of both the USG Education Strategy and the USAID Education Policy across the USG. EPIC handed over the full set of conference planning and support materials to DDI/EDU to use for reference in future conference planning. Interview respondents expressed both appreciation for EPIC’s support for the GEC and indicated some areas in need of improvement. 90 | EPIC EVALUATION REPORT USAID.GOV “EPIC did a great job organizing the Office around the Global Ed conference, which unfortunately wasn’t able to go on, and was flexible to pivot to an online format and help us prioritize and organize to make this happen.” — DDI/EDU Technical Specialist “The preparation for the GEC was fraught with a lot of things that didn’t work. I don’t think that preparing for that was a real high point for EPIC but I’m not sure where the breakdown was. It may not have been with EPIC. The panel that I was working with had challenges with getting registered and other logistics support. I’ve never seen that in any other place with EPIC. Everyone I have worked with on the EPIC team has been very smooth and all the structures were in place.” — DDI/EDU Technical Specialist “The GEC was a disaster not because EPIC did anything wrong. The leadership from our side was trying to be democratic and the approach didn’t work. They set up these parameters to involve everyone. We also know that we need to limit content. Feedback from other events that we need to have time to catch up, get direct TA. But we also recognize that it’s a unique opportunity for our partners to liaise with us. These constraints were not considered well. They asked for proposals and they got hundreds of them. People spent a lot of time reviewing and deciding on too much content. Needed to map out the sessions and fill content based on solicitations. We now have 800 proposals to fit into 50 sessions. Used the policy as criteria for selection. I worked with a junior PA and we have an identical vote on which sessions go where. There were sessions that were shut down by younger staff who don’t have a lot of experience. They had no clue. A junior person says they don’t like the title and scores it low. Just ranked sessions by scores. We worked so hard to get key people from our sector to submit. Being put through this process, their presentations got kicked out. Those presenters were notified that they weren’t coming. We were like, ‘What? You’re kidding’.” — DDI/EDU Technical Specialist “There was some pushback in political terms because it is a political event, too. Maybe I was too much of a purist, but I was hoping for the four days, two days would be open to anyone in the education community... Third day and next day was only USG. The people in the technical teams saw the second part of just USG meetings as a platform to promote their agendas. Fortunately, [DDI leadership] saw this and said this isn’t the way it should be. The main point is when do you engage leadership? If you engage them too late, it is inefficient. We could have saved dozens of person hours if [DDI/EDU leadership] had been engaged earlier.” — EPIC Sr. Management It was unfortunate that both the SOTA conference and GEC had to be cancelled due to circumstances beyond anyone’s control. However, there were many lessons learned about planning for this type of large event that will be useful to take into consideration in the future. These lessons have been documented in the AAR reports from each of these efforts. (See ECP IR 4.4 for more information related to the GEC AAR.) Some key lessons as mentioned above by interview respondents include thinking about the conference as part of a larger learning and engagement strategy that considers what will happen before, during, and after the event in light of conference and broader sector objectives, paying attention to developing efficient ways in which to engage the field in planning and implementation, involving senior leadership early in the planning process, having clear guidelines on conference governance, including a mix of interactive sharing and learning methods beyond PPT presentations and panels, and providing helpful templates, resources, and tools to support conference presenters and facilitators. Support for Partner Meetings EPIC provided meeting design, facilitation, reporting and logistics support for partner meetings over the life of the project. Some of these meetings are mentioned above under the Global Book Alliance. 91 | EPIC EVALUATION REPORT USAID.GOV Additionally, EPIC supported the Mobiles for Education Alliance and Building Evidence for Education (BE2). “Some of the things that come to mind are the partnership meeting and...support for our work for ICT for education and the symposium for the Education Alliance. They did a phenomenal job…. The partner meeting was a good example. They had us do videotaped presentations beforehand. We signed into Google Meet. The video presentations had been taped but the audience thought that it was live. This was to help make sure that the bandwidth was okay and that the sound quality was good. Those kinds of technical expertise are very important for communications. You and I have been on these webinars where there are all these technical issues and it impedes the experience. They’ve been extremely supportive across the board.” — DDI/EDU Sr. Management “We supported the Alliance symposium to facilitate conversations with the steering committee of the Ed Alliance about the goals of the group. Soon DDI/EDU will be transitioning out of the coordinating role. [They] had a lot of great ideas but had trouble making decisions about which ideas were good for the group as a whole. We tried to help them make decisions. We did that work a couple of years after another group helped do it. I don’t think anything was ever done with all that work and strategic conversations from that group. I think it could have been very useful to them.” — EPIC Team Lead IR 4.3: CAPACITY OF DDI/EDU STAFF TO ADVOCATE FOR USAID’S EDUCATION WORK STRENGTHENED. Twenty-nine percent of 24 DDI/EDU interview respondents stated that EPIC had contributed to strengthening DDI/EDU staff capacity to advocate for USAID’s education work; 8 percent believed that this intermediate result was partially achieved; 25 percent believed this IR was not achieved; and 38 percent did not feel they had sufficient knowledge to judge. All education and LPA staff who attended a communications training (46 participants) commented that they were better able to advocate for education sector work because of these courses and the many KM and communications products that EPIC supported as well as being able to refer people to EducationLinks. Additionally, 94 percent of staff who participated in the Year 1 and 2 deliveries of the Education Overview Course (n=48) indicated that they had a moderate or high ability to advocate for the critical importance of the education sector and summon resources to address challenges across sectors and with host governments, local implementers, donors, and other partners. Many of these activities were led by other workstreams in coordination with ECP. EPIC offered communications coaching to 11 DDI/EDU staff; many staff either did not know about this opportunity or did not take advantage of it. As previously mentioned, some respondents thought the Office would benefit from a comprehensive communications strategy that would better unify cross-Office messaging for different stakeholder groups. Stakeholder Consultation The International Commission on Financing Global Education Opportunity requested USAID’s input as part of the World Bank/IMF Annual Spring Meetings. EPIC collaborated with DDI/EDU to administer webinars with USAID Missions and U.S. government (USG) agencies to gather their input on five key questions and developed a summary consultation report. Communications Training, and Coaching In Year 3, EPIC developed and facilitated communications training for 46 staff from DDI/EDU, Missions, and LPA’s Strategic Communicator’s Group. The training familiarized education staff with the new education messaging and provided them with practical skills and knowledge for use in their jobs. Attendees of the session gave the training an overall rating of 5/5 and indicated that both their level of 92 | EPIC EVALUATION REPORT USAID.GOV understanding of USAID key messages and their comfort level in delivering those messages increased as a result of the training. All EEPM session attendees either agreed or strongly agreed that they felt comfortable articulating the core USAID and DDI/EDU messages to diverse stakeholders and that they would be able to apply what they learned in the next three months. They also noted that their level of knowledge or skill increased after the training. In Year 4, EPIC provided on demand communications coaching for 11 individuals to help them communicate effectively about their work under larger Agency messaging. Survey data from three coachees found that each reported increased mastery (average of 15.74 percent) in his or her learning objectives. Education Policy Facilitator’s Guide The Education Policy Facilitator’s Guide (see IR 4.1 above) was disseminated to support education sector staff’s understanding of the policy and increase their ability to communicate effectively about it to diverse audiences. “We had opportunities for communications training. I found that extremely useful for myself on how to communicate, tips and skills. Also the development – not sure who supported this – the development of our talking points for our sector, (Education and COVID-19, etc.). Help us communicate and message with a unified voice internally and externally. That’s been extremely valuable. That has helped us advocate for our education sector.” — DDI/EDU Sr. Management “I think we’ve done well in this. I haven’t noticed a huge improvement, but I think we’ve done well in this. One of those things that I’ve felt the Office has done well for a while. I may not have the inside perspective on that. Part of the thing for this is that we have a new policy. I think we have done fairly well. I like the new policy. It’s not as pithy. Not as easy to communicate it as the old policy. The new policy encompasses a lot more. It makes it harder to do the advocacy. This policy hits a lot of the areas that are important to this administration and is more flexible. Some people in the field like it and others don’t. Has nothing to do with TRG. Internal to USAID.” — DDI/EDU Team Lead “We’ve evolved a couple ways in the Office. There was a push for a standard in the Office of blogs, products. We’ve elevated it. Greater communication with our Front Office, Chief of Staff, the Hill, others. There’s the standard communication tools and products but also an expanded strategic political engagement that has grown and taken on its own momentum and staff.” — DDI/EDU Sr. Management “Some of the content in the modules and courses is around advocacy not just to partners but to ministry, members of Congress, etc. There are more places now where people can get that language certainly than there were five years ago.” — EPIC Team Lead “One thing that we shifted in how we think in light of the new policy and strategy is more asynchronous learning. Not relying on in-person workshops to be the only place where FSNs can get an introduction to the policy. The goal in 2018 was to do a series to introduce people to the policies. That didn’t happen due to government shutdown. The spirit of that remained, though and we took that online. We developed online modules structured around the priorities expressed in the policy. They are available to anyone who wants to see them any time. Short, hour-long segments to orient them. Change in how we work together: that was the impetus for more online, self-paced learning.” — EPIC Team Lead “Definitely, there has been support in achieving our larger education strategy goal. I know that one thing that I can picture Tom Lent saying this…no matter how based on research your strategy goals may be in improving learning outcomes for example, if these goals are not expressed in the right way, they will never 93 | EPIC EVALUATION REPORT USAID.GOV make a difference. You can’t shove things down people’s throats. Need to help them see that this is the best way to achieve the strategy goals. Communication, how to communicate to the field. Leadership, how do team leads advocate for their particular area within the sector. All of that improvement in communication and improvement in being supportive, particularly when it comes to supporting Education Officers in the field has made an improvement in getting our messages out to the field. And just the Office climate. When we talked about changes that I’ve seen, I didn’t go back far enough.” — DDI/EDU Sr. Management “We are technical experts, but we really needed EPIC for the training side. Rather than delegating USAID work to EPIC, use EPIC to help with the branding, presentation, higher level comms work that we can take and implement day-to-day.” — DDI/EDU Technical Specialist IR 4.4: CLA APPROACH STRENGTHENED, DOCUMENTED, AND SHARED. Most interview respondents (84 percent) did not feel they had enough knowledge about outcomes and activities related to strengthening CLA under the ECP workstream to have an opinion about the extent to which this IR had been accomplished or did not address this question. Eight percent stated that good progress had been made, one person (4 percent) said that progress was mixed, and one person (4 percent) did not see progress related to this IR. Interview respondents mentioned a few areas where they observed CLA practices used in this workstream, including using AARs effectively in planning for the GEC, EPIC collaborating effectively with other USAID mechanisms in the development of communications products, and increased cross-workstream collaboration internally with other DDI/EDU teams to develop and disseminate communications and knowledge products. “For the partnerships supported by ECP, we have expanded our collaboration with external partners pretty substantially over the last two years. Working with DEEP, another new mechanism coming on, we’ve worked with Youth Power to do a knowledge capture trip to Uganda and Kenya on youth orgs and encouraging youth led entrepreneurship, others. More and more, it’s pretty common for EPIC to work with other groups. It’s good. It shows desiloing in the Office and in its mechanisms.” — EPIC Team Lead “When we’re working on these products, we are working with different teams. In terms of teams coming together on documents, that has increased quite a bit. Part of that is people saying that it had to, organizational change, and people knowing that it was expected, more matrix design. People purposefully putting together teams for specific projects and seeing the value of that across the Office. It took a few examples of that working well for people to see the value.” — DDI/EDU Technical Specialist Global Education Conference After-Action Reviews Year 1, EPIC conducted an AAR of the 2015 Education Summit and used results to inform planning and research for the 2017 State of the Art (SOTA) conference, which was ultimately cancelled due to the government shutdown. In Year 4, the process of designing and coordinating the GEC required close collaboration with all EPIC workstreams. EPIC made concerted efforts throughout the event design process to incorporate and learn from the lessons and best practices of the previous conference and other similar conferences. The GEC and its pre, during, and post activities sought to strengthen a sustainable system and momentum for learning and application across a broad but select range of stakeholders including Missions, implementing partners, key partner country stakeholders, other donors and partnerships, the education sector, and the U.S. Government. The process of preparation that went into designing and coordinating the GEC required close collaboration with all EPIC workstreams to support the GEC Core 94 | EPIC EVALUATION REPORT USAID.GOV Committee and Session Design and Development Committee. When the GEC was cancelled due to concerns related to the COVID-19 pandemic, an AAR session and survey were conducted to collect lessons and best practices along with data on the utility and quality of EPIC’s role in event planning. Overall, the SDD team provided positive ratings on a five-point scale. On average, respondents found the resources very sufficient (4.8), workload reasonable, team members were highly committed, and communication amongst members was open (4.0). They were more neutral in response to the defined roles and responsibilities (3.2) and the clarity of the decision-making process (3.0). The Core Team participants gave high scores to commitment (4.4) and communication amongst members (4.0). “In 2019, we started again, and planning was quite far down the road, and then COVID hit and that was cancelled. We have changed to a virtual format. Because it was such a complex undertaking, there was a lot more reflection, conversations about where roadblocks were happening. Given the complexities of that, CLA was more baked into that and due to Tom, Lisa, and Daryn understanding that that was a good management approach for that type of activity. The ECP manager was asked to lead that activity, but EPIC was really supporting and other workstreams were involved. A lot of stuff was documented about that. Report on how we did it, the steps. If someone were to pick this up in the future, what considerations would they need to address. That was super important after having gone through both of those planning exercises. Now we won’t lose the learning from the monumental planning processes.” — DDI/EDU Technical Specialist “I’m convinced that if that conference had taken place, it would have been epic. It was innovative and collaborative. We had received feedback from people to say they were impressed.” — DDI/EDU Technical Specialist RECOMMENDATIONS AND LESSONS LEARNED CLARIFY THE PURPOSE AND STRATEGY OF ECP AND ITS RELATIONSHIP TO KM AND PD. While there have been some advances made, especially on ECP IR 4.1, and many high-quality communications products produced under this workstream with EPIC’s support, interview respondents point to the need for a more comprehensive Center-wide communications strategy to guide a more proactive approach to communications, engagement, and partnerships and to clarify the roles of the Engagement, KM, and PD Teams as well as the role EPIC should play in supporting the strategy. “I don’t think IR 2, 3, 4 we have put attention on. If I were to make a recommendation, I don’t think I’d call this ECP. I’d include understanding of E3’s [DDI/EDU] brand as part of the KM bucket. The person who does KM could exhibit it. This has been quite weak. I don’t see how EPIC has worked on partnerships. That to me is part of the PD goal. CLA approach for engagement? I don’t know. Maybe less work to be done but done differently.” — DDI/EDU Team Lead DEVELOP A STRATEGIC, OFFICE-WIDE APPROACH TO PARTNERSHIPS. Most respondents who commented on partnerships noted that DDI/EDU has not yet defined an Office￾wide strategic approach to partnerships that would help it to better leverage its relationships with its partners to accomplish mutually beneficial goals and objectives. Strengthening partnerships that achieve USAID’s education strategic goals is an area that would benefit from greater attention. Partnerships have been more siloed and person-focused to date because of the earlier structure, culture, and strategy. 95 | EPIC EVALUATION REPORT USAID.GOV DEFINE A STRATEGIC AND EFFECTIVE PROCESS FOR SOLICITING CONFERENCE SESSION IDEAS FROM THE FIELD. In 2019, DDI/EDU took seriously the desire to engage the field in planning and implementing the Global Education Conference. With EPIC’s support, the Center reached out to the field and invited session proposals that would be aligned with the new Education Policy, and the field responded with an abundance of submissions, which indicated a high level of interest in engaging. This was very encouraging but overwhelmed some DDI/EDU staff, and some questioned whether the selection process was effective and fair. For future conferences, it will be important to consider the lessons learned from the GEC AAR to better streamline the call for proposals and selection processes. BE AWARE THAT THE CONFERENCE PLANNING PROCESS IS POLITICAL AS WELL AS TECHNICAL. ENGAGE LEADERSHIP EARLY. As mentioned under ECP IR 4.2’s discussion of the GEC above, it can save everyone a lot of time and effort if leadership is involved early in the planning of big events to ensure that everyone is aware of guidance from the top regarding priorities and parameters of the event. AS PROCESSES FOR DEVELOPING COMMUNICATIONS PRODUCTS EVOLVE, BE SURE TO UPDATE THEM AND SOCIALIZE ANY CHANGES. DDI/EDU staff appreciate the opportunity to contribute to newsletters and other communications products. Interview respondents noted that an initial process that was put in place was working well. However, along the way, some changes were made that may not have been communicated clearly to staff (or they did not know where to access guidance on when and how to contribute). It would be helpful to designate a central information hub that provides this type of process guidance that can be updated on an ongoing basis as processes evolve and make staff aware of how to access this hub. PROJECT WIDE RECOMMENDATIONS AND LESSONS LEARNED In this section, we begin with a discussion of EPIC’s Theory of Change followed by recommendations and lessons learned across the four workstreams. EPIC’S WORKSTREAM CONFIGURATION AND THEORY OF CHANGE All but two of the 24 DDI/EDU (92 percent) and 9 EPIC (100 percent) respondents stated that they would not change the four workstream configuration as articulated in EPIC’s theory of change. They saw the synergistic and interconnected value of these four workstreams working together. And eight DDI/EDU respondents (33 percent) commented that it is a large scope for any one contract. One respondent proposed dropping KM and making that a separate mechanism to make the scope a bit smaller and/or because they conceived of the workstream in a different way. One person considered and another proposed making PD a separate contract mechanism that provides a cadre of technical SMEs to design and deliver training. Others disagreed with this vision, believing that DDI/EDU staff were hired to be the SMEs who provide technical guidance and content, and what they needed was support to help them be better trainers and communicators of this content for various audiences. One DDI/EDU respondent suggested that diversity, equity, and inclusion be added to the OE workstream in the future. Five respondents believed that the overall goal articulated in EPIC’s theory of change was too high level and that the objectives should be more within the manageable control of the contract and reflect specific Office priorities. Two DDI/EDU respondents thought it would be helpful to create a more detailed theory of change that spelled out more specifically the assumptions and sequencing of desired results. 96 | EPIC EVALUATION REPORT USAID.GOV All EPIC team respondents supported the package of the four workstreams. Two EPIC staff proposed that KM and PD be combined or integrated and another proposed that the TOC reflect in some way the greater integration among KM, PD, and ECP that has evolved through EPIC. One proposed that OE be presented as a catalyst or enabler of the other three workstreams. EPIC staff also observed that the scope was large but that the pieces were synergistic and provided an effective package. One EPIC team member echoed the DDI/EDU staff suggestion to make the large scope more feasible by bringing the higher-level goal down to more intermediate goals that would be within the potential manageable influence of the project. “There’s a lot being asked of a single contract. But one thing on the positive side is that it is really a complete picture. All these things are intricately connected. What helped to make EPIC successful was that one group could look at the whole realm of how we support the education strategy to deliver quality education in the field. Because it’s a complete picture, it’s enabled us to be more successful. I don’t think I’d make any changes. This is what helped it work and be successful.” — DDI/EDU Sr. Management “Theories of Change are not my thing. They are hard to configure. I think at a basic level it made sense. Wonder if we need to scale it down. It ended up being driven Office-wide to do a theory of change. I don’t know how useful it was at this high, high level to reach education strategy goals. I don’t know if we look back at it to refer to it. Could be a bit lower level and a bit more practical. At least the way it was written. Could consider something more manageable. I think I would start with the goal level. I think if you had the right people and systems in place, as long as you had the time available, it could have been quite possible. There were some external reasons that made it hard to take on some things. I would just scale the whole theory of change back, starting with the goal. It’s a bit lofty.” — DDI/EDU Technical Specialist “I would start with the goal. I don’t think that a contract can transform an Office. It’s unrealistic to expect that. We don’t know if the Office wants to be transformed, or a new leader can come in and do that. Have it be a more strategic goal. If it only hinges on our strategy, it’s weak if the strategy is weak. I’d start with our functions and advance things according to our priorities. Whatever we have agreed to. I don’t think we ever agreed to all these things. It makes sense. But have the Office commit to the change. Transformation doesn’t belong in the goal unless E3 decides it will be transformational and then EPIC would guide us. When I see things like ‘across all stakeholders,’ is that realistic? Try to be more specific about primary, secondary stakeholders. That allows us to be more surgical. Like for the KM part. The PD part of that theory you could drive a truck through. We should be able to be more specific about the skills that are required. Maybe this exists and it isn’t here, but it would be interesting to see this more mapped out with the assumptions for this to be successful. Then you can assess your assumptions to make sure they are true. If not, you can adjust the project.” — DDI/EDU Team Lead “I’m clear on PD, KM, and ECP. I’m drawn to OE either as an enabler of the other three or as another option. Some sort of collaboration circle around these. Add that to this. Those are the only tweaks I’d focus in on. Show OE as more of a catalyst or note some collaborative requirement to achieve their goals because if they don’t do that, they won’t achieve their goals in a way that is meaningful.” — EPIC Sr. Management RECOMMENDATIONS AND LESSONS LEARNED ACROSS THE FOUR WORKSTREAMS HAVE THE RIGHT PEOPLE IN THE RIGHT JOBS AT THE RIGHT TIMES AND TAKE ADVANTAGE OF THEIR EXPERTISE. It took time for EPIC to gain a clear understanding of DDI/EDU needs and to staff the team to best meet those needs. Similarly, on the USAID side, it was clear that the COR role is key to a mechanism’s success and that Activity Managers can have a big impact on the overall vision and progress of any given workstream. When the right people are in the right jobs at the right times and people are clear about 97 | EPIC EVALUATION REPORT USAID.GOV what they can offer each other and how they can work together, the work is much more productive and effective. The flexibility of the EPIC mechanism allowed the team to be quickly responsive in bringing in “surge capacity” resources as needed. “The first success factor is the staffing approach. Because we didn’t start off with a heavy org chart with a hierarchical structure, that gave us the flexibility to staff according to the needs that we perceived. It was also cost-effective, because we weren’t spending a lot of money, especially in the first year as we were figuring out what we needed. Fortunately, we had some really high performers who have been able to be thought partners with their counterparts. We wouldn’t have been able to be as successful if we didn’t have them.” — EPIC Sr. Management To avoid confusion and support efficiency, it is important to clarify roles and manage expectations about what EPIC staff are expected and not expected to do. For example, some DDI/EDU staff expected that EPIC staff would supplement their education sector technical expertise, while others believed that DDI/EDU staff were hired for their technical expertise and that EPIC’s role was to complement their skill sets with OD, adult learning, communications, technology, and other core skills and expertise that support organizational effectiveness, communications, and learning. LEADERSHIP BUY-IN IS ESSENTIAL. Leadership transitions at USAID are common. These transitions are challenging for the Office, both for the leaders and for the staff. The DDI/EDU Office experienced several leadership transitions over the life of the EPIC project at the Office Director level as well as the Deputy and Team Lead levels. The EPIC team worked to support these transitions by offering coaching, being a thought partner, sharing institutional knowledge about the Office, its culture and team dynamics, and offering support in the design and facilitation of Office gatherings and meetings. To take full advantage of a mechanism like EPIC, leaders need to be committed to change, understand the mechanism’s potential value, and strategically take advantage of the full range of skills, experience, and services that it offers. Additionally, when Office objectives and initiatives are agreed to, leadership needs to hold themselves and their teams accountable for progress and results. START WITH A FULL FUNCTIONAL ANALYSIS AND DO PERIODIC ASSESSMENTS OF PROGRESS ALONG THE WAY. As EPIC did, conduct an in-depth analysis of how the Office or Unit is structured and how it is functioning from diverse perspectives by involving everyone and building trust along the way. Do not assume anything about staff; start working with them from where they are as real people. WORK SYNERGISTICALLY ACROSS WORKSTREAMS AND TEAMS. Bundling the workstreams together was more powerful than not. Having champions that could bring together the various workstreams and technical teams for a larger purpose was helpful. The most effective champions had a clear and strategic vision of where they wanted to go, got the right people involved, understood the relative strengths of the various contributors, established clear processes and expectations, and held people accountable for progress. “For EPIC, the challenge was that there were a lot of things in it. In some ways, it’s almost five different things. Those could have worked together really well if they all had champions that worked together and provided a significant amount of resources for efforts to really manage this well. Bundling these things together was more powerful than not. Training, KM, OE, comms all work together to support the knowledge base, come together in a video, a training course, etc. That is a good practice. Good results combining them. If there’s going to be breaking into workstreams, have a champion who is well defined in their role, not just 98 | EPIC EVALUATION REPORT USAID.GOV in managing the partner but also being a conduit to USAID. Be a strategic partner for USAID to bring USAID’s ideas to the partner and vice versa. That’s when this worked well. Not as much when that didn’t happen. Bundling – make sure that they fit together.” — DDI/EDU Sr. Management COLLABORATING, LEARNING, AND ADAPTING: CREATE OPEN SPACES FOR REFLECTION, LEARNING, AND CO-CREATION. The semi-annual off-site meetings and other Office-wide events such as town halls helped the Office to step out of the day-to-day operations to reflect on what was happening, how they were doing, and what they were learning, get to know each other better, build trusting relationships, speak frankly about problems, and work together to create solutions and a collective picture of how to move forward. Having EPIC design and facilitate these sessions provided value that an external perspective and honest broker role brings. At the same time, the ongoing relationship that EPIC had with the Office created a level of trust and understanding that was foundational to a more open and safe space for productive dialogue. Periodic monthly all-hands and bi-annual off-site meetings provided DDI/EDU the opportunity and space to have the strategic, cross-team technical and organizational conversations that help move the Office forward toward effectiveness. WORK IN STRATEGIC PARTNERSHIP. All DDI/EDU and EPIC team respondents appreciated the collaborative, trusting, and respectful relationship they have established over the life of the project. Many DDI/EDU staff see EPIC team members as an extension of their own team. The embedding of EPIC staff into the day-to-day operations of the Office and the fact that EPIC team members were reliable, flexible, and professional, and performed well all helped to foster these relationships. EPIC team members devoted a lot of time and effort to get to know people in the Office, to understand their perspectives, their objectives, their strengths, and weaknesses, as well as interpersonal dynamics. They tried to meet Office staff where they were and were highly responsive. Over time, many EPIC staff became thought partners to DDI/EDU leadership at multiple levels. They offered new ideas, supported Office initiatives, and were patient when other priorities delayed USAID staff’s ability to engage in activities they needed to do to deliver their part on time. Many respondents commented at some point in their interview, “we couldn’t have done it without EPIC” and that they enjoyed working with the EPIC team, and that they were “really nice people.” When things didn’t go well or when a personnel change was needed, respondents commented that EPIC listened to the feedback and responded quickly to successfully resolve the issue. “Having a strategic partner on the USAID side was key. When the OE stuff worked best was when we had a strategic partner to work with EPIC. When the Director was bought in as a strategic partner and champion. That person would get buy-in. Translate the ideas generated by the EPIC team into ‘AID-speak.’ That applies to all workstreams, having a strategic partner to work with EPIC.” — DDI/EDU Sr. Management “It feels easy to get their support. For example, a webinar that’s a month away. Talk to Heather and it’s very quickly and competently getting it together. It may be how the contract was set up. It’s easy to help socialize the use of a particular tool. No formality to it. Which has been nice. There is a document, but it was flexible and we can change it if we need less support. Flexible and responsive. When you’re planning a webinar, a technical person isn’t thinking what program I’m going to use. To have someone who knows the program, knows the listserv, can set up the email to advertise it has allowed me to focus on the technical content. I’m not sure if that’s a change. The first time I did a webinar in February 2019, another program assistant liaised with EPIC to set it up.” — DDI/EDU Technical Specialist 99 | EPIC EVALUATION REPORT USAID.GOV ENSURE REQUIRED LEADERSHIP SIGN-OFF AND BUY-IN AS EARLY AS POSSIBLE. Due to the nature of its work, USAID operates through multiple levels of bureaucracy, clearance, and decision-making hierarchies that can present obstacles in the creation of PD products and affect intra￾and cross-workstream timelines. Mechanisms such as EPIC will be most successful when these timelines and requirements are anticipated, and all required leadership is engaged in decision-making from an early point. “Knowing how and when to engage key people at certain points in the design of something. ‘Engage’ could mean bring them in in the initial ideation part, be involved in the decision-making on what it is going to look like. Knowing how roles, personalities, and politics play into getting an initiative to completion. There have been a number of occasions where people who should have been involved weren’t, and then three months’ worth of work gets thrown out. Or someone was brought in later and they want to throw out everything. People want to feel respected. They may be a technical lead. They don’t like it if they are just learning about something after others have worked on it.” — EPIC Team Lead “In the implementation with the technical, putting everything together, there have been a lot of issues that came up because not all action steps were thought through about how to do these large-scale projects. A lot of that has to do with working with USAID, the bureaucracy and the limitations we have as an Agency.” — DDI/EDU Operations Specialist LEVERAGE THE ORGANIZATIONAL EFFECTIVENESS WORKSTREAM FOR THE BENEFIT OF ALL WORKSTREAMS. While some DDI/EDU staff did not initially understand the purpose or scope of the OE workstream, and some resisted some of the initial leadership development and teambuilding workshops, many came to view the OE workstream as foundational to the work that the other workstreams were able to do individually and together. Nearly everyone commented on the improvement in the organizational culture of the Office over the life of the EPIC project and acknowledged the contributions of the OE workstream to this positive change that facilitated the work of the Center. Additionally, EPIC collaborated with DDI/EDU to strengthen leadership capacity and create and improve spaces, systems, and processes for onboarding, strategic planning, priority setting, work planning, cross-team collaboration, pausing, reflecting, and learning, problem solving, and other purposes that benefitted all workstreams. “If OE isn’t correct, the other workstreams are not going to be as good as they can be. It’s the OE that really helps them be effective.” — DDI/EDU Technical Specialist “I’ve often viewed OE as the catalyst for the other workstreams. Without OE, the other workstreams could have been hindered.” — EPIC Sr. Management “The thing that’s been to me crucial is having the organizational effectiveness piece be included in the same contract as all the other stuff. Sometimes people come in to do great KM, but unless your Office is able to tackle the people issues, which are usually the barriers… Just because we have the coaching and team meetings, I feel like the staff at TRG know the personalities and how they can contribute to success or be a hindrance. They help us think about how to strategize to do all the things we’re trying to do under this.” — DDI/EDU Sr. Management ENSURE FLEXIBILITY IN THE CONTRACT MECHANISM AND IN OPERATIONS MANAGEMENT TO FACILITATE RESPONSIVENESS, ADAPTIVE MANAGEMENT, AND QUICK PIVOTING WHEN NECESSARY. The flexibility and adaptability of the EPIC contract mechanism and its management enabled the project team to manage the contract to accommodate emerging priorities or rapidly shifting contextual factors 100 | EPIC EVALUATION REPORT USAID.GOV such as the government shutdown and COVID-19. This was due in large part to the close collaboration across workstreams and maintaining a large pool of resources (internal to TRG and its partners as well as external) that could be accessed for surge capacity. Contract finances were managed so that funding could be moved between activities, as needed. “The way we set up our management structure, it was really flexible from the beginning. Allowed us to have fungibility to respond quickly. When COVID happened, our COR asked us if we could help and we were able to have a very quick turnaround and move money to COVID activities quickly. Having a flexible budgeting structure has helped enable our success.” — EPIC Sr. Management 101 | EPIC EVALUATION REPORT USAID.GOV CONCLUSION EPIC serves as a compelling and useful example for USAID of the value of a highly flexible and comprehensive mechanism to strengthen a Center’s organizational effectiveness, professional development, knowledge management, engagement, communications, and partnerships in support of Agency sector and development goals. This mechanism required a high level of collaboration between the EPIC team and DDI/EDU leadership and staff. The EPIC team worked hard to develop trusting relationships with everyone in the Center by listening to them; acquiring a deep understanding of DDI/EDU’s people, their needs, goals, priorities, and aspirations as well as the very dynamic and often challenging context in which they worked; and consistently delivering high quality, professional services. When things did not go as well as intended, EPIC team members were open to constructive feedback and worked hard to improve. When presented with major challenges such as COVID-19, EPIC was able to pivot quickly to adapt its support to meet the Center’s changing needs. The scope of the EPIC project was very broad and ambitious. We cannot say that EPIC helped the Center fully achieve all the strategic objectives and intermediate results articulated in EPIC’s theory of change and strategic results framework. This was probably unrealistic given the project’s broad scope and the many challenges the Center and EPIC team faced over the life of the project. However, it is clear that EPIC has been successful in contributing to significant and noteworthy advances in DDI/EDU’s effectiveness as an emerging Center for Education. While it has not always been easy and there were setbacks, the Center has demonstrated resilience in the face of all these challenges. With EPIC’s support, the Center has improved its organizational culture, strengthened its leadership, systems, and processes, led an inclusive process to develop and launch the USG Education Strategy and USAID Education Policy, increased collaboration across teams and with Regional Bureaus and Missions, launched an online Global Education Learning Series and established EducationLinks as a “go to” hub for education information to strengthen the capacity of field practitioners, supported the development of several global partners, and produced high quality communications and knowledge products that are helping USAID tell its education story. 102 | EPIC EVALUATION REPORT USAID.GOV ANNEXES ANNEX 1: EVALUATION WORK PLAN ACTIVITY DELIVERABLE PERSON(S) RESPONSIBLE MAY JUNE JULY AUG SEPT OCT NOV DEC Design and facilitate evaluation kick-off meeting. Pre-evaluation briefing Sr. OD/MEL Advisor Consolidate existing EPIC program monitoring and evaluation data that has been gathered over the life of the program. M&E Specialist Summarize consolidated data according to the strategic results framework by workstream and cross-cutting themes (e.g., CLA). M&E Specialist Construct a timeline of key events and activities supported by the EPIC program. M&E Specialist Develop interview protocol, focus group discussion guide, and analysis framework. Sr. OD/MEL Advisor Conduct interviews of around one hour each with approximately 40 key informants, ensuring that the evaluation is conducted in a confidential manner that protects participants. Sr. OD/MEL Advisor Conduct x number of focus group discussions with six to eight participants in each group. Sr. OD/MEL Advisor 103 | EPIC EVALUATION REPORT USAID.GOV ACTIVITY DELIVERABLE PERSON(S) RESPONSIBLE MAY JUNE JULY AUG SEPT OCT NOV DEC Consolidate, analyze, and synthesize data from interviews, identifying key results as well as intended and unintended outcomes and changes, themes, trends, and other key findings. Sr. OD/MEL Advisor Present evaluation findings in a draft evaluation report. Draft evaluation report Sr. OD/MEL Advisor & M&E Specialist Present summarized data in user￾friendly graphics and formats that highlight salient findings and key insights. M&E Specialist Triangulate M&E data with interview and focus group discussion data and identify trends, key findings, and salient insights. M&E Specialist & Sr. OD/MEL Advisor Draft the evaluation report. Sr. OD/MEL Advisor & M&E Specialist Prepare a presentation of the key findings, conclusions, lessons learned, and recommendations. Preliminary findings briefing Sr. OD/MEL Advisor & M&E Specialist Develop the evaluation briefing and feedback meeting design and materials for USAID/E3 ED and the EPIC program team. Sr. OD/MEL Advisor & M&E Specialist Co-facilitate the evaluation briefing and feedback meeting with USAID and the EPIC team. Sr. OD/MEL Advisor & M&E Specialist 104 | EPIC EVALUATION REPORT USAID.GOV ACTIVITY DELIVERABLE PERSON(S) RESPONSIBLE MAY JUNE JULY AUG SEPT OCT NOV DEC Finalize the evaluation report, incorporating any feedback from the feedback meeting. Final evaluation report and final PPT presentation of findings and recommendations Sr. OD/MEL Advisor & M&E Specialist 105 | EPIC EVALUATION REPORT USAID.GOV ANNEX 2: DATA COLLECTION MATRIX It is important to note that measures, sources, and methods of verification are illustrative. The evaluation team will adjust the overall methodology as needed during data collection based on what is feasible given the sensitive programming environment. SUB-QUESTIONS INDICATORS AND ILLUSTRATIVE MEASURES SOURCES METHODS OF VERIFICATION What have been the program’s achievements as compared with expected results and objectives of the Strategic Results Framework? To what extent has EPIC contributed to DDI/EDU’s effectiveness and work quality, particularly as related to its four workstreams of organizational effectiveness, knowledge management, professional development, and communications/engagement/ partnerships? Evidence of changes related to SOs and IRs for each workstream and across workstreams that are attributed to EPIC’s support of the program (positive and negative). USAD/E3 ED staff USAID Mission staff USAID partners EPIC team members Program documents & reports Consolidated M&E data Web analytics Key informant interviews Focus group discussions Online survey, if needed What unintended positive or negative outcomes, if any, emerged from EPIC’s support to DDI/EDU and other USAID Operating Units (OUs)? Number/type of unintended outcomes (positive and negative) perceived to be the result of EPIC’s support of the program with evidence to corroborate these outcomes. USAD/E3 ED staff USAID Mission staff USAID partners EPIC team members Program documents & reports Key informant interviews Focus group discussions Online survey, if needed What are best practices and lessons learned from the EPIC program? How did the program affect the intended beneficiaries? What factors contributed to or hindered its success? Number/type of perceived benefits of the program from participants’ perspectives Number/type of perceived costs of participating in the program from participants’ perspectives USAD/E3 ED staff USAID Mission staff USAID partners EPIC team members Program documents & reports Key informant interviews Focus group discussions Online survey, if needed 106 | EPIC EVALUATION REPORT USAID.GOV SUB-QUESTIONS INDICATORS AND ILLUSTRATIVE MEASURES SOURCES METHODS OF VERIFICATION Number/type of negative outcomes of the program from participants’ perspectives Number/type of contributing factors Number/type of hindering factors Which tools and strategies were more effective in achieving the intended program objectives and what were the contributing factors? Evidence of effectiveness and costs of different strategies and tools used to achieve similar program objectives Number/type of contributing factors to effective use of tools and strategies to achieve program objectives USAD/E3 ED staff USAID Mission staff USAID partners EPIC team members Program documents Program M&E data Interviews Focus group discussions Online survey, if needed Web analytics Did workstream collaboration lead to better or extended benefits? In what ways? Number/type of collaborations Number/type of benefits of workstream collaboration Documentation/evidence of how collaboration contributed to better or extended benefits USAD/E3 ED staff EPIC team members USAID Mission staff Program documents Program M&E data Interviews Focus group discussions How has USAID’s collaboration with the EPIC program contributed to changes that have occurred over the life of the program? What did the EPIC team do in its work with USAID to contribute to bringing about these changes? Documents that present evidence of changes over the life of the program Documents that present evidence of ways in which USAID/E3 ED collaborated with the EPIC team Reported outcomes of EPIC team’s work Documented statements that present specific examples of changes USAID/E3 ED staff EPIC team members Interviews Focus group discussions Collect documented evidence of statements, examples Program M&E data 107 | EPIC EVALUATION REPORT USAID.GOV SUB-QUESTIONS INDICATORS AND ILLUSTRATIVE MEASURES SOURCES METHODS OF VERIFICATION they perceive to have been brought about through collaboration What external factors have made a positive contribution/have had a negative influence on the design, implementation, and outcomes of the program? What are the external challenges and opportunities USAID/E3 ED faced while participating in the program? Documentation of external events that had some effect on implementation of the EPIC program (negative and positive) USAID/E3 ED staff EPIC team members USAID Mission staff USAID partners/IPs Program documents Interviews Focus group discussions What are current barriers and enablers to success? Definition of “success” from participants’ perspectives Number/type of enablers to success Number/type of barriers to success USAID/E3 ED staff EPIC team members USAID Mission staff USAID partners/IPs Program documents Interviews Focus group discussions 108 | EPIC EVALUATION REPORT USAID.GOV ANNEX 3: DATA COLLECTION INSTRUMENTS • Interview protocol • Training Follow-up Survey • Insert other instruments here…. 109 | EPIC EVALUATION REPORT USAID.GOV ANNEX 4: TIMELINE OF KEY EVENTS BY WORKSTREAM TABLE 1. TIMELINE OF ORGANIZATIONAL EFFECTIVENESS KEY EVENTS FISCAL YEAR ORGANIZATIONAL EFFECTIVENESS KEY EVENTS 2016 Two-day DDI/EDU off-site (July) Education Sector Survey (225 participants from 80 Missions, Bureaus, Offices) Week-long Education Sector Strategy feedback and drafting session (May) Streamlined Office processes for onboarding, travel, and clearances Facilitated meeting on two sections of ADS Developed standardized template and process for teamwork plans Facilitated work planning sessions with eight teams Facilitated meetings with QUEST to redefine their roles Consulted with Regional Bureaus and E3/ED together on how to improve DDI/EDU / Regional Bureau relationship and education sector effectiveness 2017 Two-day Office retreat addressing team priorities, transitions, organizational learning, partnerships, and communications (June) Implemented executive coaching program with bimonthly coaching sessions Launched Tips and Tools Playbook series. Three playbooks produced: Effective Meetings, Email Etiquette, and Productivity in Any Workspace One-day Education Sector Council retreat including representatives from all ED teams and Regional Bureaus to identify ways to work more cohesively as a sector to better support Missions 2018 November 2017 – March 2018: USAID Hiring Freeze Participatory construction of Office Strategic Priorities for 2018-2019 and incorporation of these priorities into team- and Office-level planning Two-day off-site. Regional Bureaus and Global Development Lab colleagues participated Designed and administered two organizational wellness surveys and developed two organizational wellness dashboards. Dashboards socialized in EMT retreats, periodic staff meetings, town halls, and the annual off-site retreat Facilitated monthly half-day or full-day coaching sessions/retreats for the EMT on important and urgent Office issues and key decision points Supported inclusive development of Education Strategy by providing knowledge capture, facilitation, logistics and coordination, and syntheses of stakeholder feedback from 35+ intra￾agency, inter-agency, and external consultations, webinars with more than 25 Missions, public comment period responses, and written input and feedback from USG Technical Working Group and USG Advisory Group meetings to inform development of USG Strategy on International Basic Education Organized periodic reflection sessions, town hall meetings, and initiated monthly all-hands meetings 110 | EPIC EVALUATION REPORT USAID.GOV FISCAL YEAR ORGANIZATIONAL EFFECTIVENESS KEY EVENTS Collaborated with PD, KM, and ECP workstreams on Education Strategy and USAID Education Policy (Priority #1) and Mission Support (Priority #3) to ensure products were shared and the field was kept abreast of the Education Strategy and Policy 2019 December 2018 – Jan 2019: Government Shutdown Two-day annual off-site to address new Office-level functions, priority initiatives, milestones, USAID Education Policy, and DDI progress to date (May) Implemented third iteration of customized organizational wellness assessment, and distributed synthesis report based on findings Rebranded and revamped the former Leadership Development Program into the Professional Enhancement Series (PES) to address participation issues and respond to Office needs and operating context. Twenty-three participants across five PES sessions on Emotional Intelligence, Conversational Intelligence, Effective Feedback, Managing Change, and Managing Up. Participants offered follow-up coaching and Tips & Tools Playbooks Provided synthesis of USAID Education Policy feedback (October) Facilitated two-day retreat for the Basic Education Division Team leadership to support development of a draft Basic Education Division Vision. Provided follow-up facilitation support during Basic Education Division Quarterly Meeting Support work planning retreats for four teams Facilitated eight Center for Education Working Group meetings resulting in consensus on mission, vision, mandate, comparative advantages, outcomes, and theory of change Provided support to the Mission Support Working Group by researching and drafting a process with stakeholders for systematically receiving and responding to Mission support requests 2020 March 2020 – COVID-19 forces all work and events to be moved online indefinitely Organizational wellness survey (October) Annual off-site retreat (January) EMT and Team Leads off-site meeting (January) to look at civility and respect, exchange concerns, and problem solve EMT strategy meetings (January, May, June, July, August, September, October, November, December) HEYWFD Retreat (February) All-hands meetings (February, March, July, August, September, October) Pilot program of Education Technical Assistance (EDTA) to Missions began in Senegal and Ethiopia and was expanded to DRC, Mali, and Rwanda Pulse Check Survey (May) COVID Task Force Work Planning Meeting (June) EiCC Division Retreat (July) Work Planning Working Groups (September) Senior Management Group Strategy Session (December) 111 | EPIC EVALUATION REPORT USAID.GOV TABLE 2. TIMELINE OF PROFESSIONAL DEVELOPMENT KEY EVENTS FISCAL YEAR PROFESSIONAL DEVELOPMENT KEY EVENTS 2016 First webinar delivered (July) Pilot EOC course delivered in DC (September) 2017 First EEPM course delivered in DC (October) EOC course delivered in DC (May) EEPM course delivered in DC (June) Two webinars delivered (August and September) Meta-analysis report of Ed sector PD needs Development of PD workplan that supports COL Reshaped Professional Development Working Group (PDWG) EEPM course delivered in Thailand (November) First Ed-CVE course delivered in Thailand (December) 2018 November 2017 – March 2018: USAID Hiring Freeze Daryn Cambridge becomes PD portfolio manager (May) Eight webinars delivered Gender distance learning course, accompanied by How-To Note and facilitator’s guide, designed, and created but ultimately taken down due to political sensitivities Ed-CVE course delivered in Germany (April) EEPM courses delivered in DC (June) and Ghana (August) 2019 December 2018 – Jan 2019: Government Shutdown Ed-CVE course delivered in DC (May) EEPM course delivered (June) Seven on-demand videos produced Two distance learning courses on PYD designed Eleven webinars delivered PDWG reconvened (Summer) Education Sector learning priorities survey (August) Planning for GEC begins with weekly Session Design and Development committee meetings 2020 March 2020 – COVID-19 forces all work and events to be moved online indefinitely GEC is converted into online GELS (March) PDWG reconvened (April) Delivered seven COVID-19 webinars (Spring) 112 | EPIC EVALUATION REPORT USAID.GOV FISCAL YEAR PROFESSIONAL DEVELOPMENT KEY EVENTS EOC virtual modules are released (June) Virtual GELS conference (June - September) 2021 First virtual EEPM course delivered (November) TABLE 3. TIMELINE OF KNOWLEDGE MANAGEMENT KEY EVENTS FISCAL YEAR KEY EVENTS 2016 Education Toolkit Platform redesigned and launched (September) Assumed secretariat duties for BE2 Annual meeting Supported Mobiles for Education Symposium 2017 Monthly newsletter launched QUEST landscape mapping report KM and ECP Portfolio manager begins (November) 2018 November 2017 – March 2018: USAID Hiring Freeze Launched EduLinks (Sept 2018) Coordinated 2018 M4E Developed How-To Notes Created content strategy 2019 December 2018 – Jan 2019: Government Shutdown Knowledge product packages mEd symposium mEd sustainability report Migrated ECCN website to an EducationLinks microsite/spoke site (Year 4) EducationLinks Roadshow 2020 March 2020 – COVID-19 forces all work and events to be moved online indefinitely Launch of GRN EducationLinks spoke site 113 | EPIC EVALUATION REPORT USAID.GOV TABLE 4. TIMELINE OF ENGAGEMENT, COMMUNICATIONS, AND PARTNERSHIPS KEY EVENTS FISCAL YEAR KEY EVENTS 2016 Engagement stakeholder survey and analysis AAR of 2015 Education Summit Began planning for a 2017 SOTA Books for Every Child Website GBA Retreat 2017 DDI/EDU brand assessment/audit SOTA conference postponed 2018 November 2017 – March 2018: USAID Hiring Freeze Developed and facilitated trainings Provided GBA with comms specialist Began managing key outreach platforms: Facebook, Twitter, external newsletter Designed/edited 2018 progress report 2019 December 2018 – Jan 2019: Government Shutdown Ed strategy and Ed policy launch Developed several learning agendas Restructured team Developed a creative brief On-demand coaching for 11 DDI/EDU and Mission staff 2020 March 2020 – COVID-19 forces all work and events to be moved online indefinitely 114 | EPIC EVALUATION REPORT USAID.GOV ANNEX 5: FACTORS THAT FACILITATED AND HINDERED ACHIEVEMENT OF EPIC RESULTS FACILITATING FACTORS HINDERING FACTORS External All Workstreams Transition to new administration (2017) KM Specific Interest and demand for accessible multimedia knowledge and learning products PD Specific COVID-19 facilitated transition to adoption of virtual learning modalities Increase in demand for CVE training ECP Specific Online communication amplified and accelerated by the pandemic (2020) All Workstreams Transition to new administration (2017) Government shutdowns Hiring freeze COVID-19 ECP Specific Demands from Congress, White House kept attention focused on responding with less focus on strategic communications planning and programming COVID-19 led to cancellation of Global Education Conference (GEC) Agency & Center of Education All Workstreams Strong technical expertise of DDI/EDU staff High level of commitment to Agency and sector Mission OE Specific Effective COR champion Leadership support of EPIC and the OE workstream Leadership team working and speaking together as one Leadership addressed difficult personnel issues High-level technical expertise in the education sector Growth in size of DDI/EDU from 28 to 45 staff increased Office capacity in a short period PD Specific Hiring effective, collaborative leadership that champions PD Thought partnership with EPIC and inclusion in the PD Working Group Providing consultation/collaboration with EPIC on creation of materials All Workstreams Turnover of staff, younger staff desiring more/different opportunities contributed to greater workload for those who remained, which contributed to stress and lower morale and less time for staff to participate in DDI/EDU events Lack of follow-up on agreements & plans, not holding people accountable Staff overworked (result of insufficient staffing to meet workload demands and staff turnover, and contributed to lack of follow-up) OE Specific Lack of leadership support for OE workstream at times Transition of leadership to new posts/positions can be disruptive, takes new leaders time to get established Promotion of staff who lack management skills and experience to management roles without the training necessary to be successful in these roles EMT focus on managing up to meet demands from higher-ups limited ability to lead and manage the Center 115 | EPIC EVALUATION REPORT USAID.GOV FACILITATING FACTORS HINDERING FACTORS KM Specific Strong COR and Activity Manager leadership Growing strategic support and infrastructure for knowledge and learning dissemination and engagement ECP Specific EducationLinks was a robust platform through which to disseminate comms products High quality technical expertise in communications and education Administration shift in focus to Journey to Self￾Reliance (some saw this transition as a hindering or neutral factor) New education policy Movement toward online learning and communication before COVID and then further accelerated by the pandemic Over time, increased “coming together across teams” to develop communications products Competitive nature of budgeting process Growth in size of DDI/EDU from 28 to 45 in a short period Lengthy reorganization process to Democracy, Development, and Innovation Bureau (DDI) delayed restructuring of the Center of Education to align with the new Education Policy and Education Strategy Office silos and competition among them, which decreased over time Toxic personalities PD Specific Lack of leadership support and accountability for this workstream Lack of clarity around core competencies Personalities affected effectiveness of positions, lack of willingness to collaborate with EPIC Turnover and changes in leadership Confidentiality requirements mean that not all materials can be shared for PD/KM purposes Bureaucratic difficulty: significant regulation and permissions needed to make decisions, which can lead to delays and projects being scrapped after significant investment KM Specific Limited time and staff resources Inconsistent clarity and communication of DDI/EDU’s priorities and needs Limited KM structure and strategy at the beginning of project ECP Specific LPA Agency guidelines, politics, and red tape Getting everyone on board with new administration policy Limited number of USAID staff on this team Multiple staff transitions in leadership and staffing of this team resulted in the team lacking cohesion Initial competitive internal Office and Agency culture that staff emulated externally, at times 116 | EPIC EVALUATION REPORT USAID.GOV FACILITATING FACTORS HINDERING FACTORS Many partnerships originated when Office was siloed and relied on relationships with individuals rather than being part of an Office￾wide partnership strategy No comprehensive communications strategy or partnering definition and strategy to guide the work, more product oriented Leadership transitions in the Agency (e.g., DAA) and in DDI/EDU EPIC Team All Workstreams Established trusting relationships Flexible staffing approach to meet DDI/EDU needs over time Listened to and acted on feedback Embedded as extension of DDI/EDU Center while maintaining an external perspective and role Focus on helping Office create a shared vision of what an effective Office looked like Creating spaces for open and honest reflection and dialogue Value people as people Flexible scope of work and budget facilitated responsiveness Strong OD, consultation, facilitation, and coaching skills PD Specific Splitting lead from OE/PD combination to singular PD lead Using technical experts as consultants in training design Repeated delivery of trainings allowed opportunity for CLA Team members with technical education experience KM Specific Expansion of KM staff and consultant resources including KM Portfolio Manager All Workstreams Initial delays in finding the right staffing MEL framework, systems, and processes took time to develop, in part because no MEL position was fully embedded into the core team and in part because it took time to co-create a shared results framework OE Specific Staff overutilized at times Technical expertise in OD/OE not universally understood or valued by all DDI/EDU staff PD Specific Lack of education sector technical expertise of some staff designing trainings KM Specific Need for more in-depth analysis to guide capture, synthesis, and sharing of evidence Time required to clarify roles, responsibilities, and expectations of KM, ECP, and PD workstreams and team member support (e.g., priority audiences and content, areas of technical expertise) Initial diverse expectations about EPIC’s role in evidence generation/capture vis a vis other mechanisms ECP Specific Delay in finding right person to lead this workstream Did not always have clear guidance from USAID 117 | EPIC EVALUATION REPORT USAID.GOV FACILITATING FACTORS HINDERING FACTORS Cross-workstream collaboration to strengthen the learning approach Adaptable and flexible in responding to emerging internal and external audience needs Communications expertise and robust content development Collaboration with other mechanisms ECP Specific High quality technical expertise in communications, OD, partnership, online technology platforms, and tools Good relationships with DDI/EDU Office ECP team 118 | EPIC EVALUATION REPORT USAID.GOV ANNEX 6: DETAILED ORGANIZATIONAL WELLNESS RESULTS (FEBRUARY, MAY, DECEMBER 2018) 119 | EPIC EVALUATION REPORT USAID.GOV ANNEX 7: LEADERSHIP DEVELOPMENT PROGRAM & PROFESSIONAL ENHANCEMENT SERIES EVALUATION RESULTS - LIFE OF PROJECT The worksheet with all evaluation data can be found at: https://drive.google.com/file/d/1PPM3t3gWwOfOMj9UO-LDJm3MG_2y2vBM/view?usp=sharing 120 | EPIC EVALUATION REPORT USAID.GOV ANNEX 8: PD ACTIVITIES AND PRODUCTS BY YEAR YEAR 1 ACTIVITY PARTICIPANT S Core Courses: Education Overview Course (DC) 29 Webinars: Flexible Instruments 24 YEAR 2 ACTIVITY PARTICIPANTS Core Courses: Effective Education Portfolio Management (DC) 24 Education Overview Course (DC) 19 Effective Education Portfolio Management (DC) 10 Webinars: Evidence in Early Grade Reading 9 Host Country Engagement and Sustainability 20 121 | EPIC EVALUATION REPORT USAID.GOV YEAR 3 ACTIVITY PARTICIPANTS Core Courses: Effective Education Portfolio Management (Bangkok) 11 Effective Education Portfolio Management (DC) 14 Effective Education Portfolio Management (Accra) 13 Advanced Courses: Education in Countering Violent Extremism (Bangkok) 33 Education in Countering Violent Extremism (Frankfurt) 26 Webinars: Using Evidence in Designing Ed Programs 23 Host Country Engagement 27 Collecting Data on Disabilities 24 Host Country Engagement and Sustainability 4 Non-State Schools 3 Cost Sharing 13 Toward Sustainability: Public, Private, and Hybrid Models for Education Finance 8 Cost Sharing 8 122 | EPIC EVALUATION REPORT USAID.GOV YEAR 4 ACTIVITY PARTICIPANTS Core Courses: Effective Education Portfolio Management (DC) 15 Advanced Courses: Education in Countering Violent Extremism (DC) 17 Distance Learning Courses: Positive Youth Development 101 Positive Youth Development 201 Webinars: Strengthening the Service Delivery System for Inclusive Education 35 Performance Plan and Report Webinar to the Field 8 LGT Partnership Presentation 5 Higher Ed (Morocco) 9 Higher Ed (Paraguay) 8 Revisions for Higher Education Indicators 5 GDA Call for Concept Papers I 21 Expert Panel Review on USAID's Expressive Language Module 3 GDA Call for Concept Papers II 7 GDA Call for Concept Papers III 7 123 | EPIC EVALUATION REPORT USAID.GOV Social and Emotional Learning 10 Videos: Inclusive Education 129 Integrating Gender into Education Programs: Best Practices from Honduras 62 Disability Inclusive Education 192 Rapid Evaluation and Risk Analysis 59 Social and Emotional Learning 197 Principles of Education Policy Priorities of Education Policy YEAR 5 ACTIVITY PARTICIPANTS Core Courses: Education Overview Course (virtual) 96 Distance Learning Courses: Disability Inclusive Ed 244 SEL 277 Reading and Literacy 221 Youth Workforce Development 206 Pre-Primary Education 372 Higher Education 135 124 | EPIC EVALUATION REPORT USAID.GOV Education in Crisis and Conflict 59 Webinars: COVID-19 and Education 120 Distance Learning 131 Safety and SEL during COVID-19 80 Return to Learning 118 COVID-19 Related Hygiene and Safety in Schools 112 Measuring Read, Engagement and Outcomes of Distance Learning I 55 Measuring Read, Engagement and Outcomes of Distance Learning II 60 GELS Products Panel Discussion (Opening) Webinar: Connecting Across Organizations Ignite Talks: Learning through Play by LEGO Foundation and Right to Play Panel Discussion Webinar: Distance Learning as a Tool for System Resilience Panel Discussion Webinar: Social and Emotional Learning (SEL) Ignite Talks: Evidence for Gender and Education Resource (EGER) and Waache Wasome Tea with a SME: Disability Inclusive Education: Successes and Lessons Learned from Nepal and Malawi Ignite Talks: Private Sector Engagement (PSE) with Google Read Along and USAID Internal Session: Co-creation with Office of Acquisition and Assistance (OAA) Panel Discussion Webinar: EduFinance Ignite Talks: YWFD with Educate! and Education Development Center (EDC) 125 | EPIC EVALUATION REPORT USAID.GOV YEAR 6 CORE COURSES: PARTICIPANTS Effective Education Portfolio Management (Virtual) 29