Submitted May 4, 2022 This publication was produced for review by the United States Agency for International Development/Ethiopia. It was prepared independently by Social Impact, Inc. FINAL REPORT Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity April 4, 2016 – June 3, 2022 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity ii Final Report EPMES FINAL REPORT April 2016 – June 2022 Ethiopia Performance Monitoring & Evaluation Service (EPMES) Activity Contracted under AID-663-C-16-00010 Prepared for: Awoke Tilahun (COR); and Meseret Tesfaye (Alternate COR) United States Agency for International Development/Ethiopia (USAID/Ethiopia) American Embassy Entoto Street P.O. Box 1014 Addis Ababa, Ethiopia Prepared by: Social Impact, Inc. Bole Sub City, Woreda 13 House # 478, 4th Floor Addis Ababa, Ethiopia Ethiopia Performance Monitoring and Evaluation Service iii Final Report Contents...................................................................................................................................iii Tables and Figures .................................................................................................................. iv Acronyms ................................................................................................................................. vi Executive Summary.............................................................................................................. viii The EPMES Activity Overview......................................................................................................................... viii Objective 1: Provide External Evaluation, Assessment, and Study Services......................................... viii Objective 2: Provide Performance Monitoring Technical Support Services to Strengthen Evidence￾based Decision-making ....................................................................................................................................... viii Objective 3: Improve and Enhance the Capacity for Learning and Adaptation within USAID/Ethiopia Throughout the Program Cycle........................................................................................... x Objective 4: Strengthen Local Partners’ Technical and Management Capacities................................... x Selected Lessons Learned and Recommendations........................................................................................xi EPMES Activity Overview....................................................................................................... 1 Overall Objectives................................................................................................................................................. 1 The EPMES Team................................................................................................................................................... 1 Key Achievements by Purpose/Objective.............................................................................. 2 Overall Purpose: Improve Collaboration, Learning, and Adapting to Strengthen Evidence-based Planning and Performance Management Across the USAID Program Cycle .......................................... 2 Objective 1: Provide External Evaluation, Assessment, and Study Services............................................ 2 Objective 2: Provide Performance Monitoring Technical Support Services to Strengthen Evidence￾based Decision-making .......................................................................................................................................... 8 RF Sub-purpose 2-B: Increase Utilization of GIS Services for Planning and Decision-making ...........15 Objective 3: RF Sub-purpose 3: Improve Integration of CLA in USAID/Ethiopia Programs and Processes................................................................................................................................................................18 Objective 4: RF Sub-purpose 4: Strengthen Technical & Management Capacity of Local M&E Service Providers..................................................................................................................................................25 Management Highlights......................................................................................................... 29 Start-up...................................................................................................................................................................29 Challenges and Successful Adaptations...........................................................................................................29 Contract Modifications.......................................................................................................................................31 Subcontracts ..........................................................................................................................................................33 Closeout ................................................................................................................................................................. 35 Key Lessons Learned & Recommendations......................................................................... 36 Cross-cutting.........................................................................................................................................................36 Enhanced Utilization of ESAs.............................................................................................................................37 Capacity Building and Use of LME Tools........................................................................................................37 GIS ...........................................................................................................................................................................39 CLA .........................................................................................................................................................................39 Appendix 1: EPMES Results Framework ............................................................................. 41 Appendix 2: Contract Deliverables ...................................................................................... 42 Appendix 3: Final Indicator Tracking Table ........................................................................ 45 Appendix 4: Final Financial Report....................................................................................... 48 CONTENTS Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity iv Final Report Tables Table 1: The EPMES team......................................................................................................................................... 1 Table 2: Comparison of baseline and target against achievement for Purpose 1 indicators.................... 2 Table 3: Comparison of baseline and target against achievements for Sub-purpose 1 indicators.......... 3 Table 4: Comparison of baseline and target against achievement for Output 1.1 indicators.................. 4 Table 5: Type and number of evaluations/studies completed by year........................................................... 5 Table 6: Comparison of achievement against target for Objective 2 indicator........................................... 8 Table 7: Summary of achievements by indicator for Output 2.1.................................................................... 9 Table 8: Summary of trainings to Mission Staff by training topic .................................................................... 9 Table 9: Number of IP staff and others trained................................................................................................. 10 Table 10: Summary of achievements by indicator for Output 2.2................................................................13 Table 11: Number of LME plan reviews by project..........................................................................................13 Table 12: Summary of achievements by indicator for Output 2.3................................................................14 Table 13: Summary of achievements by indicator for Output 2.2.1.............................................................15 Table 14: Summary of achievements by indicator for Output 2.2.2.............................................................17 Table 15: Summary of indicators for Sub-purpose 3........................................................................................18 Table 16: Summary of indicators for Output 3.1..............................................................................................19 Table 17: Summary of indicators for Output 3.2..............................................................................................20 Table 18: Summary of indicators for Output 3.3..............................................................................................23 Table 19: Summary of indicators for Output 3.4..............................................................................................24 Table 20: Summary of indicators for Sub-purpose 4........................................................................................25 Table 21: Summary of indicators for Output 4.1..............................................................................................26 Table 22: Summary of indicators for Output 4.2..............................................................................................27 Table 23: Summary of indicators for Output 4.2..............................................................................................28 Table 24: Challenges and EPMES's adaptations..................................................................................................30 Table 25: EPMES contract modifications.............................................................................................................31 Table 26: The EPMES Contract Deliverable Tracking Table..........................................................................42 Table 27: The EPMES Performance Indicator Tracking Table .......................................................................45 Table 28: EPMES funds reported as expended..................................................................................................48 Table 29: Estimated funds expended in the current quarter.........................................................................48 Table 30: Un-liquidated obligations......................................................................................................................49 Table 31: EPMES's significant expenses ...............................................................................................................49 Table 32: EPMES's activity-level spending ...........................................................................................................50 Figures Figure 1: Number of evaluations/studies (all years) ........................................................................................... 5 Figure 2: Group discussion during the MEL & CLA Fundamentals in-person training (May 2019) ........ 6 Figure 3: Evaluations, assessments, and studies by sector................................................................................ 7 Figure 4: Percent of Mission staff trainees by subject......................................................................................10 Figure 5: Fundamentals of MEL and CLA training (2019) ...............................................................................11 Figure 6: Discussion on “user stories” during Agile and Adaptive Management in the USAID Context training.........................................................................................................................................................................12 Figure 7: Overall rating of 18 indicators (2021 DQA) ....................................................................................14 Figure 8: Number of maps produced by year....................................................................................................16 Figure 9: Number of maps produced for each Mission technical office ......................................................16 Figure 10: USAID/Ethiopia’s CDCS Results Framework (2019-2024) ........................................................20 Figure 11: Poster from the Gender & Youth Learning Event........................................................................21 Figure 12: Output of a participatory exercise during 2022 virtual Gender Learning Workshop.........22 Figure 13: Documents uploaded to the KM Catalogue...................................................................................24 Figure 14: Baseline and endline TOCAT scores for capacity building of M&E local service providers .......................................................................................................................................................................................26 Figure 15: Number of registered local service providers...............................................................................27 Figure 16: Learning Plan Snapshot ........................................................................................................................38 TABLES AND FIGURES Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity v Final Report Figure 17: The EPMES Results Framework.........................................................................................................41 Figure 18: EPMES expenditures by category ......................................................................................................51 Figure 19: EPMES monthly invoice values...........................................................................................................51 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity vi Final Report A/COR Agreement or Contracting Officer’s Representative ADS Automated Directives System AGP Alternative Growth Poles ALMEP Activity Learning, Monitoring, and Evaluation Plan ALT Assets and Livelihoods in Transition AOR Agreement Officer’s Representative CBLD-9 Percent of USG-assisted organizations with improved performance [IM-level] CCN Cooperating Country National CDCS Country Development Cooperation Strategy CLA Collaborating, Learning, and Adapting CMDRR Community-managed Disaster Risk Reduction COP Chief of Party COR Contracting Officer’s Representative COVID-19 Coronavirus Disease 2019 DFS Digital Financial Solutions DG Democracy and Governance DIS Development Information Solution DO Development Objective DQA Data Quality Assessment EG&T Economic Growth and Transformation EPMES Ethiopia Performance Monitoring and Evaluation Service EQUI® Evaluation Quality, Use, and Impact ESA Evaluations, Studies, and Assessments EYO Education and Youth Office FFP Firm Fixed Price FY Fiscal Year GCN Gender Champions Network GESI Gender, Equality, and Social Inclusion GIS Geographic Information Systems GoE Government of Ethiopia GRAD Graduation with Resilience to Achieve Sustainable Development HAPN Health, AIDS, Population, and Nutrition HQ Headquarters HR Human Resources HRPS Human Rights Protection Support IM Implementing Mechanism IP Implementing Partner IQC Indefinite Quantity Contract ISS-Africa Institute for Security Studies-Africa ISS-Ethiopia Information Systems Services KM Knowledge Management KP Key Personnel LME Learning, Monitoring, and Evaluation LMG Leadership, Management, and Governance LOP Life of Project LSP Local Service Provider M&E Monitoring and Evaluation MEL Monitoring, Evaluation, and Learning OAA Office of Acquisition and Assistance ODA Organizational Development Activity OFDA Office of U.S. Foreign Disaster Assistance PAS Public Affairs Section PIRS Performance Indicator Reference Sheet PMP Performance Management Plan ACRONYMS Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity vii Final Report PMU Program Management Unit POP Period of Performance PPR Performance Plan and Report PRO Program Office PSE-ECE Private Sector Engagement in Early Childhood Education Q Quarter RF Results Framework R2 Relief and Resilience SAGE Strategic Advisory Group for Emergencies SART Sub Saharan Africa Research and Training Center SD Sustaining Dialogue SEUH Strengthening Ethiopia’s Urban Health SHRH Strengthening Human Resources for Health SI Social Impact, Inc. SOW Statement of Work TB Tuberculosis ToC Theory of Change TOCAT Technical and Organizational Capacity Assessment Tool USAID United States Agency for International Development USD United States Dollar USG United States Government Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity viii Final Report The EPMES Activity Overview The EPMES activity was intended to provide USAID/Ethiopia with project/activity external evaluation, performance monitoring, and performance management support services in alignment with the USAID Automated Directives System (ADS) and Evaluation Policy. EPMES was also intended to assist USAID/Ethiopia and its partners to generate empirical data on project/activity implementation as well as foster learning and knowledge management (KM) to improve evidence￾based project design and development programming in support of the Country Development Cooperation Strategy (CDCS). The period of performance (POP) for the EPMES Activity was April 4, 2016, through June 3, 2022, with a contract value of $16,658,919.88. EPMES had four objectives that aligned with the EPMES results framework (RF). Objective 1: Provide External Evaluation, Assessment, and Study Services Objective 1, aligned with Sub-purpose I of the EPMES RF, was to enhance the utilization of external evaluation findings to inform planning and management decisions by USAID/Ethiopia and its implementing partners (IPs). EPMES undertook 24 evaluations, assessments, and studies over the life of the activity.1 One of the major achievements of EPMES was its support to USAID/Ethiopia to develop the 2019- 2024 CDCS. This support included a high-level gender analysis to support the development of the CDCS and the gender synthesis to inform the design of the 11 projects. The 2020 gender synthesis report was used to develop Projects and Activities to operationalize the 2019-2024 CDCS. The Ethiopia Trends/Futures Analyses and “Emerging Giant – Potential Pathways for Ethiopia 2040,” developed by Institute for Security Studies (ISS-Africa) under EPMES, were also used to support CDCS development, as discussed further under Output 3.1. Additionally, EPMES undertook a closeout analysis of the prior CDCS. EPMES met 100 percent of targets for Output 1.1, including the number of evaluations, studies, assessments, and reviews of evidence completed to foster learning and support adaptive management. EPMES also contracted and mentored local consultants during these studies. Most products were final evaluations or analyses, with the majority conducted in the earlier years of the activity when the CDCS 2.0 was being developed. The highest demand was for cross-cutting themes, followed by those for the Education and Youth Office (EYO). In 2022, EPMES also engaged with all Mission offices to help them consider and refine their post-conflict assessment needs, ultimately completing three tools for the EYO to provide to partners to assess conflict-related damage. Objective 2: Provide Performance Monitoring Technical Support Services to Strengthen Evidence-based Decision-making Capacity Building and Utilization of Performance Monitoring (Sub-purpose 2-A) Sub-purpose 2-A, fitting under Objective 2 of the EPMES work plan, was to increase the capacity and utilization of performance monitoring by USAID/Ethiopia and its IPs. Under Output 2.1, the proportion of training participants who indicated they had applied the skills and knowledge they received from trainings was 98 percent, exceeding the planned target of 90 percent. The number of IP staff trained likewise exceeded the target. EPMES trained IPs on topics such as the fundamentals of monitoring, evaluation, and learning (MEL), collaborating, learning, and adapting (CLA), KM, and the Development Information Solution (DIS). EPMES also delivered trainings on qualitative methods and analysis and agile and adaptive management in the USAID context. EPMES, in addition, provided mentoring or coaching to 29 IPs in 2020 and 2021. 1 Counting the 11 actuals external evaluations findings informing decisions that was achieved prior to establishing targets, EPMES reached 320 percent of the target of five. EXECUTIVE SUMMARY Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity ix Final Report Targets for the number of Mission staff trained were not met due to lower-than-expected demand from the Mission, especially during the Coronavirus Disease 2019 (COVID-19) pandemic period. However, EPMES provided trainings related to Geographic Information Systems (GIS) and related interactive dashboards as requested, as well as trainings on learning, monitoring, and evaluation (LME)2 broadly and data quality assessments (DQA). Under Output 2.2, which was to support the Mission to manage the planning and processes of strategic monitoring, evaluation, and learning, EPMES was asked to support the review and development of two LME or CLA plans. However, EPMES reviewed already approved LME plans for 28 Activities. Key limitations included that 30 percent of LME plans did not identify risks or assumptions, and 40 percent of LME plans did not reference evidence in the Theory of Change (ToC). Under this output, EPMES also: • Prepared and shared a new activity learning, monitoring, and evaluation plan (ALMEP) template; • Developed and shared an indicator priority and selection checklist tool; • Prepared and shared an indicator mapping tool and review rubric with guidance; • Undertook an analysis of performance indicator reference sheet (PIRS) for multiple Activities; and • Delivered ALMEP slide deck presentation to the USAID Program Office (PRO) team. Under Output 2.3, increasing the utilization of MEL tools and practices in the Program Cycle, EPMES revised the existing USAID DQA tool to improve its effectiveness for rating against USAID’s data quality standards. In addition, EPMES: • Digitized DQA checklists used by USAID to simplify the capture, storage, and analysis of data quality trends; and • Designed an Excel database and macro-enabled entry form to store DQA information and a live dashboard to summarize five key DQA findings and identify priority gaps. EPMES also conducted two metadata analyses. The first, in 2017, covered six indicators of IPs under the Relief and Resilience (R2) Office (previously called Assets and Livelihoods in Transition (ALT) Office) and Democracy and Governance (DG) Office. The second, in 2021, covered 18 indicators from health, governance, agriculture, and education. Increasing GIS Utilization (Sub-purpose 2-B) Under Output 2.1, EPMES accomplished significant work that promoted the production of geographic information in maps, interactive dashboards, and tabular products for Mission technical offices to integrate geographic thinking into the Mission’s planning and decision-making efforts. More specifically, EPMES: • Produced more than 865 thematic maps; • Developed three interactive dashboards for emergency shocks, secondary cities, and an alternative growth poles (AGP) mapping; and • Produced tabular products with detailed information about different indicators, presence, and types of USAID activities at the woreda (district) level. Under Output 2.2, EPMES developed a geodatabase by gathering, organizing, and storing georeferenced data obtained from the Mission and other external sources such as online data portals, published atlases, and government offices. EPMES used the data to produce maps, develop dashboards, and conduct spatial analyses. 2 In this document, MEL and LME are used interchangeably. Both refer to the same thing. “L” first simply means an emphasis to the “Learning” aspect of the Program Cycle as USAID/Ethiopia is considering itself as a learning organization. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity x Final Report Objective 3: Improve and Enhance the Capacity for Learning and Adaptation within USAID/Ethiopia Throughout the Program Cycle Objective 3, aligned with Sub-purpose 3 of the EPMES RF, is designed to support the Mission to intentionally integrate CLA with programming and to build and strengthen a culture of learning. EPMES supported nine projects and processes based on Mission demand and direction. Under Outputs 3.1 and 3.2, EPMES supported the completion of two MEL/CLA plans and built the capacity of 44 IPs. More consequential was the significant support EPMES rendered for the development of USAID’s CDCS for the period 2019-2024 that built on from the support under Sub-purpose 1, such as through regional and other consultation dissemination workshops, an RF development workshop, and by developing a learning agenda. EPMES also played a pivotal role in the CDCS Closeout Analysis, which aimed to document the achievements, challenges, and lessons learned from the implementation of USAID/Ethiopia’s 2011-2019 CDCS. EPMES undertook a broad CLA Mapping Exercise and used data, findings, and analysis from a CLA survey and CLA action planning sessions to update relevant sections of the Mission’s CDCS CLA Plan and the Leadership Charter. EPMES incorporated the CLA language into the Office of Acquisition and Assistance (OAA) instruments. EPMES also supported the Gender Champions Network (GCN) and organized two gender learning events in 2018 and 2022. EPMES facilitated several processes, including an interagency strategy meeting, an EYO retreat, a strategic planning exercise for the Public Affairs Section (PAS) of the Department of State, Youth Project Design Team meetings (including conducting a rapid synthesis of evidence on youth in Ethiopia), and a session on developing a Digital Financial Solutions (DFS) learning agenda. EPMES also provided support in developing a strategy for the Ambassador’s Strategic Planning Workshop. Under Output 3.3, implementing an effective system for KM, EPMES uploaded 89 documents as an Evidence Catalogue. EPMES also assessed proposed system changes, discussed, and examined various options until the Mission Knowledge Management Specialist indicated the Mission’s KM plans no longer required further input. Objective 4: Strengthen Local Partners’ Technical and Management Capacities Objective 4, aligned with Sub-purpose 4 of the EPMES RF, was aimed to strengthen the capacity of local monitoring and evaluation service providers to provide quality monitoring and evaluation (M&E) services to USAID and its IPs, the government of Ethiopia (GoE), and other stakeholders. EPMES successfully met or exceeded its targets in working with and building the capacity of five local service providers (LSPs). However, the improvements, measured through baseline and endline measures using the Technical and Organizational Capacity Assessment Tool (TOCAT) were not as strong as hoped, as the LSPs tended to rely on consultants to undertake much of their work. As part of Output 4.1, EPMES built, maintained, and accessed a roster of local M&E experts to have up-to-date information on the number, capacity, experiences, and activities of the LSPs. EPMES also provided capacity strengthening support to 52 LSPs across various technical, operational, and business development issues, reaching 67 percent of the target. Output 4.2 was to develop a host country M&E system capacity strengthening strategy document that built a functional and sustainable national M&E system to meet the M&E requirements of the government, donor groups, and other actors in the development arena. EPMES developed and presented on a host-country M&E capacity strengthening roadmap and action plan. This result was not fully achieved, however, as the stakeholders’ workshop for the final validation of the draft strategy was placed and left on hold due to COVID-19. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity xi Final Report Selected Lessons Learned and Recommendations Selected lessons learned and recommendations are presented below. Please see the full text for the complete list and rationale. Cross-cutting • Prioritize innovative learning processes such as joint data reviews or evaluative stocktaking with IPs to drive demand for more regular collection and use of data, and spur adaptive management. • Engage learning mechanisms more directly and consistently in Mission processes. • Clarify planned use for each major contractor task and prioritize those with clear implementation plans. • Standardize need identification and utilization tracking processes for learning contractors through regular meetings with technical teams and their points of contact. Evaluations, Studies, and Assessments (ESAs) • Review ESA statements of work (SOWs) with a lens toward Mission-level learning agenda questions. • Include in ESA SOWs intentional dissemination and utilization plans engaging IPs and, as relevant, non-USAID stakeholders for in- or out-briefs, collective sense-making sessions, or recommendation workshops. • Improve and elevate the profile of post-evaluation action tracking of recommendations. • Plan ahead for all ESAs at least eight to nine months in advance to allow realistic timelines across assignments. Capacity Building and Use of LME Tools • Collect additional cross-activity context monitoring and qualitative data. • Conduct annual LME skills and needs survey among Mission and IP staff to target capacity building support. • Undertake the capacity building of Agreement or Contracting Officer’s Representatives (A/CORs) in data quality assessment so that LME contractors play a supporting role. • Align ALMEP content with Project- and Mission-level learning questions, approaches, and data, and include more outcome-level measures to improve evidence-based adaptive management. • Re-engage with other development partners on the roadmap, strategy, and action planning for local LME capacity building development, providing varying levels and types of capacity building support by providers’ needs. GIS • Establish a GIS working group or network of IPs to exchange data and work together on potential GIS tools and data applications. • Promote the integration of geographic thinking during planning among technical offices by familiarizing them with existing products via small learning events. CLA • Follow-up on the efforts undertaken as part of and related to the CLA Mapping Exercise. • Consider establishing more sectoral or thematic communities of practice with whom to share LME practices. • Continue fostering adaptive management practices by granting additional flexibility in ALMEP indicators and work by integrating the crisis modifiers across all new activities. • Improve adaptive management through further empowering A/CORs and decentralizing decision-making processes. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity xii Final Report • Build capacity in adaptive management at the management level for the Mission and IPs through training, coaching, and mentoring. • Undertake innovative and impactful learning processes that foster a deeper understanding of evidence and drive adaptive management, such as joint data reviews and evaluative stocktaking. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 1 Final Report Overall Objectives The EPMES Activity’s stated objectives are: 1. To provide external evaluation services—including impact and performance evaluations, as well as assessments (i.e., gender assessments, sustainability assessments)—to USAID at the project design stage (Project Appraisal document development) and multiple implementation levels (e.g., activity, project, etc.) and in all sectors in which the Mission works; 2. To provide performance monitoring technical support services to the Mission to strengthen evidence-based decision-making for project design and implementation of development programming; 3. To improve and enhance the capacity for learning and adaptation within USAID/Ethiopia throughout the Program Cycle; and 4. To strengthen local partners’ technical and management capacities, ensure sustainability, and improve knowledge of USAID policies, regulations, and practices related to M&E to increase the pool of potential M&E service providers for USAID and other donors in Ethiopia. The initial life of project (LOP) for the EPMES Activity was five years beginning April 4, 2016. Ultimately, the LOP was adjusted to make EPMES a six-year, two-month Activity covering the period of April 4, 2016 to June 3, 2022. Despite the adjustments in timeline, the contract budget has maintained its initial value of $16,658,919.88. The EPMES Team Below, in Table 1, is the composition of the EPMES team as of May 2022. Table 1: The EPMES team Title Name Chief of Party (COP) Marc Douglas Shapiro Deputy Chief of Party (DCOP-Administration) Mahlet Yared Sr. Director of Learning, Monitoring, & Training Abduselam Muse Usman GIS Director Zemenu Abegaz Mintesnot Sr. MEL Advisor Simegnew Eshetie Mitiku CLA Advisor Kibrewosen Worku Belete Finance Director Solomon Tesfaye Begashaw Finance Officer Melatwork Mahteme Habtemariam HR Manager Biruk Belayneh Hassen Operations Officer Mulunesh Shiferaw Woldemichael Driver & Logistics Assistant Berhanu Habte Yemiru Driver & Logistics Assistant Eritrea Lulie Yigermal Office Cleaner Gedame Amera Alemayehu EPMES ACTIVITY OVERVIEW Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 2 Final Report The Activity Learning, Monitoring, and Evaluation Plan of EPMES showed that the Results Framework of the Activity (see Appendix 1) was structured in the form of goal, purpose, sub-purpose, and outputs. The goal of EPMES was to improve Program Cycle management and accountability by USAID/Ethiopia and its IPs in support of the Mission’s CDCS. In line with this, the salient achievements of the Activity across its purpose, the sub-purposes, and outputs are described below. Overall Purpose: Improve Collaboration, Learning, and Adapting to Strengthen Evidence-based Planning and Performance Management Across the USAID Program Cycle The indicator to measure this higher-level result was the number of EPMES sub-activities or tasks with documented utilization. This referred to discrete pieces of work that EPMES conducts, such as training, evaluation, study, facilitations/events, etc. Although the structure of the Activity and agreed-upon tasks and methods of collecting indicators did not lend themselves to consistently capture utilization, EPMES conducted many discrete pieces of work during the period of the Activity where use had been documented. Table 2: Comparison of baseline and target against achievement for Purpose 1 indicators Indicator Indicator Type Baseline Target Actual Percent of Target Reached Number of EPMES sub-activities (tasks) with documented utilization during the fiscal year (FY) Output 0 11 23 209% *Note: Indicator and targets added in FY 2020 These tasks ranged from data quality assistance to studies to events. Many tasks were not conducive to documenting specific instances of utilization. This included one mainstay – training IPs in the fundamentals of CLA and MEL – although IPs did report through surveys that they used the knowledge learned. For many tasks, EPMES received evidence of appreciation and utilization, which will be presented throughout this report as relevant. However, the exact scale of utilization is difficult to report due to limitations in tracking and reporting and Mission needs that evolve from initially intended uses. The indicator was inflated somewhat by the fact that it was not established, nor targets set until 2020, and yet EPMES documented utilization of its evaluations, studies, and assessments prior to that date. EPMES documented 23 tasks utilized. Objective 1: Provide External Evaluation, Assessment, and Study Services The goal of Objective 1, which aligned with EPMES RF Sub-purpose 1, to enhance the utilization of external evaluation findings to inform planning and management decisions by USAID/Ethiopia and its IPs, was to support USAID/Ethiopia with timely access to evidence and information for activity and portfolio management. EPMES conducted evaluations, studies, assessments, and analyses of Activities upon the request of the Mission. The services of EPMES in this regard were expected to help the Mission take appropriate and timely decisions while working with USAID throughout the ESA process to strengthen utilization and foster adaptive management. KEY ACHIEVEMENTS BY PURPOSE/OBJECTIVE Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 3 Final Report Table 3: Comparison of baseline and target against achievements for Sub-purpose 1 indicators Indicator Indicator Type Baseline Target Actual Percent of Target Reached Number of evaluations/studies/ assessments with documented utilization of their findings and recommendations Output 0 5 16* 320% As noted earlier, this indicator, including targets, was added in FY 2020, and yet EPMES was able to document the utilization of findings and recommendations of evaluations, studies, and assessments prior to this date. EPMES documented 16 of its evaluations, assessments, and studies for which findings and recommendations were used. For the final fiscal year of the contract, EPMES did not document utilization, as one study was canceled due to changing Mission priorities post-conflict and another completed in the final two months of the Activity. One of the major achievements of EPMES during the period of performance was its support to USAID/Ethiopia to develop the 2019-2024 CDCS and its technical inputs in assessing gender issues. This included a high-level gender analysis to support the development of the CDCS and the gender synthesis to inform the design of the 11 projects. The 2020 gender synthesis report was used to develop Projects and Activities to operationalize the 2019-2024 CDCS. Mission staff indicated the analyses helped identify existing gaps on gender in the Mission Activities and design strategies to mitigate the shortcomings. The Ethiopia Trends/Futures Analyses and “Emerging Giant – Potential Pathways for Ethiopia 2040” accomplished through EPMES funding were used to support CDCS development. Further, EPMES undertook a closeout analysis of the prior CDCS. Another cross-cutting report was the Ethiopian Secondary Cities Assessment, which used rigorous data￾based analysis to identify those secondary cities most ready to serve as growth poles for development. As its methodology, EPMES used geographic locations and scoring of selected indicators to prioritize the surrounding alternative growth pole (AGP) woredas. This assessment was considered sufficiently important for planned Mission decision-making by Mission leadership; at the time, EPMES was assigned another task described under Sub-purpose 2 to integrate these findings with interactive mapping capability for offices and teams to use to better target and focus interventions to evolving priority areas. As the country context and Mission focus changed, the intended use of the report, however, was not fully realized. There are, however, cases where youth-focused interventions prioritized secondary cities for operations. There is a hope that the secondary cities analysis will be picked up more by other sectors and interventions. “I have started using relevant findings from the synthesis report to inform some of our project designs and continue using it as appropriate” – Meseret Kassa, USAID/Ethiopia Gender Specialist “We will use the insightful, data-driven findings and recommendations of this EPMES Secondary Cities Assessment and have conversations about the tradeoffs about where current and future activities will be if we were to invest more.” – Sean Jones, USAID/Ethiopia Mission Director Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 4 Final Report Another example is the “Youth Potential Activity: Youth Cohort Study,” the findings and recommendations used to improve the implementation of the then ongoing Youth Potential Activity. When developing the ToC for the second phase of the Youth Activity, the Study was used as one of the reference materials. The finding and recommendations were used to frame the thinking in developing a new youth-focused project, particularly to shape the problem statement and purpose of the new project. Even terms such as employability and youth empowerment, job preference and availability, labor market, the inappropriateness of dealing with private-public partnership in a rural setting, being inclusive (such as people with disabilities), etc. were studied and discussed as the new project was designed. The findings of the Study were also utilized for course corrections, including a focus on post-training support such as coaching, mentoring, and sustained engagement with stakeholders, including microfinance institutions. Output 1.1: Conduct Evaluations, Studies, Assessments, and Reviews of Evidence from Existing Sources to Foster Learning and Support Adaptive Management This output was aimed to contribute to and achieve the capacity of the Mission to enhance the utilization of external evaluation findings to inform planning and management decisions by USAID/Ethiopia and its IPs. EPMES completed many tasks toward achieving planned outputs. Each task involved several other sub￾tasks, including providing technical support to USAID/Ethiopia on designing and conducting high-quality and timely studies, engaging in data collection and analysis, and report writing and presentations. A summary of performance indicators and achievements is provided in Table 4. Table 4: Comparison of baseline and target against achievement for Output 1.1 indicators Indicator Indicator Type Baseline Target Actual Percent of Target Reached Number of evaluations/studies/ assessments completed Output 0 25 24 100% Number of local consultants contracted to conduct evaluations, studies, assessments, and review Output 0 42 42 100% Number of local consultants trained and/or mentored Output 0 42 42 100% When reviewing the progress of EPMES against the Activity indicator target, EPMES met its targets despite impediments like a worldwide pandemic, although this also reduced planned targets. As can be seen in Table 4, 24 different evaluations, assessments and studies were completed during the period of the Activity. The evaluations and studies conducted resulted in findings and recommendations that inform decision-making, mid-course implementation correction of activities, and future Activities’ design. Table 5 provides the type and number of the studies and assessments conducted during the lifetime of EPMES. “[USAID/Ethiopia] will take this Secondary Cities Assessment… and ensure that [it is] presented and used as a strategic reference during each Activity Design Phase and when we choose the geographic focus of any new USAID/Ethiopia Activity. The goal will be that the Mission will wind up with more targeted geographic and strategic synergies at the core of the 2019-2024 Country Development Cooperation Strategy.” – Paul Vaca, USAID/Ethiopia Program Office Director Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 5 Final Report Table 5: Type and number of evaluations/studies completed by year Fiscal Year Final Evaluation Mid-term Evaluation Baseline Analysis Assessment Synthesis Review Total 2016 1 1 2017 2 4 2 3 11 2018 2 2 2019 3 3 2020 1 2 1 4 2021 1 1 2 2022 1 1 Total 9 4 2 5 2 1 1 24 As reflected in Figure 1, ESAs completed by the year show that 38 percent of the studies were final evaluations, 21 percent were analyses, and 17 percent were mid-term evaluations. The remaining studies were baselines, assessments, syntheses, and reviews. The demand for studies or assessments was highest around the CDCS development and closeout period, with 46 percent of the studies completed in 2017. Figure 1: Number of evaluations/studies (all years) Figure 3 below shows how sectors including cross-sectoral reports requested these products. Cross￾sectoral requests represented the highest number followed by the offices of Education and Youth, Health, R2, then Democracy and Governance. One assessment requested by the EYO on private sector engagement in early childhood education (PSE-ECE) was halted in 2021 due to changing programmatic priorities post-conflict, and was not included in this figure. 0 1 2 3 4 5 6 7 8 9 Final Evaluation Analysis Mid-term evaluation Baseline Assessment Symthesis Review 9 5 4 2 2 1 1 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 6 Final Report Figure 2: Group discussion during the MEL & CLA Fundamentals in-person training (May 2019) Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 7 Final Report Figure 3: Evaluations, assessments, and studies by sector In addition and not counted in these figures, the Program Office Director requested EPMES and another implementer’s support to undertake sector-wise post-conflict assessments for the Mission. EPMES engaged with all offices to help them consider and refine their assessment needs, ultimately completing three multifaceted tools for the EYO to provide to partners for assessing conflict-related damage. Other sectors did not proceed with further EPMES support. Another indicator from Table 5 focused on EPMES’s involvement of local consultants, some of whom worked with SI on multiple evaluations. This indicator revolves around the building capacity of local consultants in M&E, which also relates to the work involved under Sub-purpose 2-A. EPMES met its target of involving 42 local consultants in the different evaluations and studies that the Mission assigned. Similarly, the target to train and mentor local consultants was fully achieved. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 8 Final Report Objective 2: Provide Performance Monitoring Technical Support Services to Strengthen Evidence-based Decision-making Objective 2 encompasses multiple services. To clarify these elements, the discussion below is divided by RF sub-purposes and related outputs. RF Sub-purpose 2-A: Increase Capacity and Utilization of Performance Monitoring by USAID/Ethiopia and IPs The established indicator to measure Sub-purpose 2-A under Objective 2 of EPMES’s work plan was the proportion of training participants who indicated they applied the skills and knowledge they received from trainings. This indicator was measured through post-training surveys in 2018, 2019, and 2021 to gauge the achievement of this result for the trainings of 44 IPs over the period of the Activity. Table 6: Comparison of achievement against target for Objective 2 indicator Indicator Indicator Baseline Target Actual Percent of Target Reached % of training participants who, after completing a post-training survey, say they have applied the skills and knowledge they received from the training Sub￾purpose 0 90 98 108% The survey covered some 70 trainees from USAID/Ethiopia IPs who had taken the fundamental training on MEL and CLA. For the 2018 and 2021 surveys, all the trainees responded that the training positively impacted their work both at the individual and organizational levels. Results for 2019 indicated that 96 percent of the trainees self-reported using the skills and knowledge acquired during the trainings. Anecdotal evidence supported claims from participants about the utilization of trainings from the post￾training surveys. The utilization of trainings on CLA spanned EPMES’s Sub-purposes 2A and 3. Output 2.1: Improve MEL Capacity Among USAID and IP Staff This output is designed to improve the capacity of USAID and IPs so that they can adequately monitor and evaluate the progress and results of their Activities and adapt from experience and prevailing realities on the ground during implementation. Thus, the established indicators for measuring this output and its achievements are presented in Table 7. The following are representative quotes from survey respondents: I am “more confident to explain the Activity MEL system to new staff.” The training “enhanced my understanding of CLA and learning agendas.” I have more “personal skills on how to improve data quality and conduct a data quality assessment.” “I have shared training tips and materials with all regional MEAL teams and project managers.” Now, “routine monitoring with performance indicator is maintained monthly basis,” and, this training “enabled me to support my organization’s commitment for CLA and for learning agenda establishment.” Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 9 Final Report Table 7: Summary of achievements by indicator for Output 2.1 Indicator Indicator Baseline Target Actual Percent of Target Reached Number of USAID Mission staff who received training Output 0 160 111 69% Number of USAID IP staff who received training Output 0 310 300 97% Number of IPs who received mentoring and/or coaching support on MEL or CLA topics Output 0 N/A 29 NA Mission staff training demand dropped after the onset of the COVID-19 pandemic, resulting in one of the few indicators on which EPMES fell notably below target. Trainings to USAID/Ethiopia Mission Staff EPMES’s training to Mission staff during the period of the Activity focused on three major areas: Data Quality Assessment; Monitoring, Evaluation, and Learning; and shorter trainings involving GIS. Thus, 66, 41, and four Mission staff were trained on DQA, GIS, and MEL, respectively. The training on DQA is expected to enhance the knowledge of AORs and CORs about data quality management and assessment and proper data quality management of its IPs. The trainings on GIS included ArcGIS, the Strategic Advisory Group for Emergencies (SAGE)/Emergency Shocks Dashboard, and the AGP Interactive Dashboard. The training on GIS-related topics was designed to help the Mission with respect to evidence-based decisions regarding t prioritization of needs and options for responses. Table 8 and Figure 4 show the summary of trainings provided to the Mission staff. Table 8: Summary of trainings to Mission Staff by training topic Training topic Target Actual Percent of Target Reached MEL 80 4 5% GIS 80 41 51% DQA 0 66 NA Total 160 111 69% Success case: CLA and knowledge sharing in Reading for Ethiopia’s Achievement Developed (READ)-II Following the training, the READ-II Activity staff indicated they were implementing more strategic collaboration and knowledge sharing with key stakeholders. According to READ-II, implemented by Creative Associates, the training helped to understand the need for CLA and helped with the systematic application of CLA tools with the GoE counterparts and local IPs. For this, READ-II: • Instituted pause and reflect sessions to help the Activity determine progress and results; • Synthesized existing evidence on education in Ethiopia to share with other stakeholders for evidence-based decision-making; and • Conducted a training for local IPs on MEL and USAID regulations. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 10 Final Report Figure 4: Percent of Mission staff trainees by subject Training to USAID Implementing Partners One of EPMES’s deliverables is capacity building of USAID IPs to improve their capacity in planning, implementing, and monitoring and evaluation of their Activities in line with the ADS. In this regard, EPMES has organized trainings that resulted in improved skills and knowledge of the IPs during the period of the Activity. The training that EPMES provided to USAID IPs focused on the fundamentals of M&E, CLA, KM, and DIS. EPMES had the capacity to undertake training on other subjects, which it undertook only during its final fiscal year based on a cursory needs assessment. The trainings on other subjects were on analysis of qualitative methods and agile and adaptive management in the USAID context. Out of the total of 287 IP staff working in 48 different organizations, 20 percent of the trainees were women during the period of the Activity, as indicated in Table 9. Table 9: Number of IP staff and others trained Fiscal Year Training Topic Number of Trainees (Male) Number of Trainees (Female) Total Training Modality 2018 MEL, CLA &KM 51 17 68 In-person 2019 MEL, CLA&KM 40 9 49 In-person 2020 MEL, CLA&KM 20 5 25 Virtual DIS 36 10 46 Virtual 0% 10% 20% 30% 40% 50% 60% MEL GIS DQA 4% 37% 59% Success case: Using MEL and collaboration for decision-making The Development Food Security Assistance (DFSA) Activity, implemented by CARE Ethiopia, reported the training helped improve the team’s use of performance data for decision-making. Following the training, the Activity increased the frequency of monitoring data reviews from annually to monthly. This also helped increase dissemination and engagement between the MEL staff, managers, and regional teams for decisions and course corrections. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 11 Final Report 2021 MEL, CLA&KM 55 13 68 Virtual 2022 MEL & CLA 12 1 13 Virtual Qualitative Data Analysis Methods 8 2 10 Hybrid (virtual and in person) Agile and Adaptive Management 8 0 8 In-person Total, USAID IPs 230 57 287 Due to COVID-19, trainings during the last quarter of 2020 and during 2021 were delivered to participants virtually. Figure 5: Fundamentals of MEL and CLA training (2019) In addition to the training topics mentioned above, EPMES also facilitated the following trainings: • Training on MEL, CLA, KM and Program Cycle management to 13 Ministry of Health and Regional Health Bureau staff; • Training of 126 enumerators on data collection ethics, interview procedures and sampling; • Evaluation Quality, Use, & Impact (EQUI®) training to 29 local consultants; • Business development; ADS; gender, equality, and social inclusion (GESI) training to 27 local consultants; and • CLA champions training. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 12 Final Report Figure 6: Discussion on “user stories” during Agile and Adaptive Management in the USAID Context training Post-training evaluation feedback revealed improvements in training participants’ understanding and knowledge of training topics. For example, the training evaluation results on DIS showed 77 percent of the trainees reporting they felt comfortable navigating the DIS after the training compared to 13 percent before the training. Number of IPs Who Received Mentoring and/or Coaching Support on MEL or CLA Topics The technical assistance of EPMES to USAID IPs was aimed at strengthening their capacities to implement, monitor and evaluate their Activities as per ADS regulations. To ensure that the technical support was demand-driven, EPMES conducted a technical assistance and LME capacity building support needs assessment in 2017. Based on this assessment and other consultations with the Mission, EPMES conducted trainings and technical assistance on MEL and CLA. This included providing support to 29 IPs in 2020 and 2021 focusing particularly on providing training materials and helping in accessing DIS to enable IPs to enter data and report in a timely manner. Output 2.2: Support the Mission to Manage the Planning and Processes of Strategic Monitoring, Evaluation, and Learning Under Output, 2.2, there was one indicator: number of learning, monitoring, and evaluation and/or CLA plans completed with EPMES support. The target and actuals for the indicator of this expected result are provided in Table 10. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 13 Final Report Table 10: Summary of achievements by indicator for Output 2.2 Indicator Indicator Type Baseline Target Actual Percent of Target Reached Number of LME and/or CLA plans completed with EPMES support Output 0 NA 2 NA EPMES supported the development of LME plans for two Activities – Highland Resilience and Ethiopian Youth Empowerment – during the period of the Activity. Additionally, EPMES provided considerable support to the Mission and USAID IPs on reviewing LME plans. This included conducting reviews in 2020 for LME plans of 26 Activities of which six were light reviews and 20 full reviews. EPMES also conducted two full reviews of activity LME plans in 2021 and 2022. Light reviews of the LME plan focused on reducing the number of indicators reported, while full reviews were a more thorough review of the ALMEP for compliance and learning priorities. The objectives of the plan reviews were to assess indicators for ADS 201 compliance and utility and to test and refine tools for AORs/CORs LME quality assurance. The LME plan reviews were conducted using rubric scoring criteria developed by EPMES and are presented in Table 11. Table 11: Number of LME plan reviews by project Primary Project Number of Activities Light review Full review Highlands Resilience 0 2 Lowlands Resilience 0 2 Economic Growth (Private Enterprise) 2 5 Ethiopian Youth Empowerment 0 1 Empowered Communities for Better Health (ECBH) 3 3 Health Systems Strengthening (HSS) 1 5 Education 0 1 Water, Sanitation, and Hygiene (WASH) 0 1 Governance 0 2 Total 6 22 A review of the LME plans showed notable results of which the major ones are summarized as follows: • There was variation in LME plan detail and organization across Activities. This increased the burden on AORs/CORs for oversight and quality assurance; • 30 percent of LME plans did not identify risks or assumptions which limited the ability to monitor and adapt as per the context; and • 40 percent of LME plans did not reference evidence in the ToC. The ToC may be based on faulty assumptions around causality or context. EPMES has provided the following additional support to USAID/Ethiopia regarding LME plans: • Prepared and shared a new ALMEP template for USAID; • Developed and shared an indicator priority and selection checklist tool; • Prepared and shared an indicator mapping tool and review rubric with associated video and written guidance; • Undertook an analysis of PIRS for multiple Activities; and • Conducted ALMEP slide deck presentation to USAID PRO team. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 14 Final Report Output 2.3: Increase Utilization of MEL Tools and Practices in the Program Cycle Table 12 below provides a summary of the performance indicators and achievements. Table 12: Summary of achievements by indicator for Output 2.3 Indicator Indicator Type Baseline Target Actual Percent of Target Reached Submit DQA dashboard and analysis to USAID/Ethiopia once per year Output 0 1 1 100% Submit draft site visit monitoring tool Output 0 1 1 100% EPMES not only met these two targets, but also helped the Mission concerning DQA. Thus, the salient achievements of EPMES in this regard are summarized as follows. Data Quality Assessment • The existing USAID DQA tool was revised to support the effective rating of data quality standards. • Digitized DQA checklists were used by USAID to simplify the capture, storage, and analysis of data quality trends. • Designed an Excel database and macro-enabled entry form to store DQA information and a live dashboard to summarize five key DQA findings and to identify priority gaps. • Conducted two meta data analyses: one in 2017 and the other in 2021. The meta-analysis conducted in 2017 covered six indicators of IPs under the ALT Office and DG Office. The analysis report was submitted to the Mission. • EPMES’s meta-analysis conducted in 2021 was much wider in scope, covering 18 indicators (55 percent of indicators from the health sector and the rest from governance, agriculture, and education sectors). According to the findings, all the 18 indicators received an overall rating of “acceptable” 1.5. or above on a scale of 1-2; differences across the standard parameters are shown in Figure 7. Figure 7: Overall rating of 18 indicators (2021 DQA) Data collection delays caused timeliness challenges due to COVID-19, poor infrastructure, and security￾related issues. Problems related to precision and validity were due to the lack of a common understanding of what data was to be counted for each indicator and differences in the estimation process. 0 0.5 1 1.5 2 Timeliess Precision Validity Reliability Integrity 1.56 1.71 1.76 1.85 1.89 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 15 Final Report Develop and Pilot Site Visit Reporting Tool EPMES has prepared and submitted the Excel version of the Field Visit Tool with user guidelines, site visit data base and comments and questions for the Mission. However, the practical use of the tool has been on hold by the Mission because of the need to use it with the new DIS application. Monitoring, Evaluation, and Learning Tool Kits EPMES revised, edited, and rebranded the MEL toolkits and submitted them to USAID. The toolkits were used to train IPs on MEL and the awareness creation efforts of Mission strategies toward LME. RF Sub-purpose 2-B: Increase Utilization of GIS Services for Planning and Decision-making Sub-purpose 2-B fits within Objective 2 of EPMES’s work plan. There were no established indicators to track and measure performance at this sub-purpose level. There were, however, indicators at the output levels and the achievements of the outputs as per the performance measurement indicators discussed in Table 13. Output 2.2.1: Produce Maps Table 13: Summary of achievements by indicator for Output 2.2.1 Indicator Indicator Type Baseline Target Actual Percent of Target Reached Number of static maps prepared Output 0 462 865 187% SAGE Dashboard/Map launched & used by Mission staff Output 0 1 1 100% As shown in Table 13 for the indicators under Output 2.2.1, 865 static maps were produced and submitted to the Mission to assist with Activity and Project design, geographic targeting and prioritization, planning, and communication and reporting efforts. In line with the indicators, the key achievements of this output are summarized below. • To enhance GIS utilization by technical offices and integrate the system into the Mission’s learning, planning, and decision-making efforts, EPMES developed different maps for the various offices of the Mission. EPMES’s sub-contractor, Information Systems Services (ISS- Ethiopia) has assisted the Activity in producing maps and other related deliverables. • A total of 865 maps were prepared during the lifetime of the Activity focusing on various areas like determining appropriate implementation sites for Projects and Activities by identifying locations of and needs related to infrastructure, settlement, food security hotspots, etc. The maps supported the Mission in the geographic targeting of key Projects and Strategies such as Highland and Lowland Resilience Projects, the Global Food Security Strategy/Feed the Future Zone of Influence, Malaria Operational Plan of the President’s Malaria Initiative, Development Objective (DO) 4, Priority Woredas Identification, and AGP Cities and Hinterlands. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 16 Final Report Figure 8: Number of maps produced by year The number of maps produced by fiscal year is provided in Figure 8. Demand was highest in the earlier years in line with the development of CDCS 2.0 and the development of Project/Activity designs that demanded a high and increased production of maps. Further, the decline in the number of static maps produced over the performance period was purposeful to shift from static to interactive maps, which can provide richer and synergetic information. Technical offices utilized the maps to support planning, project designs, reporting, and other endeavors of the Mission. Maps delivered to the Mission technical offices and their utilization also triggered additional mapping support requests from the Mission. Figure 9 depicts the number of maps prepared for the technical offices, Resilience Projects, and evaluations carried out by EPMES. Figure 9: Number of maps produced for each Mission technical office 0 100 200 300 400 500 600 700 800 900 2017 2018 2019 2020 2021 2022 Total 375 248 124 77 41 0 865 1% 19% 2% 25% 7% 13% 27% 6% DG EG&T EY Health Program R2 Resilience Project Evaluation Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 17 Final Report As shown in Figure 9, the largest number of maps were produced for the R2 and Health Offices, followed by the Economic Growth and Transformation (EG&T) Offices. A smaller number of maps were produced for the DG and Education and Youth Offices. Strategic Advisory Group for Emergencies (SAGE)/Emergency Shocks Dashboard Developed and Updated • EPMES designed and developed an interactive map and dashboard for SAGE to strengthen the Mission’s emergency response decision-making. The interactive SAGE Map and Dashboard (later, rebranded informally as the Emergency Shocks Dashboard) displays the most recent information on the geographic presence of USAID/Ethiopia’s programs, humanitarian assistance activities, crisis-modifiers, woreda hotspot classification and need severity category, and shocks (including conflict, flood, measles, cholera, malaria, and locust infestation). • The interactive SAGE/Emergency Shocks Dashboard allows users to geographically filter based on user-defined criteria and add additional external data to discern spatial relationships between different variables. • EPMES gathered and organized available data on the type and location of major infrastructures such as industrial parks, schools, and hospitals and uploaded the data into the AGP Interactive Maps and Dashboard to explore potential geographic synergies between the GoE and USAID investments. • EPMES continuously updated layers on the dashboard when new data was obtained from the Mission or other external credible data sources such as United Nations Office for Coordinating Humanitarian Affairs (UN OCHA), Famine and Early Warning Systems Network, and Food and Agriculture Organization Locust Hub. • In addition to the SAGE/Emergency Shocks Dashboard, EPMES developed two additional interactive dashboards for the Secondary Cities Assessment and Alternative Growth Poles Mapping. The Secondary Cities Assessment dashboard provides a list and description of factors used to score cities against seven major indicators, calculated scores of 27 cities, locations of the cities, final secondary cities and city clusters, and other additional information. The Alternative Growth Poles Dashboard maps implementation woredas and thematic areas of 54 active Activities and implementation sites of 65 past USAID Activities, location of some facilities constructed IPs using USAID fund, and some major infrastructure. Output 2.2.2: Produce USAID Activities GIS Database and Map Table 14: Summary of achievements by indicator for Output 2.2.2 Indicator Indicator Type Baseline Target Actual Percent of Target Reached USAID Activities GIS database produced and updated as needed Output 0 1 1 100% EPMES established a geodatabase for GIS data, maps, map templates, and raw data that were used for GIS analysis and mapping. All georeferenced data received from the Mission and obtained from other external sources such as government offices, online portals, published databases and reports are stored in the geodatabase. Datasets collected for assessments and used to develop maps, dashboards, and tabular data were made part of the geodatabase. The geodatabase comprised both spatial and non￾spatial data on administrative boundaries, agriculture, climate, conflict, food security and nutrition, livelihoods, socioeconomics, health, infrastructure, and hydrology. EPMES created metadata for basic datasets such as administrative boundaries, agriculture, and major infrastructures such as health facilities, schools, and road networks. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 18 Final Report In line with this output, EPMES assisted the Mission to systematically gather, organize, and update implementation woredas of its Activities. EPMES, in consultation with the Mission, created a shared Google spreadsheet-based Master Woreda/Activities List (Activities location data) that related Activities and their respective implementation woredas. The spreadsheet contained additional details such as Activities start and end date, IP name, award number, name of the AOR/COR of the Activities, Activity status, thematic areas of the Activities, and implementation levels. EPMES updated the spreadsheet as frequently as possible when new Activities were awarded and further information on existing Activities became available. The Activities location data was primarily used to produce Activities location maps and identify common implementation woredas of different Activities to support geographic targeting and prioritization as part of Project/Activity design and strategy development efforts. EPMES also created an Excel database of Activities that allowed filtering of Activities by thematic area, technical offices, implementation level, and Activity start and end date. Having a unified Activities data management system enhanced the experience of using such data for different planning and reporting endeavors at the Mission. Key Achievements • EPMES gathered, organized, and analyzed a wide variety of spatial data to explore trends, patterns, and spatial relationships that created a better understanding of variables relevant to technical offices’ planning and decision-making endeavors. • More than 865 thematic maps were produced for technical offices, working groups, evaluations, etc. The maps revealed implementation sites of Activities and geographic overlap among the Activities, and spatial distribution of different geographic phenomena such as the number of internally displaced persons (IDPs), schools, health facilities, and climatic variables. The maps facilitated discussions, enhanced the application of spatial information, and provided the geographic context of the country for planning and decision-making. • EPMES introduced the application of state-of-the-art user-friendly interactive dashboards and maps that provided rich information on emergency shocks, the geographic presence of USAID Activities, and major infrastructure, to name a few. These platforms provide tools that enable mapping, filtering, exporting data and maps, and adding external data to overlay and explore patterns, and spatial relationships. • The EPMES team assisted the Mission in gradually shifting from static maps to interactive web mapping application, which is state-of-the-art in the geospatial field, to increase GIS service demand at the Mission to integrate spatial thinking in Project Activity design, geographic prioritization, and planning efforts. Objective 3: RF Sub-purpose 3: Improve Integration of CLA in USAID/Ethiopia Programs and Processes This sub-purpose, aligned with Objective 3 of the EPMES work plan, was designed to support the Mission to intentionally integrate CLA with programming and to build and strengthen a culture of learning. Table 15 summarizes the established indicator and progress. Table 15: Summary of indicators for Sub-purpose 3 Indicator Indicator Type Baseline Target Actual Percent of Target Reached Number of programs and processes EPMES supports to enhance CLA practices in USAID/Ethiopia Sub￾purpose 0 12 9 75% Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 19 Final Report As provided above, EPMES was able to support nine projects and processes to enhance CLA practices in the USAID/Ethiopia Office. The implementation of the target was based on the demand of USAID and its IPs to enhance CLA practices; hence, this target was not achieved. Output 3.1: Support the Mission with CLA Integration in the Implementation of CDCS 2.0 The objective of this sub-purpose was to support the Mission to intentionally integrate CLA into its strategies. The key achievements under this output contributed to USAID/Ethiopia’s strategic planning process and organizational learning priorities. Table 16 shows the indicator for Output 3.1 and actuals relative to target. Table 16: Summary of indicators for Output 3.1 Indicator Indicator Type Baseline Target Actual Percent of Target Reached Number of MEL and/or CLA plans completed with EPMES support Output 0 9 2 22% As the development of CLA plans with EPMES support was demand driven and since the demand was low, this was among the weakest indicators for EPMES achieving its targets. To the extent that A/CORs deem their Activities’ plans sufficient and that they do not require support, this might reflect positively on the Mission’s portfolio. However, this equally may reflect insufficient awareness of the value of a careful review and support for the development of such plans. The major support that EPMES rendered to the Mission in this regard was the analytical and facilitation services provided for the development of USAID’s CDCS for the period 2019-2024. EPMES also played a pivotal role in the CDCS Closeout Analysis, which was aimed to document the achievements, challenges, and lessons learned from the implementation of USAID/Ethiopia’s 2011-2019 CDCS. The key achievements are highlighted as follows. CDCS 2.0 Support • EPMES effectively facilitated USAID/Ethiopia’s first Mission-wide CDCS 2.0 kick-off meeting retreat. The retreat helped the Mission reflect on its past strategy, assess the country's context, identify, and prioritize critical problems, and outline the next steps to develop the new CDCS. • The Ethiopia Trends Analysis, subcontracted to ISS-Africa and the Frederick S. Pardee Center for International Futures, provided an analytical base to inform the development of USAID/Ethiopia’s next-generation CDCS 2.0. • The development of the 2019-2024 CDCS involved a series of CDCS champion meetings, discussions, and consultative workshops at national and regional levels. The Mission requested that EPMES run the workshops and provide technical support. Thus, EPMES enabled the Mission to leverage these workshops in Addis Ababa and six regional states toward a more geographically nuanced, contextually relevant, and collaborative strategy. • To support socialization and implementation of the strategy, EPMES also supported the Mission to organize the CDCS 2.0 Key Analysis and Regional Consultation Findings Dissemination Workshop, followed by a two-week RF development workshop. • To advance and incorporate CLA in the new CDCS, EPMES provided comments and content to the LME section of the Mission’s RF paper. EPMES’s contribution enhanced language around USAID/Ethiopia’s vision for CLA, expanded readers’ understanding of the role of LME, and described the process of developing the Mission’s learning agenda with its identified focal areas. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 20 Final Report • EPMES initiated a vital strategic learning, accountability, and adaptive management exercise in the form of a CDCS closeout analysis. This task was undertaken to document the achievements, challenges, and lessons learned from implementing the Mission’s 2011-2018 strategy. • A significant contribution by EPMES was the preparation and use of a systematic process for developing a learning agenda. This guidance involved facilitated support to ToC validation, a review of current learning resources and knowledge gaps, and developing a template for outlining DO/technical team learning needs. Figure 10 below exhibits the resulting RF developed and included in the 2019-2024 CDCS. Figure 10: USAID/Ethiopia’s CDCS Results Framework (2019-2024) Output 3.2: Support MEL and CLA Capacity Building Support for Adaptive Management and Integration Table 17: Summary of indicators for Output 3.2 Indicator Indicator Type Baseline Target Actual Percent of Target Reached Number of IPs that received MEL and CLA capacity building support for CLA Output NA NA 44 NA The performance of this result is reported under Output 2.1. EPMES organized distinct CLA and KM training modules within the M&E and project cycle management trainings. EPMES trained 223 IPs (20 percent female) and several government staff during the period of the Activity. Participants from USAID IPs accounted for 94 percent of the total trainees, while the remaining six percent were from the Ministry of Health. Regarding the number of IPs addressed through the MEL and CLA capacity building initiative, EPMES reached a total of 44 IPs during its lifetime. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 21 Final Report CLA Mapping and Action Planning Exercise In 2021, EPMES deployed a Mission-wide survey to gather evidence on the current status of CLA practices and enabling conditions. This survey data informed CLA Mapping and Action Planning Sessions that EPMES facilitated with each DO team and PRO to collectively plan for better institutionalization of CLA practices and enabling conditions across the Mission and with partners. EPMES then used the data, findings, and analysis from the CLA Survey and CLA Action Planning Sessions to help the Mission update relevant sections of the CDCS CLA Plan and the Leadership Charter, ensuring that the Mission’s strategies are adaptively managed. To aid the action planning process, EPMES developed an Action Planning “Prototype” slide deck and a three-page summary of positive CLA examples collected from the CLA Mapping survey data. One of the priorities that emerged across action plans was to improve the use of adaptive management. In line with this, EPMES developed and provided training for IPs on agile management (also discussed under Output 2.1). EPMES supported the Mission and its IPs for the CLA case competition each year. EPMES also supported the Mission’s CLA Plan and Mission order review. Gender Champions Network This Network was established to support USAID IPs in sharing information and best practices and discussing problems and challenges integrating gender into their Activities. EPMES spearheaded the Gender Champions Network (GCN) in collaboration with the Mission’s Gender Specialist and members of the CLA team and provided strategic direction and coordination. EPMES also supported the GCN work plan preparation, provided training on CLA for the GCN members, and substantially contributed to the production of the GCN fact sheet. As a result, the regular meetings of the network continued across all years. This helped the Mission closely follow the IPs’ efforts and the implementation of the GCN’s CLA Action Plan. Youth Issues Youth is a priority agenda for USAID, and to help programming be more responsive, EPMES conducted a CLA case study on the EYO’s Regional Education Specialists (RES) model. This case study was the first of its kind and served as an essential milestone in the EYO’s strategic collaboration, program planning, and adaptive management efforts. Organizing Learning Events EPMES supported USAID in organizing two Gender Learning workshops in 2018 and 2022. EPMES also undertook a final closeout learning event in 2022. Gender and Youth Learning Event. In 2018, USAID/Ethiopia, in collaboration with EPMES and assistance from the Transform: Monitoring, Evaluation, Learning, and Adapting (MELA) Activity, organized a Gender and Youth Learning Event to foster learning and adapting within and across programs. The Mission and its IPs participated in a day-long event to explore the evidence and good Figure 11: Poster from the Gender & Youth Learning Event Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 22 Final Report practices in gender and youth-focused interventions. The program included presentations, poster displays, networking, small group discussions, peer learning, and action planning. Virtual Learning Workshop on Gender. EPMES prepared two Gender Learning workshops in 2022 with the objective to increase gender integration skills in a participatory and engaging manner, specifically targeting staff that worked at the activity level: A/CORs, Activity Managers, Gender Points of Contact (POC), OAA staff, and LME Officers. To achieve workshop objectives, EPMES engaged USAID staff in interactive, engaging, and relevant activities, presentations, and discussion topics using best practices for adult learning with USAID/Ethiopia’s Gender Specialist and learning team. This included staff assessing their activities along the gender integration continuum (Figure 1212). EPMES delivered the training in two virtual sessions to increase knowledge and provide an opportunity to discuss and share insights and experiences with colleagues across technical offices. EPMES also produced a substantial report based on the proceedings and Mission interactions. Final Closeout Learning Event. EPMES undertook a closeout event to share lessons and celebrate EPMES’s and the Mission’s achievements working together over the six-year period of performance. The event showcased key achievements through interactive discussions in ‘marketplace’ stations and discussions in multiple formats about critical lessons learned and opportunities for the Mission moving forward. The event was attended by the Deputy Mission Director, most of PRO including the Office Director, and representatives from other offices. Facilitation and Related CLA Support Support for Interagency Strategy Meeting. EPMES provided support to develop a strategy for the Ambassador’s Strategic Planning workshop, which aimed to clarify the U.S. Ambassador’s priorities for Ethiopia, develop a cohesive and actionable plan for the Mission, and identify areas for improving internal communication and collaboration. Incorporating CLA into OAA Instruments. In consultation with USAID/Ethiopia OAA, EPMES drafted standard CLA language for USAID/Ethiopia Requests for Proposals and award modifications. This revised CLA guidance was shared with technical offices and the Mission Management Team for endorsement. This revision to the CLA section of USAID/Ethiopia-issued contracts and agreements streamlined communication of expectations, and more intentionally map investments and accountability for CLA and its contribution to development results. Youth Project Design Support. To operationalize its highly integrated CDCS, USAID/Ethiopia designed a cross-sectoral Youth Project. In FY 2019, EPMES supported this process by facilitating design team meetings, conducting a rapid synthesis of evidence on youth in Ethiopia, and providing as-needed “Once again, thank you Marc, Alysia and the EPMES team for producing this excellent report!!” – Meseret Kassa, USAID/Ethiopia Gender Specialist Figure 12: Output of a participatory exercise during 2022 virtual Gender Learning Workshop Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 23 Final Report technical consultations between meetings. This enabled the Mission to coalesce around a draft problem statement for the ToC workshop. Facilitation of an EYO Retreat. EPMES assisted the EYO in developing their strategic planning retreat agenda and facilitated the workshop that took place on May 9, 2019, in Bahir Dar. The retreat was designed to establish the EYO core vision statement; enhance team effectiveness; and build an understanding of the new DO4 design under the new CDCS. These objectives were fully met, and a draft EYO vision was developed during the one-day retreat. Verbal feedback from the EYO reflected a decidedly successful and well-facilitated event. Public Affairs Workshop Facilitation. EPMES also facilitated a strategic planning exercise for the PAS of the Department of State on June 7, 2019, assisting the team in aligning their work plan with strategies and available resources. EPMES provided facilitation and coordination support for the planned PAS team workshop. The retreat enabled a space for the PAS team to ask questions and reflect on the implementation of ongoing initiatives, review their portfolio against Embassy priorities, and consider resourcing for the remainder of the fiscal year. EPMES shared notes from the workshop and received positive feedback about the quality of the event and facilitation. Food Security IP Meeting. EPMES participated in the EG&T quarterly food security partners’ meeting on October 24, 2018. The focus of the meeting was on how to improve communication processes and messaging related to program achievements. IPs also shared examples of policy support and CLA initiatives. EPMES reaffirmed its support to EG&T through assisting with planning for future partner meetings and contributing content (e.g., application of GIS, identifying approaches to adaptation, and helping to link KM with development and outreach communications activities). Digital Financial Solutions Workshop. The EG&T office, in partnership with Strategic Impact Advisors, hosted a two-day workshop on DFS. EPMES attended the workshop, facilitated a session on developing a DFS learning agenda, and participated in a follow-up debrief with the Mission to discuss the next steps in implementing and documenting lessons on the transition to digital payment modalities. EPMES also designed and facilitated the Department of State’s PAS retreat on October 25, 2019; the Health Office team retreat, with a focus on team building and planning, from January 22-24, 2020; and the U.S. President's Emergency Plan for AIDS Relief (PEPFAR) teambuilding retreat on January 27, 2020. Output 3.3: Implement an Effective System for Knowledge Management The objective of this output is to enhance the knowledge management system of the Mission by creating a knowledge sharing and management platform. The relevant indicator is the number of documents uploaded to the KM Catalogue, and EPMES uploaded 115 documents, representing 124 percent of the target. Table 18: Summary of indicators for Output 3.3 Indicator Indicator Type Baseline Target Actual Percent of Target Reached Number of documents uploaded to the KM Catalogue Output 0 71 115 162 To this end, EPMES achieved the following results during the Activity period of implementation: • To improve the effectiveness of USAID/Ethiopia’s KM system, EPMES conducted an assessment in January 2019 and proposed system changes. However, further work on a KM system was discontinued due to the roll-out of the DIS. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 24 Final Report • EPMES created a dedicated a centralized KM Catalogue based on Google Sheets to share various learning resources like evaluations, assessments, policies and guidelines, lessons learned and best practices, technical briefs, and templates. The Catalogue included key search terms, descriptions of key findings, and links to the original document. Google Sheets does not provide tracker functionality to track the Catalogue’s use. A total of 115 different knowledge sharing products (63 studies/assessments, 23 performance evaluations, 23 strategy documents, four reports, one data set, and one lessons learned) were added and shared on the Catalogue during 2020-2022. Figure 13 displays the number of documents uploaded to the system during the Activity’s lifetime. Figure 13: Documents uploaded to the KM Catalogue Output 3.4: Identify and Synthesize Evidence to Foster Adaptation Table 19: Summary of indicators for Output 3.4 Indicator Indicator Type Baseline Target Actual Percent of Target Reached Number of learning products (using secondary data only: synthesis, secondary document/dataset reviews/analysis) developed and shared with USAID to inform programming Output 0 9 16 178% The objective of this output was to share additional sources of information and learning products beyond the KM Catalogue. The identified sources include findings and recommendations from evaluations, syntheses of literature and evidence on topical areas, and technical briefs of different development issues. Although there were limitations in properly documenting and reporting on the Studies/Assessments 55% Peformance Evaluation 20% Strategy 20% Reports 3% Lessons Learned 1% Data Set 1% Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 25 Final Report number of learning products, EPMES synthesized and shared evaluation and assessment briefs, submitted multiple training briefs on LME and CLA, and documented and shared lessons learned and best practices notes, among others. Objective 4: RF Sub-purpose 4: Strengthen Technical & Management Capacity of Local M&E Service Providers This sub-purpose, aligned with Objective 4 of the EPMES work plan, aimed to strengthen local M&E service providers in the country to provide quality M&E services to USAID and its IPs, GoE, and other stakeholders. The result also anticipated developing a host country M&E system capacity strengthening strategy document to build a functional and sustainable national M&E system to meet the M&E requirements of the government, donor groups, and other actors in the development arena. The performance measurement indicator for this sub-purpose is provided below. Table 20: Summary of indicators for Sub-purpose 4 Indicator Indicator Type Baseline Target Actual Percent of Target Reached Number of identified M&E LSPs who improved technical and management capacity as evidenced by TOCAT Sub￾purpose 0 5 5 100% % of United States Government (USG)- assisted organizations with improved performance (implementing mechanism [IM￾level], Mission Past Performance Review (PPR) indicator, CBLD-9) Sub￾purpose 0 4 5 125% EPMES was successful in meeting or exceeding its targets, as it was possible to work with and build the capacity of five LSPs during the period of the activity. The capacity building process went through a very long process of application review, interviews, and direct observation. Although 93 local firms have applied for the support, only six were chosen, and five LSPs participated in the full duration of the capacity strengthening initiative. EPMES conducted a TOCAT in June 2020 to measure capacity improvements of LSPs after its support. The assessment covered five LSPs (Dadimos Development Consultancy PLC, JaRco Consulting PLC, Laterite Consulting Limited, REBERT Business and Consultancy PLC, and TAK Innovative Research and Development Institute). The TOCAT, adapted from USAID’s Organizational Capacity Assessment Tool, was framed around two main capacity areas: organizational capacity and technical capacity. Organizational capacity consisted of four components: governance, structure, and strategic management; human resources (HR) management; financial management; and knowledge management. Technical capacity comprised two dimensions: LME technical capacity, and gender issues in LME. The rating system applied in the TOCAT used a four-point capacity-rating scale. • The results of the assessment revealed that the two LSPs (REBERT and TAK) that started at Baseline with scores in the “Nascent/Low” range to an improvement at the “Emerging” range by the Endline scores. The three remaining LSPs, which started at a slightly higher level of assessed Baseline capacity (“Emerging”) in terms of organizational capacity, made more modest gains, as shown in the figure below. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 26 Final Report • The EPMES Activity observed a modest improvement in LME capacity between Baseline and Endline. This was likely because the training targeted long-term staff while – as was subsequently learned – the majority of the LSP firms’ LME tasks were conducted by short-term consultants, who were not the focus of the EPMES capacity-building efforts. Figure 14: Baseline and endline TOCAT scores for capacity building of M&E local service providers A separate assessment was not conducted regarding the second indicator for this sub-purpose (percent of USG-assisted organizations with improved performance). However, improvements in the organizational capacity of LSPs, as demonstrated by TOCAT findings, suggested improvement in the organizational performances of the five LSPs covered by the capacity assessment. Output 4.1: Roster of Local M&E Service Providers Maintained Table 21: Summary of indicators for Output 4.1 Indicator Indicator Type Baseline Target Actual Percent of Target Reached Number of M&E LSPs registered in the M&E service providers’ roster Output 0 774 702* 91 *Total number of LSPs registered each year (as the same LSP may register every year, the indicator definition allows double counting) The objective of this output was to create a roster of M&E service providers to ensure the most current information on the number, capacity, experiences, and activities of the LSPs. To this end, EPMES continued updating the roster of LSPs every year for capacity building and facilitation of their activities. Thus, salient achievements attained in this output are provided as follows: • EPMES developed an access-based roster to capture basic information about the service providers like basic company profile information, professional and sectoral experience, and other pertinent information. The Access-based format was later changed to Excel format upon the request of the AOR. • As indicated in Table 21, EPMES registered 702 LSPs in the roster throughout the time of the Activity (with the possibility of double-counting the same IP that may have registered every year). The number of service providers registered in each fiscal year is shown in Figure 15. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 27 Final Report Figure 15: Number of registered local service providers Output 4.2: LSPs Capacity Building Assistance Aligned with USAID Requirements Table 22: Summary of indicators for Output 4.2 Indicator Indicator Type Baseline Target Actual Percent of Target Reached Number of LSP staff that received M&E capacity strengthening support Output 0 78 52* 67% * Note that the same staff member may be eligible for and receive different support. As noted in the PIRS, this indicator thus represents staff person trainings rather than individuals having received any trainings. The purpose of this output was to strengthen the capacity of LSPs through training, mentoring, and coaching so that they can provide quality and standard services to their clients. The indicator showed that 52 LSP staff received capacity strengthening support, representing an achievement of 67 percent of the target. As a first step for the capacity building efforts, EPMES, in collaboration with its subcontractor PRIN International Consultancy & Research Services (PRIN), conducted the capacity assessment of the firms, looking at both the technical and organizational capacities. Based on the capacity assessment, a capacity strengthening action plan was developed and appropriate capacity strengthening training packages including training manuals, facilitator guides, PowerPoint presentations, practicums, and more were developed. The key achievements are summarized as follows: • Training on EQUI® was provided to 17 LSPs; • Training on USAID’s ADS and GESI was provided to nine LSPs; • Evaluation design, management, and CLA training was provided to 12 LSPs; • Strategic planning and governance, HR, contracts, and financial management training was provided to nine LSPs; and • Training on business development practice was provided virtually to nine LSPs. 0 100 200 300 400 500 600 700 800 2017 2018 2019 2020 2021 2022 Total 0 114 200 200 250 10 774 150 62 161 221 19 89 702 Target Achievement Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 28 Final Report Output 4.3: Country-level Strategy/Roadmap for M&E Local Capacity Development Table 23: Summary of indicators for Output 4.2 Indicator Indicator Type Baseline Target Actual Percent of Target Reached Number of stakeholders who participated in the development of the country level roadmap Output 0 51 25 49% Country-level roadmap for M&E local capacity finalized Output 0 1 1 100% This output aimed to develop a host country M&E system capacity strengthening strategy document that builds a functional and sustainable national M&E system to sufficiently meet the M&E requirements of the government, donor groups, and other actors in the development arena. The stakeholders’ workshop for the final validation of the draft strategy was placed and left on hold due to COVID-19. However, the following key activities were accomplished during the period of performance, leading to acceptance of the deliverable as complete by USAID. To better understand and meet USAID’s expectations, the EPMES/PRIN team developed and presented a strategy development framework outlining the purpose of the strategy and roadmap, identifying the process to be followed, itemizing the information needed, and pre-selecting the key stakeholders to be involved in the process along with the tentative timeline. EPMES and PRIN produced and submitted a host-country M&E capacity strengthening roadmap to USAID, providing an action plan to help engage and encourage development actors to further enhance the local M&E capacity in Ethiopia. EPMES incorporated USAID and the Federal Ministry of Planning and Development (formerly known as the Planning and Development Commission) inputs and feedback. Although the COR approved the report, the presentation and discussion with GoE officials on the document were interrupted because of COVID-19 restrictions. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 29 Final Report While the USAID/Ethiopia and SI teams have faced a wide array of challenges from 2016 to 2022, EPMES leadership has worked closely with the COR over the years to ensure that the implementation of EPMES has progressed smoothly. Below is a summary of management highlights of the EPMES contract, including overviews of contract start-up, challenges faced and successful adaptations, contract modifications, subcontracting activity, and closeout. Start-up Despite several challenges at the time of the contract award, the SI team established operations in Addis Ababa quickly and effectively. Following the contract start date of April 4, 2016, SI mobilized a rapid response team consisting of several members of its Business Operations & Security Services (BOSS) and Program Management Unit (PMU) teams and an Acting COP to Addis Ababa in May and June to: • Recruit, hire, and onboard EPMES technical and operational staff; • Contract subcontractors ISS-Ethiopia, Sub Saharan Africa Research and Training Center (SART), and PRIN; • Arrange to begin work from a temporary office space at the Hilton Addis Ababa; • Identify, contract, and prepare a permanent office space near Bole International Airport; • Develop and institute financial, procurement, security and personnel policies and procedures in compliance with Ethiopian law; • Procure essential equipment and furniture; • Initiate the process for opening project bank accounts; and • Complete registration as a business in Ethiopia. Ultimately, SI completed start-up of the EPMES Activity within three months. Eighty percent of our Ethiopian staff were hired and working within two and a half months of the contract award; the EPMES team was working from its permanent office space in less than three months; SI completed its registration as a business in Ethiopia in just over three months; and EPMES initiated six evaluations, studies, or analyses within its first year of implementation. Challenges and Successful Adaptations The EPMES team encountered and adapted to various challenges throughout its over six years of implementation. With the support of SI’s PMU, EPMES applied best practices wherever possible to minimize and mitigate technical, financial, security, and administrative risks. Among these best practices are: • Maintaining frequent and open communication with USAID/Ethiopia, allowing EPMES to understand and adapt to Mission needs when necessary; • Applying cost control measures and judicious spending habits, which ultimately allowed EPMES to stretch its contract budget from the initially anticipated five-year POP to one of six years, two months; • Creating an EPMES Staff Compensation Plan to standardize compensatory practices transparently, enhance pay equity throughout the team, and ensure our team members are offered competitive salaries; • Denominating staff salaries in USD (paid in birr), with USAID approval, to reduce the impact of volatile currency fluctuations on our team; • Developing and adhering to strict policies and procedures to reduce the risks of fraud and mismanagement; MANAGEMENT HIGHLIGHTS Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 30 Final Report • Developing and monitoring various tools to track and report on essential project information and ensure contract compliance, such as a contract deliverable tracker, detailed budget trackers, and vendor information trackers. Below is a table summarizing several of the challenges faced and the EPMES team’s adaptations in response. Table 24: Challenges and EPMES's adaptations Challenge Adaptations Significant technical priority changes stemming from a change in USAID/Ethiopia’s needs EPMES worked closely with the COR and Mission leadership to understand and adapt to Mission needs. EPMES adjusted its staffing structure and realigned its contract budget to position itself to better serve as USAID/Ethiopia’s learning partner. Bypassing of the GIS protocol during task requests Although USAID and EPMES established a protocol for requesting GIS support, not all requests followed the protocol. For better tracking, documentation, management, and planning of support, EPMES instituted a digital system for all GIS support requests that follow a standard protocol, which has successfully been used across the Mission. Poor subcontractor performance When a subcontractor was not meeting its contractual obligations, SI issued a Notice of Default to the partner containing curative measures, de-scoped the subcontract and transferred responsibilities in-house where possible, and closely monitored the subcontractor’s performance and delivery until the assignment was completed to USAID’s satisfaction. Key personnel transitions, as summarized below EPMES and SI leadership maintained frequent and open communication with USAID to ensure consistency in service delivery. When applicable, SI identified interim key personnel to avoid gaps in coverage. Delay in incremental funding available from USAID In anticipation of a potential stop work order, EPMES slowed down its non-essential technical activities and discretionary spending to the greatest extent possible and prepared for the possibility of a temporary shutdown. Through these measures, EPMES was able to continue operating within its funded obligation and avoid a shutdown until USAID had the funding necessary to increase EPMES’s funded obligation. Onset of the COVID-19 pandemic See subsection below. Civil unrest throughout Ethiopia at varying points in implementation, including the advancement of the Tigray People’s Liberation Front towards Addis Ababa, limiting USAID’s and EPMES’s ability to conduct in-person work outside the capital. EPMES applied its established security protocols such as a phone tree, increased communication with SI’s safety and security team and USAID/Ethiopia’s PLSO, equipped and encouraged staff to work from home when necessary, and in November 2021, evacuated its COP from Ethiopia following an Authorized Evacuation Order from the Mission. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 31 Final Report COVID-19 USAID/Ethiopia and EPMES faced a series of new challenges due to the onset of the COVID-19 pandemic in Ethiopia and around the world. EPMES responded to the pandemic by equipping staff to work effectively from home and altering its approach to technical activities to apply remote or distanced practices wherever possible. For instance, EPMES adapted its data collection approach for the Sustaining Dialogue (SD) on Ethiopian Campuses Evaluation to include a remote Computer-Assisted Telephone Interviewing (CATI) survey, and EPMES successfully planned and delivered remote learning events such as the Fundamentals of MEL and CLA Training for USAID IPs. Further, SI sought OAA approval to purchase personal protective equipment for the EPMES office and vehicles to ensure that staff who did need to do in-person work could do so safely. SI’s operations team and Institutional Review Board (IRB) established procedures to ensure that any travel or in-person activities were essential and conducted safely. At the onset of the pandemic, SI evacuated its Sr. Director of Learning, Monitoring, & Training from Addis Ababa and delayed the mobilization of its Chief of Party to minimize their risk of exposure to COVID-19. Throughout the pandemic, in coordination with SI-HQ and PLSO, the EPMES Operations team continually monitored developments related to COVID-19 and prepared to re-open the EPMES office fully or partially when possible. However, due to generally high infection rates and low vaccination rates in Ethiopia, a resumption of full-time in-person work was never deemed safe. Contract Modifications Table 25 summarizes the 20 modifications USAID/Ethiopia issued to the EPMES contract over the course of implementation. Table 25: EPMES contract modifications Date Issued Mod # Purpose September 6, 2016 1 • Update the language under Sections C.7, D.2, F.5, F.6, F.9, and H.19 • Update the budget table (B.4) • Update with the new COP and DCOP names (F.8) November 15, 2016 2 • Increase the obligated funding by $2,100,000 (B.3) • Delegate some of the CO’s roles to the COR (H.10) • Allow 6-day work week with premium pay (F.9) April 27, 2017 3 Increase the obligated funding by $1,992,000 (B.3) October 20, 2017 4 Increase the obligated funding by $1,195,000 (B.3) November 14, 2017 5 • Realign the EPMES contract budget (B.4) • Reduce the contract POP by one year, from five years to four (F.2) • Update the language in C.2 • Update with the new key personnel (KP) names (F.8) March 5, 2018 6 Update with the new KP names (F.8) March 15, 2018 7 Realign the EPMES contract budget (B.4) May 4, 2018 8 • Update the obligated amount to $6,837,000 (B.3) • Realign the EPMES contract budget (B.4) • Update the language in Sections B.5 and F.9 May 22, 2018 9 Increase the obligated funding by $2,113,000 (B.3) January 16, 2019 10 • Increase the obligated funding by $2,361,687 (B.3) • Add a new contract clause (B.9) Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 32 Final Report Date Issued Mod # Purpose • Correct an internal error in GLAAS (B.9) March 15, 2019 11 • Update with the new KP name (F.8) • Extend the delivery date for Deliverable No. 34 (F.5) April 18, 2019 12 Extend the EPMES POP end date from April 3, 2020 to June 3, 2020 (F.2) October 11, 2019 13 • Increase the obligated funding by $900,000 (B.3) • Add appropriation data (B.9) April 6, 2020 14 • Increase the obligated funding by $3,000,000 (B.3) • Add appropriation data (B.9) • Adjust the SOW (C.) • Extend the POP end date from June 3, 2020 to June 3, 2021 (F.2.) • Update with the new CoP name (F.8) August 11, 2020 15 • Extend the POP end date from June 3, 2021 to June 3, 2022 (F.2) • Rename a KP position title (F.8) December 10, 2020 16 • Realign the EPMES contract budget (B.4) • Update the Negotiated Indirect Cost Rate Agreement (NICRA) (B.5) • Update the language under Section (H.10) March 30, 2021 17 Update with the new KP name (F.8) July 23, 2021 18 Increase the obligated funding by $500,000 (B.3) December 13, 2021 19 • Increase the obligated funding to nearly its full obligation • Add appropriation data • Update with the new CoP another KP names (F.8) February 11, 2022 20 • Realign the EPMES contract budget (B.4) • Update the NICRA (B.5) Period of Performance Adjustments Over the course of implementation, USAID adjusted the contractual LOP for the EPMES Activity five times. 1. Five years: as specified in the initial contract signed on April 4, 2016 2. Four years: USAID reduced the LOP by one year in November 2017 3. Four years, two months: USAID increased the LOP by two months in April 2019 4. Five years, two months: USAID increased the LOP by one year in April 2020 5. Six years, two months: USAID increased the LOP by one year in August 2020 EPMES’s final LOP: April 4, 2016 through June 3, 2022. Summary of Key Personnel Upon notification of the EPMES contract award to SI, the candidates for COP and DCOP each rescinded their availability for their respective roles. In response, SI immediately identified and mobilized a Senior Technical Director from SI-HQ to serve as Acting COP for the start-up of EPMES. SI also immediately began re-recruitment for both full-time positions and successfully identified and mobilized a new DCOP by August 2016 and a new COP by September 2016, once USAID/Ethiopia approved each candidate. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 33 Final Report In late 2017, in consultation with the COR, EPMES proposed the creation of a new key personnel position, Training & Communication Director. Following USAID’s approval of the new position, SI identified and mobilized a candidate to Addis Ababa by April 2018. After a few years of implementation, to better reflect the responsibilities of the role, EPMES sought and received USAID approval to rename the position to Sr. Director of Learning, Monitoring, and Training. Upon the resignation of EPMES’s long-time Evaluation Methods Specialist in 2020, in consultation with the COR, SI decided not to backfill the position with a full-time staff member. Instead, it was agreed that the responsibilities of the role were covered by a combination of existing EPMES staff and limited advisory support from two technical specialists from SI-HQ. In advance of transitions between personnel, SI prepared and submitted plans to USAID to ensure each role’s responsibilities would be covered in full on an interim basis. Subcontracts EPMES hired six different firms, four of which are headquartered in Ethiopia, under subcontracts to assist with delivering services to USAID/Ethiopia. Approximately 5.25 percent of EPMES’s expenditures was on subcontracts. Below is a summary of the subcontracts with these firms. ISS-Ethiopia SI issued one Indefinite Quantity Contract (IQC) subcontract to ISS-Ethiopia. Period of Performance: May 23, 2016 – May 3, 2022 Subcontract Ceiling: $201,114.99 Spent under Subcontract: $200,433.77 Task Orders Issued: 10 time & materials (T&M) task orders Summary of Services: ISS-Ethiopia provided ongoing GIS support to EPMES and USAID/Ethiopia. ISS￾Ethiopia forwarded a staff member to EPMES throughout the IQC period of performance. SART SI issued one IQC subcontract to SART. Period of Performance: May 23, 2016 – May 3, 2022 Subcontract Ceiling: $1,645,396.63 Spent under Subcontract: $201,711.97 Task Orders Issued: Six firm fixed price (FFP) task orders Summary of Services: SART supported EPMES with data collection efforts for the Graduation with Resilience to Achieve Sustainable Development (GRAD) activity, the Community-managed Disaster Risk Reduction (CMDRR) activity, and the Human Rights Protection Support (HRPS) activity evaluations and for all three phases of the Youth Cohort Study. PRIN SI issued one IQC subcontract and one FFP subcontract to PRIN. IQC Subcontract Period of Performance: May 23, 2016 – June 3, 2020 Subcontract Ceiling: $390,748.21 Spent under Subcontract: $181,974.52 Task Orders Issued: Two FFP task orders Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 34 Final Report Summary of Services: Under the first task order, PRIN supported EPMES with data collection for the Crop Availability Study. Under the second task order, PRIN supported EPMES with capacity strengthening and mentoring efforts for several M&E-LSPs. FFP Subcontract Period of Performance: January 26, 2017 – November 30, 2017 Subcontract Ceiling: $48,390.83 Spent under Subcontract: $48,390.83 Task Orders Issued: n/a Summary of Services: PRIN assisted EPMES and USAID/Ethiopia with the development of the CDCS 2.0. ISS-Africa SI issued two FFP subcontracts to Pretoria-based ISS-Africa. FFP Subcontract 1 Period of Performance: September 23, 2016 – April 30, 2017 Subcontract Ceiling: $140,377.00 Spent under Subcontract: $140,377.00 Task Orders Issued: n/a Summary of Services: ISS-Africa conducted a Development Trends Analysis for USAID/Ethiopia. FFP Subcontract 2 Period of Performance: March 1, 2019 – August 30, 2019 Subcontract Ceiling: $38,824.34 Spent under Subcontract: $38,824.34 Task Orders Issued: n/a Summary of Services: ISS-Africa completed an update to its Development Trends Analysis to accommodate political changes and integrate data updates. Allpoints GIS SI issued one IQC subcontract and one FFP subcontract to Allpoints GIS. FFP Subcontract Period of Performance: June 11, 2020 – December 31, 2020 Subcontract Ceiling: $18,550.00 Spent under Subcontract: $18,550.00 Task Orders Issued: n/a Summary of Services: Allpoints GIS supported EPMES with the development of the SAGE/Emergency Shocks Dashboard and Map. IQC Subcontract Period of Performance: December 31, 2020 – May 3, 2022 Subcontract Ceiling: $50,700.00 Spent under Subcontract: $50,175.00 Task Orders Issued: Three FFP task orders Summary of Services: Allpoints GIS provided wide-ranging GIS support to EPMES and USAID/Ethiopia, including through the development of the AGP Interactive Dashboard. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 35 Final Report Perfect Solution Partners SI issued one FFP subcontract to Perfect Solution Partners. Period of Performance: January 1, 2017 – April 30, 2018 Subcontract Ceiling: $7,920.00 Spent under Subcontract: $7,920.00 Task Orders Issued: n/a Summary of Services: Perfect Solution Partners provided routine IT support to EPMES. Upon conclusion of the Perfect Solution Partners subcontract, EPMES utilized a different contracting mechanism—purchase orders—to procure routine IT support through BrooTech IT Solutions. Closeout In coordination with their counterparts at SI-HQ, the EPMES team first prepared to conclude the Activity in April 2020. As a result, EPMES was well-poised to resume preparations for closeout at the end of 2021, over six months ahead of the June 2022 end date of the contract. In addition to a multi￾faceted technical closeout, which includes submission of this Final Report and preparation for a Closeout Event in May 2022, the EPMES Operations team has made progress on a variety of closeout-related tasks, including preparations for: • Phasing out our staff and subcontractors incrementally to allow sufficient time for technical handovers and operational closeout activities; • Transferring the EPMES business registration to USAID/Ethiopia’s Organizational Development Activity (ODA), also implemented by SI; • Closing or transferring to ODA each of the EPMES bank accounts; • Dispossessing of EPMES’s assets, as outlined in the Asset Disposition Plan which accompanied the Demobilization Plan SI submitted to USAID/Ethiopia in January 2022; • Conducting a final audit of EPMES’s financial records to ensure compliance with GoE and USAID regulations and obtain tax clearance from GoE; • Vacating the EPMES office space by May 20, 2022; and • Demobilizing our expatriate Chief of Party. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 36 Final Report Cross-cutting Involve learning mechanisms more directly in Mission processes. Many learning mechanisms provide a larger role in supporting design work and mission processes. More consistent engagement helps facilitate learning and increases USAID staff capacity in these areas. Experience across USAID missions shows that independent learning mechanisms can manage conflict of interest issues. Prioritize innovative learning processes. EPMES has found during its work that USAID staff and IPs did not have the time and resources to allow for more innovative learning processes that look beyond performance and foster a deeper understanding of contexts, relationships, and underlying issues and coordinated response across offices and implementing mechanisms. While some offices are endeavoring to use these approaches, the Mission leadership should encourage their use more broadly by scheduling some of these processes, such as joint data reviews or evaluative stocktaking with IPs, at least semi￾annually for each office. Such processes can drive demand for more-regular collection and use of data and spur adaptive management. Clarify planned use and prioritize those tasks with clear implementation plans. Several significant undertakings by EPMES have ended up without continuing champions or known uses. These represent large investments of resources that might be directed more productively. PRO and the Mission should carefully prioritize those tasks where the planned use is known and revisit the list of tasks being undertaken to confirm all work plan endeavors still deserve prioritization. Augment LME-related responsibilities in job descriptions. Related to Mission staff being highly programmed, prioritization naturally aligns with what they are required to do to meet their job descriptions. For instance, the Mission M&E Working Group stopped functioning over time due to a lack of participation despite the importance of LME-related responsibilities for making evidence-based adaptive management. Thus, the Mission leadership may wish to consider inserting clear language on LME-related responsibilities directly into technical staff job descriptions to clarify the importance of such functions. Parallel changes should be promoted among IPs. Regularize need identification and utilization tracking processes for learning mechanisms with technical teams. PRO should build in management touchpoints between learning mechanisms and all offices and other similar mechanisms to improve coordination and allow the mechanisms to be proactive in responding to needs. For instance, some learning mechanisms sit in on part of each quarterly or monthly meeting with technical teams or technical points of contact. This also allows learning mechanisms to assess and capture utilization of its work products within the Mission. One adaptation was for the contractor’s staff to embed on certain days within the Mission to determine the needs and use of work products. Doing so is helpful not just for tracking but for potential adaptive management and to adjust efforts as priorities change, such as when the Mission stopped pursuing an alternative growth poles strategy. This ended during the pandemic but might have been substituted with periodic meetings by the technical team, a best practice among some LME platforms. Maintain larger/more senior staffing. The traditional size across most missions of Program Office staffing and implementing mechanisms may be insufficient to provide the bandwidth needed to scale up more flexibly for the needs found in a large mission like the Ethiopia Mission generally and to adaptively manage specifically. The type of expertise requested from the Mission generally required more-senior staff who understand deeply both ADS and Mission and IP processes generally. USAID/Ethiopia already has increased PRO staffing and is planning a larger follow-on support mechanism, but it might look for staffing plans that emphasize high middle- to senior-level skillsets. KEY LESSONS LEARNED & RECOMMENDATIONS Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 37 Final Report Enhanced Utilization of ESAs Review ESA SOWs with a learning agenda question lens. To increase the focus on the long￾term learning agenda, the Mission could require as part of its regular processes reviewing each ESA (or other major task) SOW to determine to what extent the research questions could include a focus on long-term learning agenda questions and higher-level indicators. This would support a longer-term view of learning and managing across programming and could affect the approach to programming, such as unifying intervention types to promote more syntheses across Activities. (See also the related recommendation under the use of LME tools sub-section.) Include in ESA SOWs intentional dissemination and utilization plans. Aligned with the above and to further enhance ESA use as evidence bases for adaptive management practices, ESA SOWs should include intentional dissemination and utilization plans. These plans can evolve to fit emerging needs but should consider how USAID or other actors will use the evidence being produced. Additional processes or outputs should be considered each time, including non-USAID key stakeholders for in- or out-briefs, collective sense-making sessions or recommendation workshops that engage the third-party evaluator/researchers, relevant IP, and Mission staff. Improve and elevate profile of post-evaluation action tracking. Post-evaluation action planning and tracking processes and tools do not suffice to consistently capture detailed feedback on how the evidence is used and recommendations implemented. Tracking by recommendation puts greater emphasis on both the contractor and the Mission on making sure all recommendations are actionable and leads to a reasonable number of recommendations. Robust post-evaluation action processes that are detailed and shared both involve and incentivize stakeholders within both PRO and the technical offices to take ownership and follow-through. This process also reinforces the need for the recommendation workshops or sense-making sessions. Start with SOW co-creation. All LME/CLA contractor engagements, including cross-sectoral engagements, should begin with determining exact needs and potential use upfront through developing realistic SOWs jointly between the relevant technical teams and contractor. This interactive engagement helps focus all involved and establishes realistic expectations and outputs in a shorter period and avoids missed opportunities. Plan ESAs ahead eight to nine months. PRO and LME contractors should communicate a norm with technical offices that they should start plan and start to develop SOWs eight to nine months ahead of the desired outputs to allow the contractors realistic timelines to recruit staffing while juggling shifting timelines and Mission needs across multiple assignments. Capacity Building and Use of LME Tools Increase use of GESI indicators and early GESI team members. To encourage better gender balance in staffing, the Mission should consider indicators to measure and encourage IPs to attend to GESI goals. The Mission should consider including more social inclusion indicators in its LME plans to provide data to adaptively manage. Another good practice is to encourage or incentivize IPs to include a gender equality and social inclusion team member during early implementation, as onboarding one late in the process has limited effect. Collect additional context monitoring data. Due to the frequently evolving context in Ethiopia, the Mission should request its future LME/CLA mechanisms to quickly assess needs and solutions additional to those undertaken as part of EPMES’s GIS dashboards to improve the ability to collect regular data across activities for context monitoring purposes. Collecting qualitative monitoring data by Activity or across activities also would deepen contextual understanding. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 38 Final Report Conduct annual skills and needs survey. The Mission should consider under follow-on mechanisms conducting an annual survey of skills and needs among both Mission staff and IPs to better gauge needed capacity building support. Careful annual work planning would include a request for training preferences and combine this survey with any other surveying needs to minimize the need for additional survey requests and allow enough time to follow Paperwork Reduction Act requirements. This survey allows a systematic, unbiased understanding of needs across IPs mapping to work planning. A post-training survey from EPMES’s 2019-2022 trainings indicate that over 90 percent of IP participants are open to taking such a survey given its utility for future training. Although this figure overstates expected response rates, an annual survey has been implemented successfully in other Missions with higher response rates given additional follow up from PRO and LME contractor staff. Strive for contractors to play a support role on DQA. With sufficient capacity building of A/CORs in DQAs, PRO’s contractors ideally should play more of a support role to A/CORs, with A/CORs increasingly only using a contractor to review and provide additional inputs as they lead the process. As it has been a long time since EPMES undertook such capacity building effort on DQAs, this may represent a forthcoming need. Activity LME plan content should enhance both Activity and Mission-level learning and adapting. There are opportunities for new Activity LME Plans to align more clearly with the Project and Mission-level learning questions, approaches, and data. Future Activity LME Plans also need to include new ADS 201 requirements (such as beneficiary feedback). Encourage inclusion of more outcome indicators. As Activity LME Plans had a high proportion of output indicators, A/CORs and outside LME contractors should push Activities to include more low- to high-level outcome measures as indicators of effectiveness and improve evidence-based adaptive management. The Mission’s PRO LME/CLA support contractors also can more easily provide capacity development support for new-to-USAID partners if the Mission facilitates direct-to CLA support contractor linkages. Improve consistency of documentation and data quality. Activity LME Plan quality varies across Activities despite award size, having local or international as prime implementers, or under which Project they reside. More frequent external reviews of data will help identify missing data, frequently missing baseline data, and misalignments in PIRS that create challenges for the Mission in terms of data aggregation and data quality. Reengage on the Roadmap. The opportunity exists post pandemic to reengage on the action plan for the country-level roadmap and strategy for local capacity development in LME. This effort might be undertaken in coordination with other development partners. Better target capacity building of local service providers. As there are vast capacity differences among LSPs, capacity building support requires different levels and types of support with different plans for each supported LSP. This generally should involve a blend of training and one-to-one mentoring. However, as LSPs frequently rely on short-term consultants, another possibility is to consider working through existing associations to build capacity of individual consultants in addition to firms. Linking LSP firms with international service providers augments capacity through modeling behaviors. Figure 16: Learning Plan Snapshot Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 39 Final Report GIS Continue joint efforts with external contractor to source GIS dashboard inputs. Data inputs to GIS databases like the SAGE/Emergency Shocks Dashboard lag in general. That is particularly true for GIS data that come from secondary sources, which gather and process data for external GIS users on their determined schedules. Official sources provide that information more quickly to USAID than an external contractor. However, overloaded schedules from Mission staff assigned the tasks of gathering this information leads to further data lags and reduced utility of the dashboards. The use of an external LME contractor to engage assigned Mission staff to encourage gathering the data reduced timelines and should be continued with a new contractor assigned in the near term. Consider alternate uses for the AGP dashboard. While the Alternative Growth Poles dashboard was contracted and developed under prior Mission management that valued consolidating locations of interventions around specific secondary cities and their hinterlands, the dashboard generated could be re-branded and retooled to help offices look for synergies and improve coordination across IPs and across technical teams or offices. A potential use might be developing communities of practice or coordination groups by region. Establish GIS working group/network of IPs. The Mission and IPs use varied types of geospatial data in many ways as part of their monitoring, reporting, geographic targeting and/or planning efforts. Creating the culture of data sharing and dissemination between IPs and the Mission helps to maximize application of geospatial data, which is done infrequently in the current context. A working group/network could design and implement mechanisms in which IPs can easily exchange data and be alerted when new datasets are available. The next mechanism may also provide GIS services directly to IPs, which both could assist the IPs in targeting, provide bottom-up demand, and demonstrate how the Mission can integrate and utilize GIS services in various ways. Promote utilization of GIS products among technical offices. Technical offices utilized GIS products in many ways. However, there is still a gap among technical offices in using already available GIS products as the internal rapid assessment on the utilization of GIS dashboards showed. Socializing existing products and increasing demand for similar products will increase integration of geographic thinking during planning. CLA Follow-up on CLA Mapping Exercise. Many concrete action points and recommendations were identified from the CLA Mapping Exercise. Similarly, based on the request from the Mission learning team, EPMES developed key recommended action items to help the mission materialize the CLA mapping recommended action points and foster CLA efforts across the Mission. Proper follow up on the recommended action points and joint action planning to strategize and improve the CLA efforts across the Mission remain major priorities worthy of attention. Three in particular include integrating scenario planning, developing a cross-Activity context monitoring and reporting system, and promoting processes and tools to improve adaptive management practices. Establish communities of practice. The Mission should consider establishing more sectoral or thematic communities of practice. During community of practice meetings, both IPs & Mission staff would share M&E and learning practices, ideally with one IP and Mission staff making presentations at each meeting. Formal and informal inter-meeting communication devices should be considered from establishing virtual community websites for longer-term KM to point-to-point messaging for low￾maintenance immediate interactions. The sector offices and PRO/support contractors would need to also focus on how to capture and share tacit knowledge through KM systems towards IPs applying learnings. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 40 Final Report Push adaptive management flexibility. The Mission can continue its efforts of fostering adaptive management practices by granting IPs additional flexibility in revising monitoring indicators, work plans and theories of change. The Mission should consider integrating crisis modifiers across all new Activities to allow space for flexibility and adjustments while shortening the time required to initiate and approve crisis modifiers. Moreover, to foster adaptive management practice, the Mission should further empower AOR/CORs and decentralize decision-making processes. Promote adaptive management capacity building broadly at management level. The respondents in the post-training assessment focus group all noted a truism that adaptive management cannot occur if those in positions to manage are not given the tools and experience to do so. The Mission should roll out capacity building in adaptive management broadly at the management level for both the Mission, as currently being planned, as well as among IPs. This could include training as well as coaching/mentoring opportunities. Promote innovative learning practices. EPMES also identified that USAID staff did not have the enabling environment (e.g., resources, time, staff) to allow for more innovative and impactful learning processes that look beyond Activity performance and foster a deeper understanding of contexts, relationships, and underlying issues. These processes could include joint data reviews, events that include government or other stakeholders, data briefs, or evaluative stocktaking that examine higher level questions including broader Office or Mission learning agenda questions and drive the uptake of adaptive management toward Mission objectives. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 41 Final Report Figure 17: The EPMES Results Framework APPENDIX 1: EPMES RESULTS FRAMEWORK Ethiopia Performance Monitoring and Evaluation Service 42 Final Report Below is the EPMES contract deliverables table as of the submission of this report. Table 26: The EPMES Contract Deliverable Tracking Table Contract Deliverable Contract Ref Status Mobilization Plan F.5.1 Approved Annual Work Plan (Year 1) F.5.2 Approved Annual Work Plan (F Y 2017, revised) F.5.2 Complete Annual Work Plan (FY 2018) F.5.2 Approved Annual Work Plan (FY 2019) F.5.2 Approved Annual Work Plan (FY 2020) F.5.2 Approved Annual Work Plan (June 4, 2020 - June 3, 2021) F.5.2 Approved Annual Work Plan (June 4, 2021 – September 30, 2021) F.5.2 Approved Annual Work Plan (FY 2022) F.5.2 Approved Procurement Plan F.5.3 Approved Public Relations, Communication, and Outreach Strategy F.5.4 Approved Branding and Marking Plan F.5.5 Approved M&E Plan F.5.6 Complete MEL Plan (2018 Update) F.5.6 Complete MEL Plan (2019 Update) F.5.6 Approved Annual Progress Report + Financials (FY 2017) F.6.5 Approved Annual Progress Report + Financials (FY 2018) F.6.5 Complete Annual Progress Report + Financials (FY 2019) F.6.5 Approved Annual Progress Report + Financials (FY 2020) F.6.5 Approved Annual Progress Report + Financials (FY 2021) F.6.5 Approved Quarterly Progress Report + Financials (FY 2016 QIII) F.5.7 Approved Quarterly Progress Report + Financials (FY 2016 QIV) F.5.7 Complete Quarterly Progress Report + Financials (FY 2017 QI) F.5.7 Complete Quarterly Progress Report + Financials (FY 2017 QII) F.5.7 Approved Quarterly Progress Report + Financials (FY 2017 QIII) F.5.7 Approved Quarterly Progress Report + Financials (FY 2017 QIV) F.5.7 Complete Quarterly Progress Report + Financials (FY 2018 QI) F.5.7 Complete Quarterly Progress Report + Financials (FY 2018 QII) F.5.7 Complete Quarterly Progress Report + Financials (FY 2018 QIII) F.5.7 Complete Quarterly Progress Report + Financials (FY 2018 QIV) F.5.7 Complete Quarterly Progress Report + Financials (FY 2019 QI) F.5.7 Approved Quarterly Progress Report + Financials (FY 2019 QII) F.5.7 Approved Quarterly Progress Report + Financials (FY 2019 QIII) F.5.7 Approved Quarterly Progress Report + Financials (FY 2019 QIV) F.5.7 Approved Quarterly Progress Report + Financials (FY 2020 QI) F.5.7 Approved Quarterly Progress Report + Financials (FY 2020 QII) F.5.7 Approved Quarterly Progress Report + Financials (FY 2020 QIII) F.5.7 Approved Quarterly Progress Report + Financials (FY 2021 QI) F.5.7 Approved Quarterly Progress Report + Financials (FY 2021 QII) F.5.7 Approved Quarterly Progress Report + Financials (FY 2021 QIII) F.5.7 Approved Quarterly Progress Report + Financials (FY 2022 QI) F.5.7 Approved Quarterly Progress Report + Financials (FY 2022 QII) F.5.7 Submitted Final Report + Financial Report F.5.9 This Report Demobilization Plan (2020 closeout) F.5.10 Complete APPENDIX 2: CONTRACT DELIVERABLES Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 43 Final Report Contract Deliverable Contract Ref Status Demobilization Plan (2022 closeout) F.5.10 Submitted Training workshops completed for IP personnel on M&E F.5.11 Complete Training workshops completed for USAID/Ethiopia personnel on GIS and Global Positioning System (GPS) F.5.12 Complete Spatial Analysis completed using data from USAID and other sources F.5.13 Complete Maps and dashboards produced using GIS or other software F.5.14 Complete Geo/Meta Databases F.5.15 Complete Training workshops completed for USAID/Ethiopia and IP personnel on the use and management of USAID/Ethiopia's Portfolio Management System, AIDtracker Plus F.5.16 Complete Training workshops completed for USAID/Ethiopia and IP personnel on the use of different software tools such as Access and Excel F.5.17 Complete Updated/refined Mission-wide Performance Management Plan (PMP) completed F.5.18 Complete M&E Plans at project and/or Activity level completed, including performance indicator reference sheets F.5.19 Complete Logistical and analytical support provided to USAID/Ethiopia for Portfolio Reviews (both internal and joint portfolio reviews with the GoE) F.5.20 Complete Logistical and analytical support provided to USAID/Ethiopia and IP on DQA F.5.21 Complete Logistical and analytical support provided to USAID/Ethiopia on data collection of cross-cutting indicators F.5.22 Complete Templates for data storage and data transfer procedures developed F.5.23 Complete Logistical assistance to USAID/Ethiopia for field monitoring of USAID activities F.5.24 Complete KM plans F.5.25 Approved Learning and adaption agenda implementation/integration plan F.5.26 Approved Database of qualified International, regional, and national experts and firms/organizations with technical expertise in USAID programmatic areas F.5.27 Complete 2017 Update F.5.27 Complete 2018 Update F.5.27 Complete 2019 Update F.5.27 Complete 2020 Update F.5.27 Complete 2021 Update F.5.27 Complete Final Update F.5.27 Complete Performance evaluation final reports and data sets F.5.28 Complete GRAD Evaluation Final Report F.5.28 Approved Health Sector Finance Reform Evaluation Final Report F.5.28 Approved HRPS Evaluation Final Report F.5.28 Approved CMDRR Evaluation Final Report F.5.28 Approved Strengthening Human Resources for Health (SHRH) Evaluation Final Report F.5.28 Approved Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 44 Final Report Contract Deliverable Contract Ref Status Strengthening Ethiopia’s Urban Health (SEUH) Evaluation Final Report F.5.28 Approved Challenge Tuberculosis (TB) Evaluation Final Report F.5.28 Approved Youth Potential Evaluation 2019 Final Report F.5.28 Approved SD Evaluation Final Report F.5.28 Approved Impact evaluation final reports and data sets F.5.29 n/a Baseline data collection reports and data sets F.5.30 Complete Youth Potential Evaluation/Baseline Survey Final Report F.5.30 Approved Special studies/assessments reports and data sets F.5.31 Complete Youth Potential Performance Monitoring and Data Systems Final Report F.5.31 Approved Youth Potential Youth Cohort Study (YCS) Baseline Report F.5.31 Approved Youth Potential YCS Midline Report F.5.31 Approved Youth Potential YCS Endline Report F.5.31 Approved Gender Analysis 2016 Final Report F.5.31 Approved Crop Availability and Market Study Final Report F.5.31 Approved Gender Analysis 2019 Synthesis Report F.5.31 Approved Gender Analysis 2019 Project-Level Reports (11) F.5.31 Approved Secondary Cities Assessment F.5.31 Approved PSE-ECE Assessment F.5.31 Complete Health Leadership, Management, and Governance (LMG) Synthesis F.5.31 Approved Roster of Select Local M&E consultants/firms/organizations/institutions with the potential to provide quality M&E services to USAID and other donors in Ethiopia F.5.32 Complete 2017 Update F.5.32 Complete 2018 Update F.5.32 Complete 2019 Update F.5.32 Complete 2020 Update F.5.32 Complete 2021 Update F.5.32 Complete Final Update F.5.32 Complete Pre-assessment of local M&E firm capacity F.5.33 Approved Post-assessment of local M&E firm capacity F.5.33 Approved Country-level strategy to enhance local M&E capacity F.5.34 Approved Learning events completed with USAID/Ethiopia, staff, IPs, and/or GoE F.5.35 Complete Gender & Youth Learning Event + Report (2018) F.5.35 Complete Gender Learning Workshop + Report (2022) F.5.35 Approved Ethiopia Performance Monitoring and Evaluation Service 45 Final Report Table 27: The EPMES Performance Indicator Tracking Table # Name of Indicator Data Type + Unit of Measurement Disaggregation Baseline Year FY16 FY17 FY18 FY19 FY20 FY21 FY22 Target (T) Actual (A) T A T A T A T A T A T A 1 # of EPMES activities with documented utilization during the fiscal year Activity n/a 2019 n/a n/a n/a n/a n/a n/a n/a n/a 4 5 5 5 2 2 2 # of evaluations, studies, and assessments (ESAs) with documented utilization of findings and recommendations completed in this fiscal year* # (integer, of ESAs) n/a 2016 0 0 - 6 - 2 - 2 2 3 2 2 1 0 3 # of evaluations, studies, assessments involving primary data collection and completed in this fiscal year # (integer, number of ESAs with primary data collection) Total 2016 0 0 11 11 3 2 4 3 4 4 2 2 1 1 Program/Cross-cutting 0 0 2 2 0 0 0 1 3 3 1 0 0 0 Tech Offices Sub-Total 0 0 9 9 3 2 4 2 1 1 1 1 1 1 ALT 0 0 3 3 0 0 1 0 0 0 0 0 0 0 EG&T 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Health, AIDS, Population, and Nutrition (HAPN) 0 0 1 1 2 1 2 2 0 0 0 0 1 1 DG 0 0 1 1 0 0 0 0 0 0 1 1 0 0 EYO 0 0 4 4 1 1 1 0 1 1 0 0 0 0 Office of U.S. Foreign Disaster Assistance (OFDA) 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 # of local consultants contracted to conduct evaluations, studies, assessments, or reviews # (integer, of cooperating country national (CCN) consultants) Total 2016 0 0 8 21 12 15 11 0 2 3 6 3 3 Male 0 0 5 20 10 13 7 0 2 2 6 3 3 Female 0 0 3 1 2 2 4 0 0 1 0 0 0 5 # of local consultants trained and/or mentored # (integer, of CCN consultants) Total 2016 0 0 8 21 12 15 11 0 0 3 6 3 3 Male 0 0 5 20 10 13 7 0 0 2 6 3 3 Female 0 0 3 1 2 2 4 0 0 1 0 0 0 6 # of USAID staff members that received training # (integer, of USAID staff) Total 2016 0 0 42 21 32 28 70 33 8 5 8 24 20 0 Primary Training Topic MEL 2016 30 0 20 2 30 2 0 0 0 0 0 0 CLA 0 0 0 0 0 0 0 0 0 0 0 0 GIS 12 0 12 4 40 8 8 5 8 24 20 0 DQA 0 21 0 22 0 23 0 0 0 0 0 0 Other (add notes) 0 0 0 0 0 0 0 0 0 0 0 0 Sex Male 4 2 4 17 15 0 Female 4 3 4 7 5 0 APPENDIX 3: FINAL INDICATOR TRACKING TABLE Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 46 Final Report # Name of Indicator Data Type + Unit of Measurement Disaggregation Baseline Year FY16 FY17 FY18 FY19 FY20 FY21 FY22 Target (T) Actual (A) T A T A T A T A T A T A 7 # of USAID IP staff members that received training # (integer, of USAID implementing partner staff) Total 2016 0 0 60 0 60 68 60 70 70 66 60 68 30 44 Primary Training Topic MEL - - 60 0 60 68 60 47 50 18 32 68 20 26 CLA - - 0 0 0 0 0 0 0 15 32 68 10 26 DIS n/a 46 20 13 0 0 GIS - - 0 0 0 0 0 0 20 0 20 0 0 0 DQA - - 0 0 0 0 0 0 0 0 0 0 0 0 Other (add notes)** - - 0 0 0 0 0 0 0 0 0 0 0 18 Sex 70 66 Male 35 52 30 55 24 39 Female 35 14 30 13 6 5 8 % of training participants who, after completing a post-training survey, can provide evidence of how they have utilized knowledge or skills from the training % (of training participants who respond to a post￾training survey) Total 2017 - - - - n/a 100% n/a 89.3% 90% 0% 90% 100 85% NA*** USAID Staff - Sub-Total n/a n/a 90% NA NA NA Male n/a n/a 90% NA NA NA Female n/a n/a 90% NA NA NA IP Staff - Sub-Total 90% 90% 100 85% NA*** Male 90% 0% 90% 100 85% Female 90% 0% 90% 100 85% 9 # of IPs that receive mentoring and/or coaching support on MEL or CLA topics # (of USAID implementing partners) 2016 0 0 0 3 0 1 0 0 0 0 0 0 0 10 # of maps prepared for USAID # (of maps) Total 2016 375 268 300 124 50 77 92 40 20 0 PRO 10 34 15 13 7 ALT 34 64 9 6 EG&T 122 42 5 1 HAPN 24 32 65 36 33 DG 8 2 1 0 EYO 13 2 4 0 OFDA 0 0 0 0 USAID Working Groups 138 81 15 0 EPMES-internal 26 11 10 21 11 # of programs and processes EPMES supports to enhance CLA practices in USAID/Ethiopia # (of Projects and Processes) Total 2019 6 5 4 4 2 Programs (CDCS or Projects) 3 2 1 1 0 Processes 3 3 3 3 2 12 # of Project or Activity designs developed with EPMES support # (of designs) Total 2018 1 2 1 1 1 0 Project n/a 1 1 1 0 0 0 Activity n/a 0 1 0 1 0 0 13 # of learning products shared with USAID to inform programming # (integer, of learning products) 2018 1 4 5 6 2 4 2 1 Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 47 Final Report # Name of Indicator Data Type + Unit of Measurement Disaggregation Baseline Year FY16 FY17 FY18 FY19 FY20 FY21 FY22 Target (T) Actual (A) T A T A T A T A T A T A 14 # of documents added to the KM Catalogue in the fiscal year # (integer, of documents) 2018 0 40 70 20 42 12 3 15 Number of MEL and/or CLA plans completed with EPMES support # (integer, of MEL and/or CLA Plans) Total 2018 1 0 7 2 1 0 0 00 PMP (CDCS- and DO￾level) 1 0 1 1 0 0 Project 0 0 3 1 0 0 Activity 3 0 1 0 16 # of identified M&E Local Service Providers (LSPs) who improved technical and management capacity as evidenced by TOCAT # (integer, of LSPs receiving capacity building support form EPMES) Both Organizational and Technical Capacity 2019 5 3 0 0 0 0 Organizational Capacity n/a (based on increase) 5 4 Technical Capacity n/a (based on increase) 5 3 17 % of USG-assisted organizations with improved performance [IM-level] (Mission PPR indicator, CBLD-9) % (numerator: total number of Ethiopian MEL LSP with improved performance; denominator: total number of Ethiopian MEL LSP receiving organizational capacity development support) Type of firm: Private sector firms (not cooperative) are the only disaggregates on which EPMES will report 2019 4 4 0 0 0 0 18 # of M&E LSPs registered in EPMES M&E Service Providers Roster for the fiscal year # (integer, of M&E LSPs) 2016 0 0 0 150 114 62 200 161 200 221 250 19 10 89 19 # of LSP staff that received M&E capacity strengthening support # (integer, of staff) Total 2016 0 0 20 0 20 0 20 28 18 24 0 0 0 0 Male 0 0 15 0 16 0 16 27 12 16 0 0 0 0 Female 0 0 5 0 4 0 4 1 6 8 0 0 0 0 20 # of stakeholders who participated in the development of the country level roadmap # (of people participating in roadmap development) Total 2016 0 0 0 0 0 0 20 25 31 0 31 0 0 0 Male 0 0 0 0 0 0 15 21 20 0 20 0 0 0 Female 0 0 0 0 0 0 5 4 11 0 11 0 0 0 21 Country-level roadmap for M&E local capacity finalized # (roadmap document) 2016 1 0 1 1 0 0 0 0 * Utilization of the Health LMG Synthesis is not known yet ** Qualitative data analysis and agile management *** The trainings for 2022 were held in April, which does not allow a follow-up survey six months later to determine evidence of utilization in line with the PIRS. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 48 Final Report In accordance with F.6.5(vi) of the EPMES contract, EPMES hereby submits its Final Financial Report. Total funds awarded to date by USAID into the contract As of May 4, 2022, USAID/Ethiopia obligated $16,658,917.71 of the $16,658,919.88 contract budget ceiling to SI for the implementation of EPMES. Total funds previously reported as expended by contractor by main line items SI has expended the following amounts in support of EPMES implementation and invoiced USAID/Ethiopia accordingly through March 2022. Table 28: EPMES funds reported as expended Cost Elements Total Invoiced Direct Labor $5,817,809.68 Materials/Services $280,359.86 Overhead/Fringe $5,004,825.66 Information Technology Support $37,664.18 Travel $841,744.77 Grants Under Contracts - Subcontracts $803,463.79 Other Costs $1,199,536.93 Total Cost before G&A $13,985,404.87 General and Administrative $1,237,727.53 Total Costs before Profit/Fee $15,223,132.40 Fee/Profit $882,942.06 Total Est. Cost + Fixed Fee $16,106,074.46 Total Funds Expended in the Current Quarter by the contractor by Main Line Items Below is an estimate of funds expended under EPMES in April 2022. Table 29: Estimated funds expended in the current quarter Cost Elements Estimated Expenditure Direct Labor $78,507.92 Materials/Services $220.00 Overhead/Fringe $58,555.65 Information Technology Support $581.00 Travel $1,325.00 Grants Under Contracts - Subcontracts $14,263.21 Other Costs $11,091.69 Total Cost before G&A $164,544.47 General and Administrative $25,277.31 Total Costs before Profit/Fee $189,821.78 Fee/Profit $11,009.66 Total Est. Cost + Fixed Fee $200,831.44 APPENDIX 4: FINAL FINANCIAL REPORT Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 49 Final Report Total Un-Liquidated Obligations by Main Line Items A total of $2.19 of the EPMES contract budget remains un-obligated. Below is a table summarizing the unspent, un-liquated dollar value remaining under each line item of the EPMES contract budget as of April 1, 2022. Table 30: Un-liquidated obligations Cost Elements Amount Remaining Direct Labor $193,710.07 Materials/Services $4,487.13 Overhead/Fringe $176,516.62 Information Technology Support $1,078.30 Travel $11,862.49 Grants Under Contracts - Subcontracts $34,652.43 Other Costs $29,984.70 Total Cost before G&A $452,291.74 General and Administrative $70,246.83 Total Costs before Profit/Fee $522,538.57 Fee/Profit $30,306.86 Total Est. Cost + Fixed Fee $552,845.43 List of the Significant Expenses Incurred During the Period Below is a summary of subcontractor and consultant services valued over USD $50,000.00 and fixed assets valued over USD $5,000.00 incurred over the implementation of EPMES. Table 31: EPMES's significant expenses Service Provider or Asset Expense Incurred Subcontractor: ISS-Ethiopia (ongoing support) $200,433.77 Subcontractor: SART (six assignments) $201,711.97 Subcontractor: PRIN (three assignments) $230,365.35 Subcontractor: ISS-Africa (two assignments) $147,698.34 Subcontractor: Allpoints GIS (four assignments) $67,625.00 Consultant: Deepika Chawla (three assignments) $141,856.38 Consultant: Abebe Alebachew (two assignments) $105,349.45 Consultant: John Osika (two assignments) $103,550.04 Consultant: Beulah Jayakumar (two assignments) $65,370.00 Vehicle: Toyota Land Cruiser $34,111.40 Vehicle: Toyota RAV4 $24,423.00 Equipment: HQ ProLiant ML150 Server $16,579.04 Activity-level Spending through March 2022 Below is a table summarizing the value spent on each individual activity completed by EPMES. Note that EPMES accounted for its expenditures using client line-item number (CLIN)-level tracking, representing each of the four EPMES contract objectives, from April 2016 through August 2016. Beginning in September 2016, EPMES switched to tracking expenditures by individual activities or as “general support.” Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 50 Final Report Table 32: EPMES's activity-level spending Service Provider or Asset Expense Incurred CLIN 1-4 $680,190 EPMES General Support $9,448,386 Youth Potential PE $333,500 Youth Potential PMDS $53,103 Youth Potential Cohort Study (three phases) $704,393 Development Trends Analysis $221,756 Gender Analyses $319,490 GRAD Evaluation $361,119 HSFR Evaluation $314,569 HRPS Evaluation $334,914 SEUH Evaluation $299,023 CMDRR Evaluation $419,242 Crop Availability Study $228,381 SHRH Evaluation $368,714 DQA Support $11,931 GIS Support $328,238 MEL Training $102,283 CDCS/CLA Support $422,900 Capacity Strengthening $180,378 Challenge TB Evaluation $262,238 Education Workshops $4,158 Secondary Cities Assessment $96,733 ALMEP Reviews $191,382 SD Evaluation $242,101 PSE-ECE Assessment $91,690 Health LMG Synthesis $81,301 Post-Conflict Assessment $3,962 Total $16,106,074 Additional Financial Information Between April 2016 and March 2022, EPMES submitted 72 monthly invoices to USAID averaging USD $223,695 in value. As of March 31, 2022, EPMES has spent and invoiced 96.68% of its contract budget value to USAID/Ethiopia. USAID/Ethiopia and SI realigned the EPMES budget four times via contract modification numbers 5, 8, 16, and 20 over the course of implementation. The contract ceiling of $16,658,919.88 was never adjusted, though the EPMES period of performance was extended from its initial timeframe of five years to a period of six years and two months. Below is a chart depicting EPMES’s expenditures by cost category, which shows that over 82 percent of funding is related to direct labor, subcontracts, or overhead and fringe. With the pandemic interrupting international travel, travel represents only six percent of costs. Figure 19 displays invoice values over the course of the EPMES Activity with higher costs incurred later in year one and early in year two, with average costs stabilizing afterward. Ethiopia Performance Monitoring and Evaluation Service (EPMES) Activity 51 Final Report Figure 18: EPMES expenditures by category Figure 19: EPMES monthly invoice values Direct Labor 38.22% Materials/Services 1.84% Overhead/Fringe 32.88% IT Support 0.25% Travel 5.53% Subcontracts 5.28% Other Costs 7.88% G&A 8.13% $- $100,000 $200,000 $300,000 $400,000 $500,000 Apr-16 Apr-17 Apr-18 Apr-19 Apr-20 Apr-21 U.S. Agency for International Development Entoto Street Addis Ababa, Ethiopia