December 2017 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Chris Thompson, Stephanie Schmidt, Jade Lamb, Marissa Germain, Anne Wagner, and Nicolle Nelson of Social Impact, Inc. META-ANALYSIS OF USAID’S DEVELOPMENT ASSISTANCE UNDER THE COUNTRY DEVELOPMENT COOPERATION STRATEGY FOR LIBERIA 2013 - 2018 META-ANALYSIS OF USAID’S DEVELOPMENT ASSISTANCE UNDER THE COUNTRY DEVELOPMENT COOPERATION STRATEGY FOR LIBERIA 2013 - 2018 December 2017 CONTRACT NUMBER 669-C-12-00004 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS Acronyms.............................................................................................................................................................................. i Executive Summary ........................................................................................................................................................... ii Background..........................................................................................................................................................................1 Purpose & Research Questions .....................................................................................................................................3 Purpose.................................................................................................................................................................................................3 Research Questions..........................................................................................................................................................................4 Users & Utilization.............................................................................................................................................................................4 Methods & Limitations......................................................................................................................................................4 Design ...................................................................................................................................................................................................4 Methods................................................................................................................................................................................................4 Limitations ...........................................................................................................................................................................................6 Findings.................................................................................................................................................................................8 Question 1: To what extent has progress been made in achieving USAID’s strategic goal from the current CDCS for FY 2013 – FY 2018? ......................................................................................................................................................8 Question 2: How have USAID’s development objectives contributed to achieving USAID’s strategic goal? .........16 Question 3: To what extent were the programmatic and contextual assumptions identified in USAID’s development hypothesis sufficient to achieve its strategic goal?.........................................................................................19 Question 4: What were the unintended outcomes of USAID’s development assistance under the current CDCS for FY 2013 – FY 2018?..................................................................................................................................................................22 Conclusions.......................................................................................................................................................................23 Question 1: To what extent has progress been made in achieving USAID’s strategic goal from the current CDCS for FY 2013 – FY 2018? ....................................................................................................................................................23 Question 2: How have USAID’s development objectives contributed to achieving USAID’s strategic goal? .........24 Question 3: To what extent were the programmatic and contextual assumptions identified in USAID’s development hypothesis sufficient to achieve its strategic goal?.........................................................................................25 Question 4: What were the unintended outcomes of USAID’s development assistance under the current CDCS for FY 2013 – FY 2018?..................................................................................................................................................................26 Recommendations...........................................................................................................................................................27 Question 1: To what extent has progress been made in achieving USAID’s strategic goal from the current CDCS for FY 2013 – FY 2018? ....................................................................................................................................................27 Question 2: How have USAID’s development objectives contributed to achieving USAID’s strategic goal? .........28 Question 3: To what extent were the programmatic and contextual assumptions identified in USAID’s development hypothesis sufficient to achieve its strategic goal?.........................................................................................28 Question 4: What were the unintended outcomes of USAID’s development assistance under the current CDCS for FY 2013 – FY 2018?..................................................................................................................................................................29 Annexes.............................................................................................................................................................................30 Annex I: Evaluation Statement of Work ....................................................................................................................................30 Annex II: Original Results Framework from the USAID/Liberia CDCS for FY 2013 – FY 2017................................34 Annex III: Evaluation Methods......................................................................................................................................................35 Annex IV: Disclosure of Any Conflict of Interest ...................................................................................................................40 Annex V: Desk Review Parameters............................................................................................................................................46 Annex VI: Interview Guide ............................................................................................................................................................53 Annex VII: Performance Indicator Data.....................................................................................................................................55 Annex VIII: Bibliography from Desk Review.............................................................................................................................60 Annex IX: Integration in Strategic Planning...............................................................................................................................66 TABLES 1. Table: Assumptions to the Development Hypothesis.........................................................................................................1 2. Table: Strategic Planning Process, Milestones, and Dates..................................................................................................3 3. Table: Desk Review Sample .......................................................................................................................................................5 4. Table: KII Matrix ...........................................................................................................................................................................6 5. Table: Limitations and Mitigation Measures...........................................................................................................................7 6. Diagram: USAID/Liberia Results Framework and IR-level Achievements.....................................................................8 7. Chart: Percentage of Evidence of Achievement from the Desk Review by DO, Normalized by the Number of Documents Available for Each DO ....................................................................................................................................................9 8. Table: Performance Indicator Data from the Numerical Review ..................................................................................10 9. Table: Data from the Desk Review........................................................................................................................................11 10. Table: Data from KIIs............................................................................................................................................................12 11. Table: USAID/Liberia’s Gender Indicators......................................................................................................................14 12. Graphic: Impactful Quotes about Gender and Youth..................................................................................................15 13. Chart: Frequency of Gender-related Findings from the Desk Review by Theme ................................................15 14. Diagram: Timeline of Milestones During the Current CDCS....................................................................................16 15. Graph: Positive Relationship between DOs and Evidence of Achievement...........................................................17 16. Matrix: Evidence Co-Occurrence from Desk Review .................................................................................................17 17. Chart: Percentage of Integrated and Non-Integrated DOs in CDCSs Globally ...................................................18 18. Charts: Commodity Prices and GDP; and Population Growth.................................................................................20 19. Diagram: Challenges to Assumptions to the Development Hypothesis.................................................................21 20. Table: Percentage of Evidence of Challenges to Assumptions by DO, Normalized by the Number of Desk Review Documents and KIIs..............................................................................................................................................................22 21. Table: Status of Assumptions to the Development Hypothesis................................................................................25 i ACRONYMS ADS Automated Directives System AFT Agenda for Transformation AMELP Activity Monitoring, Evaluation, and Learning Plan AYP Advancing Youth Project CDCS Country Development Cooperation Strategy CLA Collaborating, Learning, and Adapting CSO Civil Society Organization DO Development Objective DQA Data Quality Assessment DRG Democracy, Rights, and Governance DVC Digital Video Conference EG Economic Growth EQUAL Education Quality and Access in Liberia EVD Ebola Virus Disease FARA Fixed Amount Reimbursement Agreement FED Food and Enterprise Development Program F Foreign Assistance Framework FY Fiscal Year GOL Government of Liberia HICD Human and Institutional Capacity Development IBEX Investing for Business Expansion IP Implementing Partner IR Intermediate Result KII Key Informant Interview LSA Liberia Strategic Analysis MEL Monitoring, Evaluation, and Learning PAD Project Appraisal Document PIDS Performance Indicator Database System PMP Performance Management Plan PPD Program and Project Development Office PPR Performance Plan and Report PROSPER People, Rules, and Organizations Supporting the Protection of Ecosystem Resources PSMP Public Sector Modernization Program PTA Parent-Teacher Association SHOPS Smallholder Oil Palm Support SI Social Impact SMI-L Sustainable Markets in Liberia SOW Statement of Work STIP Science, Technology, Innovation, and Partnerships UNDP United Nations Development Programme USAID United States Agency for International Development USG United States Government ii EXECUTIVE SUMMARY USAID engaged the Liberia Strategic Analysis (LSA) activity to conduct a Meta-Analysis of the United States Agency for International Development’s (USAID) development assistance under the Country Development Cooperation Strategy (CDCS) for Liberia Fiscal Year (FY) 2013 - FY 2018 (the Meta￾Analysis). Please see Annex I for the statement of work (SOW). Background In 2013, USAID/Liberia finalized a CDCS for FY 2013 – FY 2017, which it later extended through FY 2018. This CDCS established USAID’s strategic goal as: Strengthened Liberian institutions positioned to drive inclusive economic growth and poverty reduction. To accomplish this, it put forth four Development Objectives (DO): • DO 1: Democracy, Rights, and Governance (DRG): More effective, accountable, and inclusive governance; • DO 2: Economic Growth (EG): Sustained, market‐driven economic growth to reduce poverty; • DO 3: Health: Improved health status of Liberians; and • DO 4: Education: Better educated Liberians. USAID/Liberia commenced the designing of a new strategy for FY 2019 – FY 2023 in February 2017. It anticipates the design, preparation, and approval of the strategy will continue through November 2018. Purpose and Research Questions The Meta-Analysis generates recommendations for USAID’s new strategy for Liberia by examining progress toward the strategic goal of the current CDCS. The Meta-Analysis grounds the new strategy in evidence produced by the Mission over the past six years by synthesizing monitoring, evaluation, and learning (MEL) under the CDCS. Four research questions guide the Meta-Analysis: 1. To what extent has progress been made in achieving USAID’s strategic goal from the current CDCS for FY 2013 – FY 2018? a. What has been achieved b. What has not been achieved? c. What gaps exist? 2. How have USAID’s DOs contributed to achieving USAID’s strategic goal? 3. To what extent were the programmatic and contextual assumptions identified in USAID’s development hypothesis sufficient to achieve its strategic goal? 4. What were the unintended outcomes of USAID’s development assistance under the current CDCS for FY 2013 – FY 2018? The primary intended user of the Meta-Analysis is USAID/Liberia, specifically the Mission’s CDCS Working Group. The CDCS Working Group is taking the lead in developing the next strategy and gauging the progress made in achieving the strategic goal from the current CDCS. Methods and Limitations The Meta-Analysis utilizes a mixed-methods research approach with a concurrent triangulation design. This approach systematically combines qualitative and quantitative data from multiple sources to provide a broad perspective and strengthen the validity of the findings. The Meta-Analysis uses a numerical iii review to measure quantitative data from performance indicators; a desk review to gather qualitative data from assessments, evaluations, reports, and other documents; and key informant interviews (KIIs) to collect qualitative data from stakeholders, including USAID, the Government of Liberia (GOL), other donors, implementing partners (IPs), and civil society actors. Key Findings and Conclusions The Meta-Analysis generated the following key findings and conclusions: • USAID/Liberia has made progress toward its strategic goal of strengthened Liberian institutions positioned to drive inclusive economic growth and poverty reduction. Its progress has been mixed with some DOs experiencing more success than others, and achievements have been unevenly distributed in certain intermediate results (IRs). • USAID/Liberia’s progress has been most pronounced in improving the health status of Liberians (DO 3: Health). It increased the effectiveness of the health system at the national and county levels (IR 3.2) and increased the utilization of quality reproductive, maternal, and child health services (IR 3.1). • USAID/Liberia made progress towards more effective, accountable, and inclusive governance (DO1: DRG), where it had success assisting Liberia to manage public resources more transparently and accountably (IR 1.1). • The Mission also made progress fostering better educated Liberians (DO 4: Education). It increased equitable access to education (IR 4.1), and to a lesser extent, improved the quality of education (IR 4.2). • Compared to other DOs, USAID/Liberia made less progress supporting sustained, market￾driven economic growth to reduce poverty (DO 2: EG). The Mission did not make significant progress enhancing food security in Liberia (IR 2.1). • USAID/Liberia experienced limited progress supporting gender equality and youth development. It was unable to improve women’s participation in the labor force, gender parity in education, and the competencies that youth need to become informed and productive citizens. • A gap emerged in the Mission’s monitoring of its projects and activities. USAID/Liberia failed to systematically collect data for performance indicators due in part to a delay approving its Performance Management Plan (PMP). • USAID/Liberia’s DOs contributed to achieving the strategic goal in line with the logic model from its results framework. A causal relationship exists between IRs and their respective DOs. IRs contributed to the achievement of DOs, and DOs contributed to the achievement of the strategic goal. • The Mission made progress on each DO independent of other DOs. The assistance provided by the DRG, EG, Health, and Education teams was disconnected. This lack of integration challenged the CDCS’s assumption that there are critical feedback loops among DOs. • The current CDCS does not contain an integrated DO, although USAID has developed these in other CDCS globally. An integrated DO takes a cross-sectoral approach to achieving outcomes by involving more than one technical team. USAID has developed integrated DOs to support iv gender equality, youth development, and science, technology, innovation, and partnerships (STIP) in other strategies. • Overall, USAID/Liberia’s assumptions to its development hypothesis were sufficient to achieve the strategic goal. Assumptions related to decentralization, gender inequalities, and coordinated international support addressed pertinent issues in Liberia during this period. • Conditions during the CDCS challenged five assumptions, including Liberia’s rich base of natural resources, the momentum of positive change since 2005, the policy framework for inclusive and sustainable growth, critical feedback loops between DOs, and the Mission’s internal capacity. • USAID/Liberia’s assistance resulted in positive and negative unintended outcomes. A common positive unintended outcome was the multiplier effect between the development corridor and neighboring counties. Negative unintended outcomes included one socio-economic group receiving lopsided benefits and several non-target groups losing their livelihoods. Recommendations The Meta-Analysis recommends the following to USAID/Liberia and the CDCS Working Group: • Utilize evidence from MEL to design the new strategy for FY 2019 – FY 2023 and its subsequent projects and activities. Specify a mechanism for how the Mission will apply data to measure progress toward the new strategic goal. Closely align projects with new DOs and IRs. • Rapidly develop and implement a PMP to be approved within six months of the new strategy’s approval. Recognize the PMP will not be comprehensive when approved, and update it annually or as circumstances change. • Use monitoring data to identify performance issues and develop solutions before they impede progress toward DOs or IRs. Employ evaluations and learning events to examine problems with activity implementation and make course corrections. • Continue to monitor and evaluate activities under the current CDCS. Complete revisions to the current PMP. This plan may serve as a foundation to develop the PMP for the new strategy. • Develop integrated DOs in the new strategy, and consider gender, youth, and STIP as potential focuses. Secure Mission leadership, systematize cooperation, and combine funding streams to support integrated DOs, as these elements will be critical for success. • In the new strategy, replace the assumptions from the current CDCS that are being challenged. Consider how Liberia has changed since 2013, and develop assumptions for these new conditions based on what the Mission expects to achieve going forward. • Develop contingency scenarios in the new strategy to adapt to any uncertainty due to election outcomes or changes in the political situation. Explore a range of positive, negative, and neutral possibilities for the future of Liberia, and develop approaches in case these possibilities arise. • Further explore unintended outcomes through summative performance evaluations and whole￾of-project evaluations under the current CDCS and the new strategy. Schedule these in the current revised PMP or the upcoming PMP for the new strategy. 1 BACKGROUND Country Development Cooperation Strategy A CDCS defines a Mission’s approach in a country and provides a focal point for the broader context of projects and activities. It presents expected results, provides a common vision and framework, and summarizes the status of the portfolio. A CDCS also explains how a Mission will address new priorities, lessons learned, or changing circumstances. USAID/Liberia commenced the development of the CDCS for FY 2013 – FY 2017 in 2010 and finalized it in April 2013. The Mission extended the CDCS from FY 2017 to FY 2018, after a stocktaking exercise facilitated by LSA in February 2016. The current CDCS intends to build sustainable foundations for long￾term development and address sources of potential conflict and instability in the short-term. This is a departure from other USAID strategies since the Liberian civil war, which focused on post-war humanitarian relief and recovery. The CDCS is closely aligned with the GOL and supports development goals from its Agenda for Transformation (AFT). The CDCS establishes USAID/Liberia’s strategic goal as: Strengthened Liberian institutions positioned to drive inclusive economic growth and poverty reduction. The following development hypothesis describes the theory of change that underpins this goal: In order to achieve and sustain long-term development progress, Liberia must establish and strengthen inclusive, indigenous institutions that enable it to effectively mobilize its own resources and efficiently manage those resources for development purposes. The CDCS puts forth a results framework as the logic model for this development hypothesis with the following DOs: • DO 1: DRG: More effective, accountable, and inclusive governance; • DO 2: EG: Sustained, market‐driven economic growth to reduce poverty; • DO 3: Health: Improved health status of Liberians; and • DO 4: Education: Better educated Liberians. This results framework outlines the causal logic for achieving these DOs, showing the contribution between IRs and DOs toward the strategic goal. Please see Annex II for the original results framework from the USAID/Liberia CDCS for FY 2013 – FY 2017. USAID/Liberia later updated this results framework. Please see Diagram 6 for the current results framework used by the Meta-Analysis. The CDCS identifies assumptions outside the control of its strategy that must occur for USAID/Liberia to achieve these results. It organizes the assumptions into strategic choices, considerations, opportunities, challenges, and game changers (see Table 1). 1. Table: Assumptions to the Development Hypothesis Opportunities Challenges Game Changers • Momentum of positive change since 2005 • Emergent policy framework for inclusive and sustainable growth • Strong and coordinated international support • Low starting point for post￾conflict recovery • Low level of government capacity • Social divisions and distrust between groups • Weak legitimacy of the state • Renewed conflict • Loss of confidence in President Sirleaf’s administration 2 • Rich base of natural resource wealth • US Government (USG) and USAID access and experience • Destabilizing potential of the youth cohort • Population growth rate • Narrow window to demonstrate results Considerations • Human and institutional capacity development (HICD) advances development policies and plans • Feedback loops enable progress toward one DO to support progress toward others • Internal mission capacity provides oversight • Consolidate and extend civil peace and security • USAID Forward contributes to country-led development and capacity building Strategic Choices • Two-track approach: build capacity to sustain long-term development; and improve access to goods and services to maintain short-term stability • Support for the decentralization of services • Geographic focus in the development corridor of Bong, Grand Bassa, Lofa, Margibi, Montserrado, and Nimba counties • Address gender and other inequalities • Lock in reforms through broad-based participation, civil society, and community structures • Ensure environmentally sound approaches to development USAID/Liberia elected to support youth, nutrition, and jobs as cross-cutting sectors during the CDCS period. However, the Mission set these priorities after USAID approved and enacted the CDCS. Project Design Following approval of the CDCS, USAID/Liberia initiated its project design process to operationalize the new strategy. The DO teams developed project appraisal documents (PADs) between November 2013 and July 2014, outlining complementary activities to achieve DOs and IRs. The project design process resulted in USAID/Liberia issuing over 40 awards in support of its DOs by the end of FY 2017. Stocktaking In October 2014, USAID/Liberia conducted a course correction exercise in response to the outbreak of Ebola Virus Disease (EVD) in March 2014. The exercise concluded that the goal and DOs of the CDCS remained valid and that EVD had stressed but not contradicted the assumptions of the CDCS. It recommended tactical, rather than operational, changes to USAID’s development assistance. In June 2015, USAID/Liberia conducted a Mission retreat to discuss the CDCS and project design. The retreat found the slow project design process had delayed developing a PMP, and the Mission’s emergency response to EVD had delayed activities intended to support the CDCS. In February 2016, LSA assisted USAID/Liberia to conduct a mid-course CDCS stocktaking exercise. The exercise found many activities commenced in 2015 and had only recently begun contributing to the strategic goal. Furthermore, the lack of a PMP had hindered performance management, and assumptions to the CDCS had remained valid but required monitoring. LSA recommended extending the CDCS’s period of performance and revising the results framework to reflect changes to IRs by DO teams. In May 2016, USAID/Liberia extended the CDCS from FY 2017 to FY 2018. In November 2016, it approved a PMP and revised results framework. For the Mission’s current results framework, please see Diagram 6 on Page 8. 3 Strategic Planning In February 2017, USAID/Liberia commenced the design of a new strategy for FY 2019 – FY 2023. The design, preparation, and approval of the new strategy is anticipated to continue through November 2018 (see Table 2). 2. Table: Strategic Planning Process, Milestones, and Dates Phase Objective Milestones Anticipated Dates Analytical Phase Complete analyses to inform strategic planning process • Analyses, assessments, studies, and other reports February 2017 – October 2017 Phase 1 – Initial Consultations and Parameter Setting Initiate formal engagement with Washington to present the planned path to developing the CDCS • Concept Note • Questionnaire and Consolidation of Washington Operating Unit Inputs • Phase One Digital Video Conference (DVC) • Summary of Conclusions February 2018 - April 2018 Phase 2 – Results Framework Development Gain agency consensus on USAID/Liberia’s planned approach over three months • Results Framework Paper • Review of Results Framework Paper • Phase Two DVC • Summary of Conclusions May 2017 - July 2018 Phase 3 – Full CDCS Preparation, Review, and Approval Prepare, approve, and disseminate CDCS over four months • Draft Full CDCS • Review of CDCS Draft • Phase Three DVC • Summary of Conclusions • Final CDCS Approval • Disseminations of CDCS August 2018 - November 2018 The Analytical Phase includes over 30 studies, assessments, and reports conducted by USAID/Liberia and LSA. In June 2017, USAID/Liberia requested that LSA conduct the Meta-Analysis and scheduled it to occur during September and October 2017. This schedule intends to enable recommendations from the Meta-Analysis to be incorporated in subsequent phases of the strategic planning process. PURPOSE & RESEARCH QUESTIONS PURPOSE A meta-analysis is a systematic review of empirical evidence from a set of studies to estimate a summary effect. It includes research questions, a reproducible methodology, a systematic search to identify studies, and an assessment of the validity of these studies. The findings and conclusions from a meta￾analysis may be more persuasive than those from a single study, because a meta-analysis has a greater number and diversity of data sources. USAID/Liberia’s Meta-Analysis measures progress towards the strategic goal of the current CDCS and generates recommendations for the new strategy for FY 2019 – FY 2023. It examines unintended outcomes of the Mission’s assistance and assumptions to the development hypothesis from the CDCS. The Meta-Analysis utilizes a numerical review, desk review, and KIIs to ground USAID/Liberia’s new 4 strategy in evidence and analysis produced by the Mission and IPs over the past six years. RESEARCH QUESTIONS Four research questions guide the Meta-Analysis and frame its findings, conclusions, and recommendations to USAID/Liberia for designing a new strategy for FY 2019 – FY 2023: 1. To what extent has progress been made in achieving USAID’s strategic goal from the current CDCS for FY 2013 – FY 2018? a. What has been achieved b. What has not been achieved? c. What gaps exist? 2. How have USAID’s development objectives contributed to achieving USAID’s strategic goal? 3. To what extent were the programmatic and contextual assumptions identified in USAID’s development hypothesis sufficient to achieve its strategic goal? 4. What were the unintended outcomes of USAID’s development assistance under the current CDCS for FY 2013 – FY 2018? The research questions used in the Meta-Analysis differ from those posited by its original SOW (see Annex 1). LSA redeveloped the questions above based on guidance from the Program and Project Development Office (PPD) on August 31, 2017, during its review of the Meta-Analysis Inception Report. USERS & UTILIZATION The primary intended user of the Meta-Analysis is the USAID/Liberia Mission, specifically the Mission’s CDCS Working Group. The CDCS Working Groups is taking the lead in developing the next strategy and gauging progress made achieving the current strategy’s DOs and IRs. The Meta-Analysis may prove useful for secondary users, such as IPs, donors, and other development stakeholders, as a review of monitoring data, assessments, evaluations, and the perceptions of stakeholders. The Meta-Analysis produces content for USAID/Liberia to use in its Portfolio Review and CDCS Reflection Exercise scheduled for November 2017, and in the Concept Note from Phase 1 and the Results Framework Paper from Phase 2 of the strategy design process (see Table 2). In the findings for Question 1, the Meta-Analysis identifies positive and negative outliers, which are anticipated sessions of the Portfolio Review and CDCS Reflection Exercise. USAID/Liberia can use the conclusions from Question 1 to complete the “Past Results, Lessons, and Stocktaking” section of the Concept Note. Furthermore, the Meta-Analysis makes recommendations for the results framework of the new strategy that USAID/Liberia can incorporate in its Results Framework Paper. METHODS & LIMITATIONS DESIGN The Meta-Analysis utilizes a mixed-methods research approach with a concurrent triangulation design. This approach systematically combines qualitative and quantitative data from multiple sources to provide a broad perspective. The triangulation design analyzes data separately, and then compares and combines data to confirm and corroborate findings. This design offsets the weaknesses inherent in qualitative and quantitative types of data by allowing them to validate each other, providing for stronger conclusions. METHODS The Meta-Analysis uses the following methods in its research approach: 5 • Numerical review to measure quantitative data from performance indicators and other relevant metrics for the period of the current CDCS; • Desk review to collect qualitative data from assessments, evaluations, reports, and other documents conducted during the current CDCS; and • KIIs to collect qualitative data from key stakeholders, including individuals serving with USAID, the GOL, other donors, IPs, and civil society, who have been active during the current CDCS. For an in-depth background on each method, please see Annex III. Team and Fieldwork Schedule A team of researchers implemented the Meta-Analysis, including a Team Leader & Evaluation Specialist, a Monitoring Specialist, and four Desk Reviewers. Each team member completed a Disclosure of Conflict of Interest before beginning work. Please see Annex IV for copies of these forms. LSA and USAID gathered materials for the Desk Review in July and August 2017. In September, the Team Leader and Monitoring Specialist traveled to Liberia to conduct field work for the numerical review and KIIs, while the Desk Reviewers worked remotely. Data collection in September precluded access to data reported by IPs in October for the fourth quarter of FY 2017. The schedule permitted the team to deliver a draft final report in October 2017 for the Mission to use in its Portfolio Review and CDCS Reflection Exercise scheduled for November 2017. Numerical Review The team conducted a numerical review of quantitative data from 120 indicators used during the CDCS for FY 2013 - FY 2018. A numerical review is an assessment of quantitative data to measure changes over time against intended outcomes. The team defined eligibility criteria and identified goal, DO, IR, and context indicators from the CDCS and PMP. It identified baselines, targets, and performance data for these indicators through USAID/Liberia’s Performance Indicator Database System (PIDS), Performance Plan and Reports (PPRs), activity MEL plans (AMELPs), and IP reports. The team used 74 performance indicators, ensuring three or more indicators were available to measure each IR and DO. Of these, 37 indicators originated in the CDCS and PMP, and 37 were substituted from other sources, such as PIDs, PPRs, AMELPs, and IP reports. The team preferred to use indicators from the CDCS and PMP, but many of these lacked baselines, targets, and results. The team used 46 context indicators to measure assumptions and unintended outcomes, resulting in 120 indicators in total. To ensure the Meta-Analysis included trustworthy data for decision making, the team reviewed data quality assessments (DQA) before including indicators, when available. When data were of inadequate quality, the team substituted in similar indicators that measure the same DO or IR. Substitute indicators were carefully selected to ensure they applied to logical relationships from the results framework. Desk Review The Meta-Analysis employs a desk review of qualitative data from 353 performance evaluations, assessments, other studies, reports, and documents undertaken during the current CDCS. A desk review collects, organizes, and synthesizes information from literature. The team defined eligibility criteria for relevant studies and a strategy for gathering these documents (see Annex V). The sample for the desk review is the entire population of performance evaluations, assessments, other studies, annual and final 3. Table: Desk Review Sample DOs No. of Docs. No. of Activities DRG 71 10 EG 94 13 Health 126 6 Education 62 4 TOTAL 353 33 6 reports, and other relevant documents, including AMELPs, site visits, and quarterly reports, for 33 activities identified by USAID/Liberia and LSA as relevant to the CDCS (see Table 3). LSA circulated a list of 105 activities to PPD and the M&E focal persons for the DO teams. In the list, LSA identified a subset of activities that supported the CDCS, and Mission staff corroborated these selections and added activities for the desk review. The Meta-Analysis team lacked the time and resources to review all current and past USAID/Liberia activities. Moreover, the Mission excluded EVD￾related activities and Global Development Lab activities from the Meta-Analysis, because USAID manages these directly from Washington, D.C. The team reviewed, coded, and applied descriptors to these 353 documents using Dedoose. Dedoose is software for analyzing mixed-methods research. The team applied 38 main codes and 58 sub-codes to organize, summarize, and interpret data from 101 of the 353 documents. The team reviewed the methodologies of documents to verify they employed empirical methods. It excluded documents unrelated to the research questions and those without sound methodologies from its analysis, resulting in the difference between 353 total documents and 101 coded documents. Key Informant Interviews The Desk Review utilizes 29 KIIs to gather qualitative data from staff of USAID, GOL, other donors, IPs, and civil society. KIIs are qualitative in-depth interviews utilizing open-ended questions to collect information from individuals with firsthand knowledge on a subject. 4. Table: KII Matrix Institutions Sector Total DRG EG Health Education PPD USAID 3 4 3 3 3 16 Other Donors 1 1 2 2 0 6 GOL 0 1 1 1 0 3 IPs & Civil Society 2 1 1 0 0 4 TOTAL 6 7 7 6 3 29 The team formulated an interview guide to help discuss USAID/Liberia’s assistance, key assumptions, and unintended consequences (see Annex VI). LSA, PPD, and the M&E focal persons for the DO teams developed the sample plan of KIIs and divided these into two approximately equal groups: 16 USAID staff, including US nationals and foreign service nationals; and 13 members of stakeholders external to the Mission, including GOL, other donors, IPs, and civil society (see Table 4). The team reviewed and coded interview transcripts using Dedoose and sought to ensure informant confidentiality by using unique identifiers rather than names and avoiding references to positions. Data Analysis Plan The team analyzed data from its numerical review, desk review, and KIIs concurrently to confirm and corroborate findings. It explored data through descriptive analysis to identify trends, distributions, frequencies, and ratios. The team integrated quantitative and qualitative data through shared codes and descriptors. For example, numerical review, desk review, and KII data was coded if it indicated the Mission had achieved an IR. Codes were tallied to determine the extent that USAID/Liberia made progress toward a DO. The team visualized data in tables, charts, and diagrams to show findings. LIMITATIONS The Meta-Analysis acknowledges six potential biases associated with its methodology. The team developed and implemented mitigation strategies to limit the likelihood of each occurring (see Table 5). 7 5. Table: Limitations and Mitigation Measures Bias Method Description Mitigation Strategy Attribution issues Numeric review Changes observed through indicators by comparing before and after data may only be partly caused by a development intervention. A causal link may or may exist. The Meta-Analysis compares and combines the numerical review with the desk review method and KIIs to confirm and corroborate findings and strengthen attribution. Inclusion bias Desk review Consciously or unconsciously, the team may be influenced to select studies corroborating pre-existing viewpoints. LSA defined the eligibility criteria for identifying studies in coordination with PPD, the CDCS Working Group, and M&E focal persons. The involvement of multiple groups offers checks on inclusion bias. Incomplete data Numeric review Data from 2013 - 2017 may be incomplete, lost, not reported by IPs, or inaccurate. USAID/Liberia lacks monitoring data because it approved its PMP in Nov. 2016, which is late in the FY 2013 – FY 2018 CDCS period. Also, PPRs for FY 2013 and FY 2015 were not provided, and several IPs did not report in PIDS prior to FY 2017. Furthermore, some data may be incomplete due to being tied to external events (i.e. Liberia’s general election in 2017). USAID and LSA identified as many sources of indicator data as possible (PPRs, PMP, PIDS, IP reports, etc.) to include in the Meta-Analysis. The team substituted indicators and data from secondary sources, if the primary source had gaps. Moreover, the Meta-Analysis compares and combines the numerical review with the desk review method and KIIs to confirm and corroborate findings when data is incomplete. Other biases inherent in secondary sources Desk review & numeric review Because it draws on secondary sources, the Meta-Analysis may be exposed to the biases of those who conducted the studies in the desk review and who collected indicator data used in the numerical review. The team reviewed DQAs, when available, to confirm the integrity of indicator data. The team also reviewed the methodologies of studies to ensure they employ rigorous, investigative methods prior to collecting data from them. Publication bias Desk review The odds of publishing results are greater if they are significant or positive. Insignificant results may not be published and may be less available than significant or positive results. USAID and LSA identified as many eligible studies as possible (353). The Meta-Analysis included data from PIDS and reports where IPs are required to report results, regardless of significance. Recall bias KII Participants in KIIs may experience differences in accuracy and completeness of recollections when describing the current CDCS period. The team designed its interview guide to specify the period being referenced, and the Meta-Analysis triangulated data from the KIIs with the numerical review and desk review, which are unaffected by recall bias. 8 FINDINGS QUESTION 1: TO WHAT EXTENT HAS PROGRESS BEEN MADE IN ACHIEVING USAID’S STRATEGIC GOAL FROM THE CURRENT CDCS FOR FY 2013 – FY 2018? USAID/Liberia has made mixed progress towards achieving the strategic goal from the current CDCS. It has made more progress toward certain DOs and IRs than others (see Diagram 6). These findings were generated using the numerical review, desk review, and KIIs. The numerical review was encumbered because USAID/Liberia did not systematically track performance indicator data relevant to its projects and activities during the CDCS. 6. Diagram: USAID/Liberia Results Framework and IR-level Achievements Diagram 6 uses shades of red to indicate where methods (numerical review, desk review, and KIIs) used by the Meta-Analysis found evidence of achievement of an IR. Red shows where all three methods found evidence of achievement; pink where two of three methods did; light red where one of three methods did; and no color where none did. The sub-questions for this research question are organized by DO. Question 1.a includes DO 1: DRG, DO 3: Health, and part of DO 4: Education. It is organized in descending order, presenting the DO with the most progress first. Question 1.b includes part of DO 4: Education, DO 3: EG, gender, and youth. Question 1.c focuses on deficiencies in the performance indicator data collected by the Mission. Question 1.a: What has been achieved? DO 3: Health USAID/Liberia has achieved DO 3: Health, according to findings from the numerical review, desk review, 9 and KIIs. It accomplished targets for three of four performance indicators associated with the DO (see Table 8). The Mission’s assistance corresponded with a decrease in the infant mortality rate, under-five mortality rate, and overall fertility rate. Furthermore, a significant number of participants in KIIs felt USAID/Liberia had improved the health status of Liberians (see Table 10). In addition, findings from the desk review indicated that USAID/Liberia has achieved DO 3: Health. The team observed 25 pieces of evidence indicating that the Mission has improved the health status of Liberians (see Table 9), which is greater than the average amount of evidence supporting a DO. When the raw counts of evidence from the desk review were normalized1, more evidence was found that USAID/Liberia has achieved DO 3: Health than any other DO (see Chart 7). 7. Chart: Percentage of Evidence of Achievement from the Desk Review by DO, Normalized by the Number of Documents Available for Each DO USAID/Liberia increased the effectiveness of health systems at a national and county level (IR 3.2). IR 3.2 represents a positive outlier, where the numerical review, desk review, and KIIs found evidence of achievement (see the red IRs in Diagram 6). The Mission accomplished targets for two of four performance indicators associated with IR 3.2. The indicators showed an increase in the detection rate of tuberculosis and a decrease in the percentage of deaths from communicable disease for females aged 15 – 34. The desk review found nine pieces of evidence that the Mission has achieved IR 3.2, which is greater than the average amount of evidence supporting an IR in DO 3 (see Table 9). Furthermore, a significant number of participants in KIIs felt USAID/Liberia had increased the effectiveness of the health system. For instance, a USAID participant said, “The initiation of USAID’s government-to-government support was a big achievement. It put resources directly into the hands of GOL and helped them to strengthen health services in the counties.” The Mission made progress increasing the utilization of quality reproductive maternal, newborn, and child health services (IR 3.1). It accomplished targets for two of four performance indicators associated with IR 3.1 (see Table 8). The indicators demonstrated an increase in the 1 The Meta-Analysis normalized the raw counts of evidence that USAID/Liberia achieved the DOs (29 for DRG, 19 for EG, 25 for Health, and 14 for Education) to account for the differences in the number of documents related to each DO. This adjusted the values to a common scale by assigning weights and generated percentages to more easily compare the DOs. 26% 15% 35% 25% DRG EG Health Education Average: 25% Utilization-Focused The findings for Question 1 describe positive and negative outliers, which are anticipated sessions of USAID’s Portfolio Review and CDCS Reflection Exercise in November 2017. 10 proportion of children treated for malaria within 24 hours of the onset of a fever and a decrease in the number of women with HIV. The desk review found seven pieces of evidence that the Mission has increased the utilization of reproductive health services, which is greater than the average amount of evidence for an IR in DO 3 (see Table 9). For instance, the Fixed Amount Reimbursement Agreement (FARA) Health Outputs Final Evaluation Report states, “The FARA program provides non-cash incentives to motivate [trained traditional midwives] to refer pregnant patients to local facilities. This has increased the number of in-facility deliveries with skilled attendants, and significantly reduced home deliveries.” 2 The Mission made relatively less progress improving GOL capacity to control infectious diseases (IR 3.3) and increasing access to safe water and sanitation (IR 3.4). Participants in KIIs mentioned the success of USAID/Liberia’s emergency response to EVD but did not suggest it had made progress controlling other infectious diseases. Although the Mission accomplished targets for two of three indicators associated with IR 3.4, the desk review and KIIs did not find corroborating evidence of progress toward this IR. DO 1: DRG USAID/Liberia made progress supporting more effective, accountable, and inclusive governance (DO 1: DRG). In the desk review, 29 pieces of evidence were found that the Mission has achieved DO 1: DRG, which is greater than the average amount of evidence for a DO (see Table 9). When raw counts of evidence are normalized, 26 percent of all evidence supports that USAID/Liberia has achieved DO 1: DRG (see Chart 7). This is approximately the average percentage of evidence for a DO. However, USAID/Liberia accomplished targets for only one of three performance indicators associated with DO 1: DRG (see Table 8). These indicators showed an increase in the perception of corruption and a lower score in the World Bank’s Governance Indicators. USAID/Liberia made progress ensuring public resources are managed more transparently and accountably (IR 1.1). IR 1.1 represents a positive outlier, where all three methods found evidence of achievement (see red IRs in Diagram 6). The Mission accomplished targets for two of three 2 LSA. Liberia FARA Final Evaluation: Health Outputs Report. May 2017. 8. Table: Performance Indicator Data from the Numerical Review # Indicators # Achieved Target (+/- 10%) % Indicators Achieved Target Achievement (≥ 50% indicators achieved target) DOs DRG 3 1 33% No EG 6 2 33% No Health 4 3 75% Yes Education 7 3 43% No IRs IR 1.1 3 2 67% Yes IR 1.2 3 1 33% No IR 1.3 3 3 100% Yes IR 1.4 3 1 33% No IR 2.1 4 2 50% Yes IR 2.2 4 2 50% Yes IR 2.3 4 1 25% No IR 3.1 4 2 50% Yes IR 3.2 4 2 50% Yes IR 3.3 3 1 33% No IR 3.4 3 2 67% Yes IR 4.1 3 2 67% Yes IR 4.2 7 1 14% No IR 4.3 3 1 33% No Gender 8 1 13% No Youth 12 5 42% No 11 performance indicators associated with IR 1.1 (see Table 8). Its assistance contributed to an increase in the number of USG-supported anti-corruption measures enacted and a higher score in the International Monetary Fund’s Public Expenditure and Financial Accountability assessment. The desk review found 13 pieces of evidence that USAID/Liberia has achieved IR 1.1, which is greater than the average amount of evidence for an IR in DO 1 (see Table 9). For example, the Liberia Public Sector Modernization Project (PSMP) Aide Memoire reports, “The outcome of this activity is that there is now a harmonized personnel listing with the payroll. All seven participating ministries now undertake monthly validation of the payroll...”3 In addition, a significant number of participants in KIIs felt USAID/Liberia had supported public resources to be used more transparently and accountably. For instance, a USAID participant said, “Our work in public financial management and administration was a success. We helped to establish procurement payment and asset management systems, increasing the integrity of public-sector institutions.” The Mission made relatively less progress improving policies, models, and providers of access to justice (IR 1.2); strengthening capacity to implement election processes (IR 1.3); and, assisting civil society and the media to exercise advocacy and oversight roles (IR 1.4). The desk review found 10 pieces of evidence that USAID/Liberia has achieved IR 1.2, which is greater than the average amount of evidence for an IR in DO1 (see Table 9). However, the numerical review and KIIs did not find similar evidence of progress. The Mission accomplished targets for all three performance indicators associated with IR 1.3 (see Table 8), demonstrating an increase in the number of citizens reached through voter education and an increase in the numbers of election observers and officials trained. Yet, the desk review and KIIs did not find evidence of outcome-level results to support these outputs. For IR 1.4, a significant number of participants in KIIs felt civil society and the media had begun to exercise advocacy and oversight roles (see Table 10). However, the numerical review and desk review did not confirm this. DO 4: Education USAID/Liberia made progress better educating Liberians (DO 4: Education), according to findings from 3 USAID Mission to Liberia. Liberia PSMP – Aide Memoire. November 2016. 4 In the desk review, USAID/Liberia achieved a DO or IR when the frequency of evidence for that DO or IR was greater than the average number of evidence for the group. For example, USAID has achieved IR 1.1 because 13 pieces of evidence are greater than 9.5. The desk review found evidence that USAID/Liberia has achieved a DO or IR when the team identified empirical facts confirming this. The team only included facts generated by sound methodologies with the ability to attribute outcomes to the Mission. It did not consider documents with nonscientific methodologies. For example, the team would not have used a quarterly report stating an IP had trained individuals, but it may have used a performance evaluation with a mixed-methods approach. 9. Table: Data from the Desk Review4 # Evidence of Achievement Average # Evidence for Group Achievement (> average) DOs DRG 29 21.75 Yes EG 19 21.75 No Health 25 21.75 Yes Education 14 21.75 No IRs IR 1.1 13 9.5 Yes IR 1.2 10 9.5 Yes IR 1.3 6 9.5 No IR 1.4 13 9.5 Yes IR 2.1 2 6.67 No IR 2.2 2 6.67 No IR 2.3 16 6.67 Yes IR 3.1 7 6 Yes IR 3.2 9 6 Yes IR 3.3 3 6 No IR 3.4 5 6 No IR 4.1 7 5 Yes IR 4.2 6 5 Yes IR 4.3 2 5 No 12 the research methods. In the desk review, the team found 14 pieces of evidence that the Mission has achieved DO 4: Education, which is less than the average amount of evidence for a DO (see Table 9). However, when raw counts of evidence are normalized, 25 percent of evidence supports that USAID/Liberia has achieved DO 4: Education, which is equal to the average percentage of evidence for a DO (see Chart 7). The Mission accomplished targets for only three of seven performance indicators associated with DO 4: Education (see Table 8). The indicators showed an increase in the proportion of students who demonstrate they can read by the end of grade two, an increase in the primary net enrollment rate, and a decrease in the number of repeaters in primary school. On the other hand, there was a decrease in the gross primary completion ratio, a decrease in girls’ gross enrollment in secondary school, and a decrease in the gender parity index for primary and for secondary schools. The Mission increased equitable access to education in Liberia (IR 4.1), according to findings from the Meta-Analysis. IR 4.1 represents a positive outlier, where all three methods found evidence of achievement (see red IRs in Diagram 6). USAID/Liberia accomplished targets for two of three performance indicators associated with the IR (see Table 8). It demonstrated an increase in the number of teachers trained to support learner’s psychological wellbeing and an increase in the individuals from disadvantaged groups accessing tertiary education programs. Furthermore, the desk review found 7 pieces of evidence that USAID/Liberia has achieved IR 4.1, which is greater than the average amount of evidence for an IR in DO 4 (see Table 9). For example, the Education Quality and Access in Liberia (EQUAL) Final Performance Evaluation reported: “The back-to-school campaigns and the community visits contributed the most to an increase in enrollment, and the continuous efforts by all stakeholders with students and parents about the importance of school attendance had a positive effect on attendance.”6 A significant number of participants felt USAID/Liberia had increased equitable access to education, according to KIIs. One donor participant said, “One of USAID’s greatest achievements in Liberia has been its support for female students. Through mentoring, more girls have enrolled, and more are passing. Very few are dropping out.” USAID/Liberia made relatively less progress improving the quality of education in Liberia (IR 4.2). The desk review found 6 pieces of evidence for IR 4.2, which is greater than the average amount of evidence for an IR in DO 4 (see Table 9). A significant number of participants in KIIs expressed that 5 USAID/Liberia achieved a DO or IR when the number of informants saying the DO or IR was accomplished is greater than or equal to the average number of informants for the group. For example, KIIs suggested USAID has achieved IR 1.1 because three informants are equal to three, the average number of informants for DO 1. 6 LSA. EQUAL Final Performance Evaluation: Final Report. July 2017. 10. Table: Data from KIIs5 # Informants Stating Achieved Average # Informants for Group Achievement (≥ average) DOs DRG 8 9 No EG 6 9 No Health 15 9 Yes Education 7 9 No IRs IR 1.1 3 3 Yes IR 1.2 2 3 No IR 1.3 1 3 No IR 1.4 6 3 Yes IR 2.1 1 2 No IR 2.2 2 2 Yes IR 2.3 3 2 Yes IR 3.1 2 5 No IR 3.2 8 5 Yes IR 3.3 7 5 Yes IR 3.4 3 5 No IR 4.1 5 3 Yes IR 4.2 3 3 Yes IR 4.3 1 3 No 13 USAID/Liberia had increased the quality of education through its assistance (see Table 10). However, the numerical review did not corroborate these findings. The Mission accomplished targets for only one of seven performance indicators associated with the IR 4.2 (see Table 8). It failed to reach its targets for the number of textbooks and materials provided with USG assistance, the pupil-teacher ratios in primary and secondary schools, the number of educators and administrators to complete professional development activities, and the number of teachers to complete in-service training with USG support. For information on IR 4.3, please see Question 1.b below. Question 1.b: What has not been achieved? DO 4: Education, Continued The three research methods corresponded and found USAID/Liberia has not achieved IR 4.3 (see the white IR in Diagram 1). IR 4.3 represents a negative outlier, where the numerical review, desk review, and KIIs failed to demonstrate evidence of achievement. USAID/Liberia accomplished targets for one of three performance indicators for IR 4.3 (see Table 8). It failed to achieve targets for the number of laws or policies developed to improve primary reading programs and for the number of parent teacher associations (PTAs) supported. Furthermore, the desk review found only two pieces of evidence of progress (see Table 9). The literature noted USAID/Liberia had failed to develop standards and positive incentives for good governance in the education system. Few participants in KIIs felt USAID/Liberia had improved governance in the education system (see Table 10). For example, a donor participant said, “There’s a gap between the students and institutions in education, and USAID is not addressing it. It’s not putting its foot down in the policy dialogue. If USAID does not insist on making officials answerable, who at the Ministry will do so?” DO 2: EG As compared to other DOs, USAID/Liberia experienced relatively less progress supporting sustained, market-driven economic growth to reduce poverty (DO 2: EG). In the desk review, 19 pieces of evidence were found that the Mission has achieved DO 2: EG, which is less than the average amount of evidence for a DO (see Table 9). Additionally, when raw counts of evidence are normalized, only 15 percent of evidence supports that USAID/Liberia has achieved the DO 2: EG (see Chart 7). This is below the average percentage of evidence for a DO. The Mission accomplished targets for only two of six performance indicators associated with DO 2: EG (see Table 8). During the CDCS, Liberia experienced a decrease in daily per capita expenditures, a decrease in the annual gross domestic product (GDP) growth rate, a decrease in agriculture’s share of GDP, and a decrease in the ratio of female-to-male labor force participation. However, Liberia did improve in the World Bank’s Ease of Doing Business Rankings, and there was a decrease in the percentage of people living on less than $1.25 per day. A significant number of participants felt USAID/Liberia had not contributed to economic growth and reducing poverty during the current CDCS, according to KIIs (see Table 10). USAID/Liberia made relatively little progress in enhancing food security (IR 2.1). It accomplished the targets for two of four performance indicators associated with the IR 2.1. The Mission decreased the prevalence of households with moderate or severe hunger and increased the gross margin per unit of land. However, these achievements were undercut by an increase in the prevalence of underweight children under the age of five and a decrease in the Feed the Future Women’s Empowerment in Agriculture Index. Few participants in KIIs felt USAID/Liberia had success enhancing food security in Liberia (see Table 10). For example, a USAID participant said, “[The Food and Enterprise Development (FED) Program] did not focus on marketing, and this really affected the rice value chain. The Ministry of Agriculture did not buy the rice at such a high price.” USAID/Liberia made more progress managing natural resources sustainably (IR 2.2.) and fostering an enabling environment supporting private enterprise growth (IR 2.3). It accomplished targets for two of 14 four performance indicators associated with the IR 2.2 (see Table 8), increasing the number of hectares showing improved biophysical conditions and increasing the number of people benefitting from sustainable natural resource management. Yet, the Mission did not accomplish its target for the number of policies adopted to support biodiversity and for the number of institutions with the capacity to improve environmental awareness. Regarding IR 2.3, the desk review found 16 pieces of evidence for IR 2.3, which is greater than the average amount of evidence for an IR in DO 2 (see Table 9). USAID/Liberia made progress toward facilitating access to credit for small and medium-sized enterprises (SMEs) and linking them into the supply chains of international and national buyers. For instance, the Investing for Business Expansion (IBEX) and Sustainable Markets in Liberia (SMI-L) Midterm Performance Evaluation noted, “The evaluation team asked the sample of SMI‐L clients about the impact of the program in terms of sales and employment… 31 percent reported a positive contribution to sales growth, while 23 percent of respondents noted that they had increased staff as a result of SMI‐L support.” 7 Gender and Youth The three methods found USAID/Liberia had made limited progress supporting gender equality and youth development. The Mission accomplished targets for one of seven performance indicators for gender (see Table 11), and five of 12 performance indicators for youth (see Table 8). The indicators for youth used by USAID/Liberia are primarily DO 4: Education indicators that fail to measure key intended results from USAID’s Policy on Youth in Development, such as youth access to economic and social opportunities, participation in democratic and development processes, and role in local and national institutions. 11. Table: USAID/Liberia’s Gender Indicators Indicator Baseline Endline Achieved Progress Year Baseline Year Target Result DO 2 Ratio of female to male labor force participation rate 2012 90.2% 2016 90.8% 90.5% No Higher is better DO 3 Lifetime risk of maternal death (%) 2012 4.0% 2015 3.7% 3.6% Yes Lower is better IR 3.1 Percent of deliveries in health facilities assisted by skilled attendants 2013 66.0% 2014 68.0% 58.0% No Higher is better IR 3.1 Number of pregnant women with known HIV status 2014 53,257 2017 29,685 64,428 No Lower is better DO 4 Girls' gross enrollment in secondary 2011 36.9% 2015 37.3% 32.5% No Higher is better DO 4 Gender parity index, primary, total 2011 0.916 2015 0.936 0.904 No Higher is better DO 4 Gender parity index, secondary, total 2011 0.82 2015 0.840 0.776 No Higher is better Furthermore, KIIs suggested USAID/Liberia has made limited progress supporting gender and youth. Approximately one-third of participants felt USAID/Liberia should improve its support for gender equality and women’s empowerment. Around half of participants believed USAID/Liberia should enhance its assistance for a safe and productive transition of youth to adulthood. The Meta-Analysis considers these findings significant because KIIs did not ask questions specific to gender or youth. Participants introduced these crosscutting themes without prompting or probing. Graphic 12 below 7 International Development Group LLC. Mid-term Performance Evaluation: IBEX and SMI-L. February 2016. 15 features impactful and representative quotes from participants in KIIs about gender and youth. 12. Graphic: Impactful Quotes about Gender and Youth The desk review found a lack of progress by USAID/Liberia on gender equality. As shown in Chart 13, negative themes about gender found in documents, including women lack resources, gender-based violence, and women feel overburdened, outnumbered positive themes, such as the emergence of female role models, increased public freedom for women, and increased economic empowerment for women. Chart 13 denotes negative themes with (-) and positive themes with (+). 13. Chart: Frequency of Gender-related Findings from the Desk Review by Theme Question 1.c: What gaps exist? The Meta-Analysis found gaps in monitoring by USAID/Liberia during the current CDCS. The numerical review used 74 performance indicators to measure USAID/Liberia’s progress against its results framework. Of these, 37 indicators originated from the CDCS and PMP, and 37 came from other sources, such as PIDs, PPRs, AMELPs, and IP reports. The team preferred to use indicators from the CDCS and PMP, but it had to seek them from these other sources because the USAID/Liberia lacked 12 8 7 3 3 3 Women lack resources (-) Gender-based violence (-) Women feel overburdened (-) Female role models emerge (+) Increased public freedom for women (+) Increased economic empowerment for women (+) 16 baselines, targets, and results for at least 37 indicators. As shown in Diagram 14, USAID/Liberia approved its PMP in November 2016, three years after the CDCS was approved. USAID’s Operational Policy Automated Directives System (ADS) 201 states a Mission must approve its PMP within six months of CDCS approval. The approval of the PMP also occurred 11 months after the Mission awarded most activities for the current CDCS. The period between the approval of the PMP and data collection for the Meta-Analysis was less than a year. 14. Diagram: Timeline of Milestones During the Current CDCS Most participants in KIIs felt USAID/Liberia should strengthen its internal capacity and promote accountability. Informants cited gaps in project management from staff turnover, ineffective knowledge management systems, lengthy internal approval processes, and overburdened workloads. For example, a USAID participant said, “Monitoring was a shortcoming during this CDCS. As an AOR, I have too many priorities, too many taskers. We need to close this monitoring gap. We need more accountability.” A donor participant described one outcome of these shortcomings, “This lack of accountability means that broken programs continue to be implemented, even after we know they are broken.” QUESTION 2: HOW HAVE USAID’S DEVELOPMENT OBJECTIVES CONTRIBUTED TO ACHIEVING USAID’S STRATEGIC GOAL? USAID/Liberia’s DOs contributed to achieving the strategic goal in line with the logic model from the results framework. However, the progress made by the Mission towards one DO was disconnected from its progress on other DOs. In general, DOs in the CDCS were not integrated. Cause and Effect Relationship A positive correlation exists between evidence found by the numerical review, desk review, and KIIs. This positive correlation behaves like a cause and effect relationship, where IRs contribute to DOs, and DOs contribute to the strategic goal. The CDCS posits a similar cause and effect relationship as the logic model for its results framework. As shown in Graph 15, when the team found relatively more evidence that USAID achieved a DO using one method, it also found relatively more evidence using the other two methods. The slope of the trendline (the dotted line in Graph 15) illustrates this positive correlation. Small values of the X axis correspond to small values of the Y axis and to small bubble sizes. Whereas, large values of the X axis correspond to large values of the Y axis and to large bubble sizes. For example, for DO 3: Health, USAID/Liberia accomplished a high number of indicator targets, which corresponds to a high number of participants in KIIs perceiving USAID/Liberia achieved the DO and to a high amount of evidence supporting the DO in the desk review. 17 15. Graph: Positive Relationship between DOs and Evidence of Achievement Disconnected DOs The desk review found that evidence supporting one DO is disconnected from evidence supporting another DO. This means that the positive correlation from Graph 15 applies only within a DO and its supporting IRs. For example, progress towards IR 1.1 contributes to DO 1: DRG, but progress towards IR 1.1 does not contribute to DO 2: EG. 16. Matrix: Evidence Co-Occurrence from Desk Review DO 1 IR 1.1 IR 1.2 IR 1.3 IR 1.4 DO 2 IR 2.1 IR 2.2 IR 2.3 DO 3 IR 3.1 IR 3.2 IR 3.3 IR 3.4 DO 4 IR 4.1 IR 4.2 IR 4.3 DO 1 1 2 1 IR 1.1 1 1 1 IR 1.2 2 1 IR 1.3 1 IR 1.4 1 1 1 DO 2 2 2 17 IR 2.1 2 1 IR 2.2 2 1 IR 2.3 17 1 1 DO 3 6 9 5 2 IR 3.1 6 IR 3.2 9 IR 3.3 5 IR 3.4 2 DO 4 7 6 2 IR 4.1 1 7 1 IR 4.2 6 1 IR 4.3 1 2 8 7 10 7 0 2 4 6 8 10 12 14 16 18 0 5 10 15 20 25 30 35 No. of Participants Who Felt a DO Was Achieved (KIIs) No. of Findings that USAID Achieved a DO (Desk Review) Numerical Review vs. Desk Review vs. KIIs No. of DO Indicator Targets Accomplished (Numerical Review) DRG EG Health Education 18 Matrix 16 shows how often DO and IR achievements overlapped in studies and reports. The numbers in the matrix are the number of documents that have evidence of multiple DOs or IRs being achieved. For example, as seen in the above left of Matrix 16, the team found one document had evidence of USAID/Liberia achieving DO 1: DRG and had evidence of the Mission achieving IR 1.1. The team found two documents had evidence of USAID/Liberia achieving DO 1: DRG and had evidence of the Mission achieving IR 1.2, etc. Grey squares indicate where a co-occurrence in an overlapping document is inconsequential. For example, finding evidence of DO 1: DRG being achieved in a document that also has evidence of DO 1: DRG being achieved is insignificant. A cluster pattern emerges in Matrix 16, indicating IR achievements within a DO largely overlapped other IR achievements within that DO. Only in two cases did IR achievements overlap with IR achievements outside their respective DOs. The team found one document had evidence of USAID/Liberia achieving IR 1.1 and had evidence of USAID/Liberia achieving IR 4.3. Additionally, it found one document had evidence of USAID/Liberia achieving IR 1.4 and had evidence of USAID/Liberia achieving IR 4.1. These cases are the two outliers on either side of the central cluster pattern in Matrix 16. Integrated DOs The Desk Review found USAID typically establishes non-integrated DOs in its CDCSs globally. A non￾integrated DO focuses on one technical area, such as education or health. Whereas, an integrated DO involves several technical areas. For instance, an integrated youth DO may have an education component for skills development and an economic growth component for employability. As shown in Chart 17, when reviewing 59 CDCSs available on www.usaid.gov, 41 (70 percent) have DOs that are not integrated, and 18 (30 percent) have at least one DO that is integrated. Eight percent of CDCSs have an integrated DO supporting youth development. Four percent have an integrated DO on STIP, and three percent have an integrated DO supporting gender equality. 17. Chart: Percentage of Integrated and Non-Integrated DOs in CDCSs Globally KIIs found mixed support for altering USAID/Liberia’s strategic goal and some support for an integrated DO. Approximately half of participants felt USAID should change its strategic goal for Liberia in its next strategic plan. A significant number of participants, 15 out of 26 who raised the issue of integration, felt USAID should adopt an integrated DO in the next strategic plan, according to KIIs. 70% 8% 3% 7% 12% Non-Integrated DOs Integrated Youth DO Integrated Gender DO Integrated STIP DO Other Integrated DO 19 QUESTION 3: TO WHAT EXTENT WERE THE PROGRAMMATIC AND CONTEXTUAL ASSUMPTIONS IDENTIFIED IN USAID’S DEVELOPMENT HYPOTHESIS SUFFICIENT TO ACHIEVE ITS STRATEGIC GOAL? This section describes the most significant findings related to 25 assumptions from the current CDCS, including: six strategic choices, five considerations, five opportunities, seven challenges, and two game changers. Findings suggested that most assumptions from the development hypothesis were sufficient to achieve the strategic goal. However, themes emerged that challenge five of the assumptions. Sufficient Assumptions Most participants in KIIs believed USAID/Liberia correctly focused on the decentralization of government services. They also felt GOL had failed to devolve power during the period. For example, a USAID participant said, “The GOL never passed the decentralization legislation. You see, it’s all linked. Land rights is linked to decentralization, which is linked to agriculture, which is linked to this country feeding itself. Without decentralization, this won’t happen.” Approximately two-thirds of evidence related to decentralization from the desk review indicated that GOL had failed to support it. The desk review found evidence of multiplier or spillover effects from the development corridor to neighboring counties. This occurred in activities supporting natural resource management and rural road development under DO 2: EG. However, according to KIIs, most participants felt USAID should expand its geographic focus beyond the development corridor in its next strategy. For example, a USAID participant said, “We need a way to get around the geographic corridor and to work outside the six counties… the southeast is completely cut off from our assistance.” Most participants in KIIs felt USAID/Liberia successfully locked in reforms using civil society and community structures. USAID’s Civil Society Organization (CSO) Sustainability Scores for Liberia remained constant at 4.8 from 2010 to 2015.8 In the desk review, half the evidence suggested CSOs succeeded in locking in reforms and half intimated CSOs had failed at this. The research methods did not produce significant findings on the two-track approach of long-term development and short-term service delivery. Most participants in KIIs felt USAID correctly focused on strengthening HICD. Context indicators measuring HICD remained constant during the CDCS period. The World Bank’s Worldwide Governance Indicators for Government Effectiveness stayed constant at a percentile rank of 8 from 2013 to 2016. The UNDP’s Human Development Index for Liberia also remained constant at 0.43 from 2012 to 2014. The desk review found evidence of a need for additional capacity building support to governmental and nongovernmental institutions in the new strategy for FY 2019 – FY 2023. Most participants in KIIs felt USAID continued to have unprecedented access and experience in Liberia during the CDCs period. For instance, one GOL participant said, “USAID is the voice we seek the most when we consider our policies.” Regarding civil peace and security, the Knoema Global Peace Index for Liberia remained relatively constant, moving from 2.01 to 2.02 from 2014 to 2017. Few participants in KIIs felt Liberia was likely to experience renewed conflict during the next strategic period. Findings supported the assumptions of social divisions and distrust between groups due to inequality, the existence of a large youth cohort, and a growing population. The Gini Coeffient, a measure of the deviation in the distribution of household income, increased from 36.5 in 2007 to 38.2 in 2017, 8 The CSO Sustainability Index measures seven dimensions: legal environment, organizational capacity, financial viability, advocacy, service provision, infrastructure, and public image. 20 indicating increased inequality. Net overseas development assistance per capita in Liberia increased from $125 in 2013 to $243 in 2015. The population growth rate and youth’s share of the population continued to grow, according to the UN Department of Economic and Social Affairs (see Chart 18). The decrease in GDP growth rate and a decline in the price of iron ore and rubber emphasized the challenge of Liberia’s low starting point for post-conflict recovery. 18. Charts: Commodity Prices and GDP; and Population Growth Challenged Assumptions As shown in Diagram 19, most participants in KIIs felt USAID/Liberia should strengthen its internal capacity. Informants cited gaps in project management from staff turnover, ineffective knowledge management systems, lengthy internal approval processes, and overburdened workloads within the Mission. For example, staff from an IP said, “They still don’t know where information is at the Mission. We’re doing all this sharing of lessons learned. But, what do they do with it?” Findings from the Meta-Analysis challenged the assumption that feedback loops enable one DO to support another DO. The desk review found that evidence of USAID/Liberia achieving one DO is disconnected from evidence of it achieving another DO (see Matrix 16). For example, progress towards IR 1.1 contributes to DO 1: DRG, but this progress does not contribute to DO 2: EG. Most participants in KIIs felt USAID should adapt to the development priorities of the next GOL following the general election in 2017. They often expressed this when commenting on whether the positive changes since 2005 would continue. Informants expressed positive and negative views about the change in GOL. For example, a USAID participant said, “The new government is a golden opportunity. It’s time to reshape how USAID interacts with GOL. We can start a new discussion and set new priorities.” And, a donor participant expressed, “The transition in leadership will be a complicated one. Continuity will be difficult. GOL will need USAID support more than ever.” Most participants in KIIs believed USAID should enhance its support for anticorruption initiatives. They believed corruption has undercut the inclusivity and sustainability of development in Liberia. Informants raised the issues of entrenched political elites, special interests, economic monopolies, weak oversight and advocacy by CSOs, and a lack of transparency in government. A USAID participant said, “The current CDCS only nibbles at corruption. It’s there, but anticorruption programming is not always explicit. Many of these programs fail, and it was not always diplomatic to bring up these failures when President Sirleaf was in government.” 0 2 4 6 8 2010 2015 2020 2025 2030 Millions Total Population Total Under 25 8.7% 0.7% 0% -1.6% -5% 0% 5% 10% $0 $50 $100 $150 2013 2014 2015 2016 Price of Iron Ore (USD/ton) Price of Rubber (USD/lb) Annual GDP growth rate 21 19. Diagram: Challenges to Assumptions to the Development Hypothesis The opportunity of a rich base of natural resource wealth was dashed by decreasing GDP and a decline in the price of iron ore and rubber (see Chart 18). Most participants in KIIs felt USAID/Liberia needed to enhance the sustainability of its assistance. The World Bank’s Country and Policy Institutional Assessment Index for policies for social inclusion and equity has remained constant at 3.0 from 2012 to 2016. Challenged Assumptions Affect DOs Challenges to the assumptions affected each DO differently. The effects of these challenges may partially account for the mixed progress made by USAID/Liberia toward its DOs and IRs (see Question 1). As shown in Table 20, the Mission’s internal capacity negatively affected DO 2: EG, DO 3: Health, and DO 4: Education more than DO 1: DRG, according to the Desk Review and KIIs. Differences in the staffing and experience of technical teams may account for this divergence. Most references to corruption concerned its negative affect on DO 2: EG. The Desk Review cited the effects of oligopoly markets, dominance of cartels in several sectors, lack of competition between suppliers, and weak enforcement mechanisms. The low economic growth during the CDCS solely affected DO 2: EG, and participants in KIIs felt it had undercut the Mission’s progress toward reducing poverty. 22 20. Table: Percentage of Evidence of Challenges to Assumptions by DO, Normalized by the Number of Desk Review Documents and KIIs Lack of Mission Capacity Corruption New GOL in 2018 Low Economic Growth Disconnected DOs EVD DO 1: DRG 8% 20% 19% 0% 8% 17% DO 2: EG 30% 60% 29% 100% 30% 30% DO 3: Health 38% 13% 29% 0% 0% 6% DO 4: Education 24% 7% 24% 0% 62% 47% The challenge of a new GOL in 2018 concerned DOs equally. Participants in KIIs expressed a desire to meet with the new government and discuss their agenda, and concern about the potential for a lack of continuity between the development priorities of the new and old GOL. Disconnect DOs challenged DO 4: Education and DO 2: EG more than other DOs. Their experience implementing integrated activities and supporting cross-cutting themes may account for this difference. For example, DO 4: Education, DO 2: EG, and DO 3: Health co-funded the Advancing Youth Project (AYP). The Desk Review and KIIs suggested EVD negatively influenced DO 1: DRG, DO 2: EG, and DO 4: Education more than DO 3: Health. DRG, EG, and Education activities had to redesign their assistance to support the emergency response to EVD during the CDCS. This disrupted progress toward their DOs and IRs. Health activities received more funding because of the EVD outbreak and were designed in part to help control infectious diseases, like Ebola. QUESTION 4: WHAT WERE THE UNINTENDED OUTCOMES OF USAID’S DEVELOPMENT ASSISTANCE UNDER THE CURRENT CDCS FOR FY 2013 – FY 2018? The desk review and KIIs found positive and negative unintended outcomes from USAID’s development assistance under the CDCS. As a positive unintended outcome, a multiplier or spillover effect occurred where some development results spread to neighboring counties from USAID/Liberia’s development corridor of Bong, Grand Bassa, Lofa, Margibi, Montserrado, and Nimba counties. This occurred in activities supporting rural road development under DO 2: EG and strengthening the health system under DO 3: Health. An Assessment of Farm-to-Market Roads claims, “A majority of participants in [focus group discussions] in Bong, Lofa, and Nimba indicated transportation costs had decreased to reach neighboring counties following the rehabilitation of roads.”9 In addition, the FARA Health Outputs Final Evaluation Report states, “The evaluation team observed strengthened components of health systems, particularly improvements in critical risk areas and spillovers to non-FARA teams and facilities.”10 Additionally, the desk review found USAID/Liberia’s assistance had improved life in an unintended way for some beneficiaries. This positive unintended outcome occurred in activities supporting food security 9 LSA. Qualitative Assessment of Farm-to-Market Road Rehabilitation. August 2016. 10 Intermedia. Assessment of FARA 1.0 Outcomes on the Capacity of MOH and CHTs, and the Quality of Service Provision at Health Facilities in Liberia. April 25, 2017. 23 under DO 2: EG and strengthening the health system under DO 3: Health. For example, the FED Program’s Impact Survey from 2017 states, “Beneficiaries are significantly happier and more optimistic than comparison farmers. FED beneficiaries are more likely to… see a brighter future for themselves and their families.”11 Regarding negative unintended outcomes, the desk review and KIIs found evidence that one socio￾economic group received lopsided benefits and that livelihoods were lost by non-target groups. For instance, a GOL participant said, “USAID trained these teachers and then demanded that their teachers get jobs. These teachers jumped the line and passed teachers who were trying to access the education system the normal way. USAID crowded out the normal processes of the government.” Also, the People, Rules, and Organizations Supporting the Protection of Ecosystem Resources (PROPSER) Performance Evaluation report states, “Youth are often a high percentage of those who have been farming in the forest. When communities restrict those activities, the youth, in particular, may not have access to alternate land…. Youth who perceive (whether real or not) increased barriers to their livelihood efforts may be at risk of anti-social or criminal behavior…”12 The desk review found evidence of women losing their traditional occupations to men. For instance, the Small Holders Oil Palm Project Support (SHOPS) I Midterm Evaluation states, “The anticipated trend is a decrease in the proportion of women to men [Freedom Mill 2] clients. It is likely that due to the profitability of the activity, more men are becoming involved… In many instances, men have overtaken women’s economic activities as they became more lucrative.” 13 Several participants in KIIs noted instances of women losing their traditional occupations to men. For example, a USAID participant said, “More men are becoming midwives now that it pays well and there’s a training program. Typically, women were the midwives, but they’ve been displaced.” CONCLUSIONS QUESTION 1: TO WHAT EXTENT HAS PROGRESS BEEN MADE IN ACHIEVING USAID’S STRATEGIC GOAL FROM THE CURRENT CDCS FOR FY 2013 – FY 2018? USAID/Liberia has made progress toward its strategic goal of strengthening Liberian institutions positioned to drive inclusive economic growth and poverty reduction. This progress has been mixed with some DOs experiencing more progress than others, and achievements have been unevenly distributed in certain IRs. Given the short period since USAID awarded activities under the current CDCS and approved its PMP, more evidence of outcomes will likely emerge over time. USAID/Liberia’s achievement is most pronounced in improving the health status of Liberians (DO 3: Health). It has increased the effectiveness of the health system at the national and county levels and has enjoyed success increasing the utilization of quality reproductive, maternal, and child health services. The Mission experienced relatively less progress improving GOL capacity to control infectious diseases and increasing access to safe water and sanitation. The outbreak of EVD benefited and challenged USAID/Liberia’s health assistance. It resulted in additional funding and 11 International Development Group LLC. FED Project Impact Survey. February 2017. 12 Liberia Strategic Analysis. Performance Evaluation of PROSPER. July 2016. 13 Bishop, Catherine. SHOPS I Midterm Impact Evaluation. Winrock. May 2013. Utilization-Focused The conclusions for Question 1 can be directly applied to the Past Results, Lessons, and Stocktaking section from the CDCS Concept Note, which is a requirement for Missions in Phase 1 of the strategic planning process. 24 resources, but EVD overwhelmed the capabilities of the Liberian health system. USAID/Liberia made progress towards more effective, accountable, and inclusive governance (DO1: DRG). It experienced success assisting Liberia to manage public resources more transparently and accountably. The Mission made relatively less progress in improving policies, models, and providers of access to justice; strengthening capacity to implement election processes; and assisting civil society and the media to exercise advocacy and oversight roles. In addition, USAID/Liberia made progress fostering better educated Liberians (DO 4: Education). It increased equitable access to education, and to a lesser degree, improved the quality of education. However, USAID/Liberia did not make significant progress in improving the governance of the education system and experienced a lack of success outside supporting PTAs. Compared to other DOs, USAID/Liberia experienced relatively less progress supporting sustained, market-driven economic growth to reduce poverty (DO 2: EG). It assisted with managing natural resources more sustainably and supporting an enabling environment for private enterprise growth. However, progress toward these results did not translated into achieving outcomes at the DO level, and the Mission made little notable progress enhancing food security in Liberia. USAID/Liberia experienced limited progress supporting gender equality and youth development. It did not significantly influence women’s participation in the labor force or improve gender parity in primary and secondary education. The Mission’s assistance to youth also did not demonstrate notable achievements. Needs went unaddressed in preparing youth with the competencies to become informed and productive citizens, and to contribute to their communities’ development. A lesson learned was that USAID/Liberia failed to systematically track performance indicators from its CDCS and PMP during FY 2013 – FY 2018. The Mission identified indicators in its strategic planning and project design to measure DOs and IRs, but it did not develop baselines and targets for many of these indicators in the PMP. It did not ensure IPs collected data for these indicators, resulting in a lack of data for decision making. Issues with the Mission’s internal capacity also emerged during the CDCS. Consequently, USAID/Liberia was unable to have manage adaptively and promote accountability. QUESTION 2: HOW HAVE USAID’S DEVELOPMENT OBJECTIVES CONTRIBUTED TO ACHIEVING USAID’S STRATEGIC GOAL? USAID/Liberia’s DOs contributed to achieving the strategic goal per the cause and effect logic model from its results framework. Based on the available evidence, a causal relationship exists between the IRs and DOs. USAID/Liberia’s succeeded in achieving several IRs, and in turn, this contributed to USAID/Liberia making progress toward the DO associated with these IRs. However, USAID/Liberia made progress toward each DO independently of its other DOs. Its achievement of IRs from one DO did not overlap with IRs from other DOs. In other words, the assistance provided by the Mission’s DRG, EG, Health, and Education teams was disconnected. This lack of integration challenged the assumption to the development hypothesis that there are critical feedback loops among the DOs. USAID has developed integrated DOs in several CDCS’s globally. Integrated DOs take a cross-sectoral approach to achieving outcomes and involve more than one technical team. USAID most frequently develops integrated DOs to support gender equality, youth development, and STIP. 25 QUESTION 3: TO WHAT EXTENT WERE THE PROGRAMMATIC AND CONTEXTUAL ASSUMPTIONS IDENTIFIED IN USAID’S DEVELOPMENT HYPOTHESIS SUFFICIENT TO ACHIEVE ITS STRATEGIC GOAL? USAID/Liberia’s assumptions to its development hypothesis were sufficient to achieve the strategic goal from the current CDCS. However, conditions in Liberia challenged five assumptions, calling into question whether they apply to a new strategy for FY 2019 – FY 2023 (see Table 21). USAID/Liberia’s strategic choice to support decentralization proved sufficient. A consensus exists among USAID, GOL, other donors, IPs, and civil society for the Mission to continue to support decentralization in its new strategy. A similar consensus exists for USAID to expand its assistance beyond the development corridor in its next strategy. However, the Meta-Analysis confirmed the geographic focus was sufficient for the current CDCS. Positive unintended outcomes identified a multiplier effect, where the interventions of several USAID activities in focus counties benefitted neighboring counties. The strategic choice to prioritize addressing gender and other inequalities proved relevant during the CDCS. However, USAID/Liberia’s decision to support gender equality did not result in progress on the issue. The Mission achieved mixed success locking in reforms through civil society and community structures. Community structures proved more able to sustain reforms than CSOs. USAID/Liberia’s focus on HICD proved relevant during the current CDCS. Human and institutional capacity remained an ongoing issue, and a consensus exists among USAID and stakeholders to continue to focus on HICD in the new strategy. The security environment in Liberia did not change during the CDCS, indicating this assumption proved sufficient. As discussed in Question 2, the assumption of critical feedback loops among the DOs proved insufficient. Corruption was a prominent theme that challenged the momentum of positive change and sustainable growth. A consensus exists among USAID, GOL, other donors, IPs, and civil society to enhance anticorruption initiatives to foster greater accountability of institutions and leaders. A similar consensus is present for USAID/Liberia to continue to improve the sustainability of its development assistance in its new strategy for FY 2019 – FY 2023. The opportunity presented by USAID’s access and experience in Liberia proved sufficient during the CDCS. However, the general election in 2017 will necessitate that USAID adapt and align its priorities to those of the new GOL in 2018. Moreover, the assumption of continued international support proved sufficient. Liberia continued to receive substantial contributions from the international community. The economic situation during the CDCS challenged the assumption of Liberia’s rich base of natural resources. The decrease in commodity prices obviated the usefulness of the country’s natural resources. The challenge of social divisions and the destabilizing potential of the youth cohort and population growth rate proved sufficient as assumptions. Inequality increased in Liberia, and the population growth rate and youth’s share of the population continued to grow. The two game changers of renewed conflict and loss of confidence in President Sirleaf’s administration did not occur. 21. Table: Status of Assumptions to the Development Hypothesis Type Assumption Status for Current CDCS Next Strategic Period Strategic Choice Support for the decentralization of services Sufficient Continue Strategic Choice Geographic focus in the development corridor of the six counties Sufficient Reevaluate Strategic Choice Address gender and other inequalities Sufficient Continue 26 Strategic Choice Lock in reforms through civil society and community structures Sufficient Continue Strategic Choice Environmentally sound approaches to development No Findings No Findings Strategic Choice Two-track approach: build capacity long-term; and improve access to goods and services short-term. No Findings No Findings Considerations Feedback loops between DOs Insufficient Reevaluate Considerations Internal mission capacity provides oversight Insufficient Reevaluate Considerations Consolidate and extend civil peace and security Sufficient Continue Considerations HICD Sufficient Continue Considerations USAID Forward No Findings No Findings Opportunities Momentum of positive change since 2005 Insufficient Reevaluate Opportunities Emergent policy framework for inclusive and sustainable growth Insufficient Reevaluate Opportunities Strong and coordinated international support Sufficient Continue Opportunities Rich base of natural resource wealth Insufficient Reevaluate Opportunities USG and USAID access and experience Sufficient Continue Challenges Low starting point for post-conflict recovery Sufficient Continue Challenges Low level of government capacity Sufficient Continue Challenges Social divisions and distrust between groups Sufficient Continue Challenges Weak legitimacy of the state Sufficient Continue Challenges Destabilizing potential of the youth cohort Sufficient Continue Challenges Population growth rate Sufficient Continue Challenges Narrow window to demonstrate results No Findings No Findings Game Changers Renewed Conflict Sufficient Continue Game Changers Loss of confidence in President Sirleaf Sufficient No Findings The challenged assumptions affected each DO to a different extent. The effects of these challenges may partially account for the mixed progress made by USAID/Liberia toward its DOs and IRs (see Question 1). The economic situation during the CDCS and corruption negatively affected the Mission’s ability to foster sustained and market-driven economic growth. The lack of feedback loops between DOs detracted from the ability of the EG and Education teams to implement integrated activities. The EVD outbreak caused the Mission to restructure its DRG, EG, and Education activities to support the emergency response, which disrupted progress toward their DOs and IRs. QUESTION 4: WHAT WERE THE UNINTENDED OUTCOMES OF USAID’S DEVELOPMENT ASSISTANCE UNDER THE CURRENT CDCS FOR FY 2013 – FY 2018? USAID/Liberia’s assistance resulted in positive and negative unintended outcomes during the current CDCS. A positive unintended consequence was the multiplier effect that existed between the geographic focus on the development corridor and neighboring counties. Interventions by several USAID/Liberia activities in the focus counties benefitted neighboring counties. These occurrences did not prevent the consensus from emerging that USAID/Liberia should expand its focus beyond the six counties of the development corridor in its new strategy. In some cases, one socio-economic group received lopsided benefits and several non-target groups lost their livelihoods. These negative unintended outcomes affected activities from DO 4: Education and DO 2: EG, respectively. Furthermore, activities supporting DO 2: EG and DO 3: Health experienced women losing their traditional occupations to men when those occupations became more profitable. The lack of gender integration in these activities corresponds with USAID/Liberia’s overall lack of progress toward gender equality and female empowerment during the current CDCS. 27 RECOMMENDATIONS QUESTION 1: TO WHAT EXTENT HAS PROGRESS BEEN MADE IN ACHIEVING USAID’S STRATEGIC GOAL FROM THE CURRENT CDCS FOR FY 2013 – FY 2018? USAID/Liberia should utilize MEL to develop its new strategy for FY 2019 – FY 2023 and design its subsequent projects and activities. Evidence-based design and results-oriented management require implementing processes and procedures to apply lessons learned from monitoring and evaluation. The information produced by MEL becomes constructive when the Mission utilizes it for decision making. The CDCS Working Group should integrate MEL into the new strategy by having technical teams describe how they intend to measure their progress toward new DOs and IRs in Section C of the strategy. In Section D, the CDCS Working Group should set MEL priorities, such as developing baselines and targets for indicators and ensuring IPs collect data regularly, and technical teams should carefully select indicators to gauge the effectiveness of their assistance on DOs and IRs. In addition, USAID/Liberia should develop the Mission’s PMP to be approved within six months of the new strategy’s approval, as required by ADS 201. The PMP should describe the monitoring approach, the evaluations to be conducted, a plan for collaborating, learning, and adapting (CLA), and a list of indicators, including Foreign Assistance Resources (F) indicators. USAID/Liberia should include tools and practices, like site visits, data quality assessments (DQAs), PIDS, and learning events. In it, the technical teams should include well-researched baselines and meaningful and realistic targets. The Mission should acknowledge that the PMP will not be comprehensive when approved and should update it annually or as circumstances change in Liberia and it develops new projects and activities. USAID/Liberia should base the projects supporting its new strategy on evidence produced by MEL. The project design teams should align their theories of change with the new strategic goal and results framework in the PADs. They should define the logical relationships of how projects will affect DOs and IRs. In the PADs’ Management and Implementation Plans, the teams should describe how they intend to apply performance data to measure progress toward the project’s purpose. They should establish plans to collect this data in the PADs’ Project MEL Plans, based on the Mission’s PMP. The Mission should apply knowledge from MEL when implementing its activities. The technical teams should utilize monitoring data from PIDS, site visits, and DQAs to identify performance issues. They should address these issues in meetings with IPs and stakeholders, and develop solutions before issues impede progress toward DOs or IRs. USAID/Liberia should also use evaluations and learning events to examine problems with activity implementation and make course corrections. The Mission should encourage IPs to report progress towards outcomes in annual reports and update AMELPs annually. USAID/Liberia should continue to monitor and evaluate ongoing activities under the CDCS. Although the new strategy presents an opportunity to systematically strengthen MEL in the future, the Mission should remain committed to enhancing MEL in the current period. USAID/Liberia should complete revisions to its current PMP to integrate new and revised F indicators, as this plan will serve as a basis to develop the PMP for the new strategy. Utilization-Focused The recommendations for Question 1, 2, and 3 can be applied to the Results Framework Paper, which is a requirement for Missions in Phase 1I of the strategic planning process. 28 QUESTION 2: HOW HAVE USAID’S DEVELOPMENT OBJECTIVES CONTRIBUTED TO ACHIEVING USAID’S STRATEGIC GOAL? USAID/Liberia and the CDCS Working Group should develop integrated DOs in its new strategy for FY 2019 – FY 2023. Interconnected approaches help to address complex development challenges with multiple causes and may assist the Mission to align the new strategy with the Sustainable Development Goals, which have integrated goals and targets. It should consider two options to integrate DOs: 1. Applying a cross-cutting theme across DOs. Missions can integrate a cross-cutting theme throughout their results framework (see diagram to the right). Technical Offices are responsible for achieving their DO and IRs, while ensuring they simultaneously address the cross-cutting issue in their sector. For example, a Mission may integrate gender across its results framework. This is a traditional approach, which USAID/Liberia used to try to integrate youth, nutrition, and jobs during the current CDCS. 2. Developing an integrated DO to address a cross-cutting theme. Missions can develop an integrated DO in their results framework on a specific cross-cutting theme (see diagram to the right). Several Technical Offices combine to form a DO working team. They address the cross-cutting issue directly by achieving the related DO and IRs. For example, a Mission may establish an integrated DO on youth with an education component for skills development and an EG component for employability. Missions are increasingly electing this approach in CDCS’s. The Meta-Analysis specifically recommends developing an integrated DO (Option 2) to address a cross￾cutting theme. An integrated DO requires strong coordination between technical teams and innovative approaches to combine funding streams. Mission leadership, systematized cooperation, and constant commitment will be critical to the success of an integrated DO. The CDCS Working Group should engage in a dialogue internally and with stakeholders to decide on an integrated DO. The Mission should consider gender, youth, and STIP as potential focuses. Such an approach may help overcome the lack of progress by USAID on gender and youth during the current CDCS. Once it decides on a focus, USAID/Liberia should take a methodical approach to collaborating on the integrated DO among its technical offices, GOL, and IPs. The Mission will need to research and prepare a development context, links to the GOL policy agenda, development hypothesis, IRs, assumptions and risks, and MEL priorities to support an integrated DO in its new strategy. USAID/Liberia should consider the following as potential models for an integrated DO: the CDCS for Tanzania for FY 2014 – FY 2019, which includes one integrated DO that addresses both gender and youth; the CDCS for Rwanda for FY 2015 – FY 2019, which features an integrated DO on youth; the CDCS for Jordan for FY 2014 – FY 2017, which includes contains an integrated DO on gender; and the CDCS for Indonesia for FY 2014 – FY 2018, which includes an integrated DO for STIP. For more information on an integrated DO, please see Annex IX. QUESTION 3: TO WHAT EXTENT WERE THE PROGRAMMATIC AND CONTEXTUAL ASSUMPTIONS IDENTIFIED IN USAID’S DEVELOPMENT HYPOTHESIS SUFFICIENT TO ACHIEVE ITS STRATEGIC GOAL? USAID/Liberia and the CDCS Working Group should reevaluate the assumptions from the current CDCS and replace those that are being challenged in its new strategy. The development context in 29 Liberia has evolved since the approval of the CDCS in 2013. The Mission should consider these new conditions and determine if and how they support what it expects to achieve going forward. USAID/Liberia should take a CLA approach to reevaluating its assumptions, drawing on information from the Analytical Phase, portfolio reviews, and the Mission’s institutional memory. It should redesign the following assumptions: feedback loops between DOs; internal Mission capacity; the positive momentum that has existed in Liberia since 2005; the policy framework for inclusive and sustainable growth; and Liberia’s natural resources. The Mission should share the revised assumptions by documenting and disseminating them internally and among stakeholders to foster a better understanding of USAID/Liberia’s new strategic approach. The CDCS Working Group should develop contingency scenarios in its new strategy. Scenario planning explores a range of positive, negative, and neutral possibilities for the future of a country and develops problem solving approaches in case they arise. This can help the Mission adapt to uncertainty due to election outcomes and changes in the political situation. USAID/Liberia should consider the following as examples scenario planning: the CDCS for the Democratic Republic of the Congo for FY 2014 – FY 2019; and, the CDCS for Zimbabwe for FY 2016 – FY 2021. In its new strategy, USAID/Liberia should align its priorities with those of the new GOL, strengthen anticorruption programming, and prioritize domestic resource mobilization. To adhere to the principles of country-led development, USAID/Liberia should develop its new strategy in line with the GOL. The Mission should take advantage of the momentum of a new government to strengthen its commitment to combating corruption. It should continue to foster accountability of leaders and institutions by strengthening systems that enable rule of law and by supporting civil society and the media to provide oversight. Finally, USAID/Liberia should continue to assist with domestic resource mobilization to help Liberia take advantage its own resources and compensate for lower GDP growth rates. QUESTION 4: WHAT WERE THE UNINTENDED OUTCOMES OF USAID’S DEVELOPMENT ASSISTANCE UNDER THE CURRENT CDCS FOR FY 2013 – FY 2018? USAID/Liberia should further explore the unintended outcomes of its assistance through summative performance evaluations and whole-of-project evaluations, both during the current period and in its new strategy for FY 2019 – FY 2023. Summative or final evaluations gauge unintended consequences by analyzing baseline and endline data collection. Single point in time studies cannot do this. Whole-of￾project evaluations examine an entire project and its constituent activities, which assists with discerning unintended outcomes. Summative and whole-of-project evaluations utilize mixed-methods research designs that can examine unintended consequences because they combine multiple data sources to capture a broad perspective. USAID/Liberia should consider utilizing outcome harvesting as one of the methods in its evaluations. Outcome harvesting collects evidence of changes in outcomes first and then determines whether and how interventions affected these changes. It gathers all outcomes without a prearranged partiality toward intended outcomes. The Mission should continue to include questions on how unintended outcomes contribute or detract from achieving objectives in the SOWs for performance evaluations. It should utilize the information gathered through these questions to design projects and activities that anticipate common types of unintended consequences. Furthermore, USAID/Liberia should build in opportunities for course corrections in the design of activities to address unintended outcomes as they arise. 30 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK A Meta-Analysis of USAID’s Development Assistance under the current CDCS USAID’s Liberia Strategic Analysis (LSA) proposes to conduct a meta-analysis of the development assistance provided by USAID/Liberia through its Country Development Cooperation Strategy (CDCS) for 2013-2018. The meta-analysis will review the monitoring, assessments, and performance evaluations conducted by USAID during this period and synthesize findings and conclusions from these data sources into recommendations for designing the new CDCS for 2018 - 2023. Background The CDCS is a strategic plan that defines the most effective way of achieving development results and serves as the foundation for subsequent decision making for project design, activity design, staffing levels, and budget negotiations. Commencing in February 2017, USAID/Liberia began to design a new CDCS for the period of October 2018 to September 2023 (FY 2019 – FY 2023. The current CDCS applies to April 2013 to December 2018 (FY 2013 – FY 2019). USAID extended it from April 2018 to December 2018, following a stocktaking exercise facilitated by LSA in March 2016. The design, preparation, and approval of the new CDCS is taking place from February 2017 to May 2018, including the following phases: • Analytical Phase: complete analyses to inform strategic planning process over nine months, from February to October 2017 • Phase 1 – Initial Consultations and Parameter Setting: initiate formal engagement with Washington to present the planned path to developing the CDCS over three months, from August to October. Note Phase I begins while the Analytical Phase continues. • Phase 2 – Results Framework Development: gain agency consensus on USAID/Liberia’s planned approach over three months, from November 2017 to January 2018. • Phase 3 – Full CDCS Preparation, Review, and Approval: prepare, approve, and disseminate CDCS over four months, from February to May 2018. LSA proposes to conduct a meta-analysis of the development assistance provided by USAID/Liberia through its current CDCS to generate recommendations for the new CDCS, during the Analytical Phase. This meta-analysis will be in addition to the eight analyses and performance evaluations that LSA is currently undertaking for the Analytical Phase. A meta-analysis of the development assistance provided by USAID/Liberia through its current CDCS is included in LSA’s Work Plan for Year 2, approved by USAID on December 22, 2016. It is found in Section 1.1.4 Plan the CDCS development process, “c. Conduct a meta-analysis of recommendation: Starting in Year 2, LSA will compile and code qualitative data on recommendations provided throughout its monitoring, evaluation and learning activities as well as through implementing partner (IP) reports.” LSA will focus on IPs reporting in PIDS and will review Annual Reports and others identified by the Program Office. Objective LSA proposes to conduct a meta-analysis of the development assistance provided by USAID/Liberia through its CDCS for 2013-2018. The meta-analysis will review the monitoring, assessments, and performance evaluations conducted by USAID during this period and synthesize findings and conclusions 31 from these data sources into recommendations for designing the new CDCS for 2019-2023. For the evaluations where a follow-on project has been designed, reviewers will assess to what extent lessons learned from the previous evaluation have improved the results achieved or what pitfalls have been avoided in the new project. During the current CDCS, USAID has monitored its activities by collecting performance indicator data on hundreds of indicators and reported this data through the Mission’s Performance Plan Report (PPR), Performance Management Plan (PMP), Performance Indicator Database System (PIDS), and narrative quarterly, annual, and final reports submitted by IPs. USAID has also analyzed its activities and projects through several assessments, evaluations, and other studies during this period. Furthermore, USAID’s stakeholders in Liberia have accumulated knowledge of USAID’s development assistance during the current CDCS. LSA proposes to systematically review the findings and conclusions of these data sources to address the following research questions: 1. What recommendations for the new CDCS can USAID gain from the monitoring of its projects and activities that occurred during the current CDCS period? 2. What recommendations for the new CDCS can USAID garner from the assessments and performance evaluations that were conducted during the current CDCS period? 3. What recommendations for the new CDCS can USAID gain from consultations with key stakeholders in Liberia? 4. What common themes occur throughout these recommendations and how can they apply to the new CDCS? Methodology LSA will conduct this meta-analysis using a mixed-methods approach. LSA will engage expert consultants with a broad understanding of USAID’s approach to monitoring, evaluation, and learning (MEL) to conduct desk reviews of monitoring, assessments, and performance evaluations, and to facilitate key informant interviews (KIIs) and focus group discussions (FGDs) with stakeholders in Liberia. The consultants will quantitatively analyze data garnered from USAID’s monitoring of its projects and activities in the PPR, PMP, PIDS, and narrative reports during the current CDCS to generate recommendations for the new CDCS. They will also review and code qualitative data from the assessments and evaluations conducted under the current CDCS to do quantitative analysis and generate recommendations for the new CDCS. Furthermore, the consultants will collect qualitative data through KIIs and FGDs that will be coded for analysis and synthesis. LSA proposes to conduct the desk reviews of monitoring data from the PPR, PMP, PIDS, and quarterly, narrative, and annual narrative reports, as well as the desk reviews of assessments and performance evaluations out-of-country to most efficiently utilize its resources. It proposes to conduct the KIIs and FGDs in Liberia to gather data on-the-ground and contextualize the desk review with key stakeholders, as identified by the Mission. Having several members of the consultant team to Liberia will also allow for in-brief and de-brief presentations with USAID to incorporate their feedback in shaping the meta￾analysis. Tasks 1. To compile and code all recommendations for all available reports on monitoring, evaluation and learning activities, assessments and IP reports. 2. To establish a system to compile recommendations from IP reports, site visit report, DQAs and 32 learning events. 3. Conduct Stakeholder consultations by sector with key counterparts in Government, Civil Society, Donor partners and USAID/Liberia’s Communities of Practice, using KII’s and FGDs. The objectives are to get the perceptions of how well the USAID projects have run and whether they have accomplished their objectives. Approach 1. Organize all reports: To begin with, LSA has the list of mid-term and final evaluations conducted for several activities since the implementation of the current CDCS. LSA will use the list as guide to gather all evaluation reports; and find all monitoring reports, DQA reports, learning event reports, relevant IP reports and assessment reports to be analyzed. The DEC, PIDS and USAID/Liberia KMP will serve as sources to gather these reports. 2. Analyze the executive summaries and recommendations for all reports: LSA will develop codes or a framework for analyzing the recommendations of reports. All reports will be grouped by category. For example, all assessment reports, all evaluation reports, etc. LSA will undertake an in-depth analysis of all reports in view of each DO. Analysis will consider if the recommendations are related to improving the achievement of the DO, IR and Sub-IR level results. This will help inform the mission on how they have learned from evaluations and other types of performance monitoring. LSA will focus on what has been achieved and the trends throughout all DOs. 3. Establish a system: upon the completion of the analysis, LSA will establish a system for filing recommendations from various reports. An Excel Workbook will be created to track the types of reports, the date it was approved, an outline of the recommendations and indicate who was responsible for implementing the recommendations. 4. Produce a Summary Report: Based on the information gathered from the KIIs and FGDs, LSA will write a summary report, capturing lessons learned from the current CDCS implementation period and recommendations for future CDCS design, including a Mission-wide and sector￾specific analysis. Level of Effort & Schedule LSA will employ a team of Social Impact’s (SI) MEL experts and full-time employees to serve as consultants for the meta-analysis. The use of SI’s experts will enable LSA to rapidly begin to conduct the meta-analysis and conclude its prior to the end of the Analysis Phase of the CDCS design process in October 2017. SI will also utilize experts with prior experience working with LSA to ensure they already understand the context of USAID/Liberia’s strategy, projects, and activities. The consultant team will include the following positions and level of effort (LOE): • Team Leader & Evaluation Expert to lead the meta-analysis initiative, including facilitating the team’s data collection and analysis, presentations and report writing, and to lead the desk review and analysis of the assessments, evaluations, annual reports etc. This position will utilize 20 days LOE out-of-country for the desk review and report writing, and 15 days in-country to lead the team and briefings with USAID. • Monitoring Specialist to lead the desk review and analysis of monitoring data from the PPR, PMP, PIDS, and quarterly, narrative, and annual narrative reports. The Monitoring Specialist will also conduct the KIIs and FGDs with key stakeholders in Liberia and to contribute to garnering USAID’s feedback through the in-brief and de-brief presentations. This position will utilize 5 days LOE for the desk review and 10 days in-country to conduct the KIIs and FGDs. 33 • Desk reviewers will review documents for the 4 DO teams and draft a write up to be reviewed and analyzed by the Team Leaders. • LSA’s CLA/Knowledge Management Advisor will facilitate the consultant team to liaise with USAID, gaining access to the necessary data sources, and ensure its deliverables contribute to the new CDCS. LSA hereby requests a six-day work week for the Team Leader & Evaluation Expert and the CLA expert to maximize the amount of time they can work in Liberia and to make more efficient use of per diem. Upon approval of this SOW by USAID, LSA will immediately begin to assemble the team. LSA aims to gather data and commence the desk review within one month of receiving USAID approval, and to field the consultants that will work in country within two months of receiving USAID approval. Assuming approval by early May 2017, LSA aims to conclude the meta-analysis and furnish USAID with recommendations for the new CDCS by October 2017, prior to the conclusion of the Analytical Phase and when the Mission intends to begin Phase I. Position Days Estimated Rate Total Notes Team Leader 35 $550 $19,250 Average expatriate team leader daily rate Flights, Per Diem, etc. for Team Leader and Monitoring Specialist - - $13,250 This includes $2,200 for airfare for two staff and the State Dept. per diem rate for 10 work days /15 days total in Liberia, which totals to $8,850 ($295 per diem rate x 15 days in-country x 2 people). Monitoring Specialist 15 $400 $6,000 Average mid-level, expatriate monitoring specialist daily rate Desk Reviewers 40 $274 $10,960 This is an average rate for junior to mid￾level local and expatriate reviewers. TOTAL $49,460 Deliverables • The meta-data analysis will result in the following deliverables: • In-brief presentation in Microsoft PowerPoint with USAID to incorporate their feedback into the design of the meta-analysis • De-brief presentation in Microsoft PowerPoint with USAID to deliver the study’s recommendations for the new CDCS • Databases of quantitative and qualitative data, including analysis and coding sheets, for the desk reviews, KIIs and FGDs • Final report in Microsoft Word of 25 pages addressing the SOW’s research questions. 34 ANNEX II: ORIGINAL RESULTS FRAMEWORK FROM THE USAID/LIBERIA CDCS FOR FY 2013 – FY 2017 35 ANNEX III: EVALUATION METHODS The Meta-Analysis intends to accomplish its objective through a mixed-methods research approach with a concurrent triangulation design. A mixed-methods approach systematically combines qualitative and quantitative data from multiple studies and other sources to provide a broad perspective. A concurrent triangulation design is where data is analyzed separate and then compared and combined to confirm and corroborate findings. This approach and design intend to offset the weakness inherent in qualitative and quantitative types of data by allowing them to validate each other and providing stronger evidence for conclusions in the process. The Meta-Analysis uses the following methods in its research approach: • Numerical review to measure quantitative data from performance indicators and other relevant metrics for the period of the current CDCS for FY 2013 - FY 2018; • Desk review to collect qualitative data from assessments, evaluations, reports, and other documents conducted during the current CDCS for FY 2013 - FY 2018; and • Key informant interviews to collect qualitative data from key stakeholders, including individuals serving with USAID, GOL, other donors, IPs, and civil society actors, who have been active during the current CDCS for FY 2013 – FY 2018. Please see the table below for more information on how these methods and data sources address the research questions. Table 1: Preliminary Evaluation Design Matrix Research Question Data Source Method 1. To what extent has progress been made in achieving USAID’s strategic goal from the current CDCS for FY 2013 – FY 2018? a. What has been achieved b. What has not been achieved? c. What gaps exist? • Performance indicators from USAID/Liberia’s PIDS; USAID/Liberia’s PPRs; DQAs; AMELPs; final and annual reports from IPs; and other sources of performance indicator data • Studies and documents including assessments; impact evaluations; performance evaluations; separate surveys; final and annual reports from IPs; and other documents • Interviews with USAID, GOL; donors other than USAID; IPs; and civil society actors • Numerical Review • Desk Review • KIIs 2. How have USAID’s development objectives contributed to achieving USAID’s strategic goal? • Performance indicators from USAID/Liberia’s PIDS; USAID/Liberia’s PPRs; DQAs; AMELPs; final and annual reports from IPs; and other sources of performance indicator data • Studies and documents including assessments; impact evaluations; performance evaluations; separate surveys; final and annual reports from IPs; and other documents • Interviews with USAID, GOL; donors other than USAID; IPs; and civil society actors • Numerical Review • Desk Review • KIIs 3. To what extent were the programmatic and contextual assumptions • Context and sentinel indicators from USAID/Liberia’s PIDS; USAID/Liberia’s PPRs; final and annual reports from IPs; and • Numerical Review • Desk Review 36 identified in USAID’s development hypothesis sufficient to achieve its strategic goal? other sources of context and sentinel indicators, including the World Bank and other donors • Studies and documents including assessments; impact evaluations; performance evaluations; separate surveys; final and annual reports from IPs; and other documents • Interviews with USAID, GOL; donors other than USAID; IPs; and civil society actors • KIIs 4. What were the unintended outcomes of USAID’s development assistance under the current CDCS for FY 2013 – FY 2018? • Performance indicators from USAID/Liberia’s PIDS; USAID/Liberia’s PPRs; DQAs; AMELPs; final and annual reports from IPs; and other sources of performance indicator data • Studies and documents including assessments; impact evaluations; performance evaluations; separate surveys; final and annual reports from IPs; and other documents • Interviews with USAID, GOL; donors other than USAID; IPs; and civil society actors • Numerical Review • Desk Review • KIIs Numerical Review Plan The team conducted a numerical review to measure quantitative data from performance indicators and other relevant metrics for the period of the current CDCS for FY 2013 – FY 2018 through the following steps: • Define eligibility criteria for relevant performance indicators to be reviewed. The team focused on indicators that monitor project and strategic outcomes for USAID’s CDCS for FY 2013 – FY 2018 – notably the indicators from the CDCS’s Annex C. Summary of Performance Indicators. Indicators at the goal, DO, IR and sub-IR levels were included. By focusing on data describing the outcomes of projects and activities, rather than outputs, LSA put forth meaningful recommendations to inform judgments about USAID’s future strategic approach in Liberia. The team examined context and sentinel indicators from USAID and non-USAID sources when addressing Research Question 3. • Identify relevant performance indicators to be examined. The team focused on indicators reported to the Office of US Foreign Assistance Resources through the PPR, since FY 2013. Furthermore, USAID/Liberia’s PPD, CDCS Working Group, and DO M&E focal persons selected activities closely aligned with its development hypothesis for the team to focus on in the Meta-Analysis. USAID implemented over a hundred activities during the current CDCS, and LSA lacked the time and resources to review them all in this study. LSA reviewed the AMELPs to identify outcome indicators for the selected activities. USAID/Liberia lacked an approved PMP from FY 2013 – FY 2016, most of the duration of the current CDCS for FY 2013 – FY 2018, which limited the PMP’s usefulness as a source of indicators for the Meta-Analysis. The PMP was approved in November 2016. • Ensure performance indicator data is valid. The team reviewed any DQAs conducted during the current CDCS for the relevant indicators to ensure they have the integrity necessary to be utilized in management decisions before being included in the Meta-Analysis. • Collect and aggregate performance indicator data. The team accessed the PPRs, PIDS, and annual and final reports for activities to collect and aggregate data from relevant, selected performance indicators. LSA organized these in a single dataset to facilitate analysis and to triangulate findings 37 with qualitative data. Each performance indicator included its baseline, results, targets, and disaggregations, along with several descriptors to filter and index the data. Desk Review Plan The team conducted a desk review to collect qualitative data from assessments, evaluations, reports, and other studies and documents from the current CDCS for FY 2013 - FY 2018 through the following steps: • Define eligibility criteria for the relevant studies to be included. The team included high quality studies and documents that examine outcomes directly related to USAID/Liberia’s overarching goal and DOs as defined by the current CDCS for FY 2013 - FY 2018. These included assessments, impact evaluations, performance evaluations, separate surveys, final and annual reports from activities, and other studies of a similar nature conducted by USAID and IPs. To be included in the desk review, such studies must contain an executive summary, description of their methodology, and specific sections on conclusions and recommendations. • Delineate a strategy for identifying the studies. In coordination with USAID/Liberia’s PPD, CDCS Working Group, and DO M&E focal persons, LSA identified studies for the desk review. LSA collected those documents immediately available and circulated a list of sought after, priority studies to the Mission to contribute. These included evaluations, assessments, and surveys conducted prior to LSA’s period of performance, and final and annual reports submitted by IPs. • Create standardized system for data collection. The team used the Dedoose cross-platform app for analyzing qualitative and mixed methods research as a standardized system to collect, code, share, and analyze data collected through the desk review. Dedoose streamlined the desk review, because the software can organize and analyze large amounts of qualitative data and offers a single online platform for team members in Liberia and at SI’s headquarters in the US to share and review studies. The team developed descriptors and codes to organize, summarize, and interpret data from relevant studies. Emergent codes arose and codes were refined during data collection. • Ensure relevant studies are valid before their inclusion. The team reviewed the methodologies of eligible and identified studies to ensure they employed rigorous, empirical investigative methods prior to collecting data from them for the Meta-Analysis. • Collect data from the relevant studies. The team entered the executive summary, conclusion, and recommendation section of each included study as a single excerpt in Dedoose, reading and reviewing these sections for their relevance to USAID’s future strategic approach and what has been achieved to date in Liberia, and applying descriptors and codes to organize data and prepare for analysis. The desk review employed the same descriptors used in the numerical review to assist with analysis. KII Plan The team conducted KIIs to collect qualitative data from key stakeholders, including individuals serving with GOL, other donors, IPs, and civil society actors, who have been active during the current CDCS for FY 2013 – FY 2018 through the following steps: • Formulate study questions and interview guide. The team developed interview questions on lessons learned from development assistance implementation, changes to the country context, and USAID/Liberia’s programmatic priorities related to the current CDCS for FY 2013 – FY 2018 and the new strategy for FY 2019 - FY 2023. LSA used open-ended questions to encourage a free flow of ideas and discussion, and organized them in an interview guide of 12 or fewer topics and issues to cover. • Develop a sample plan for key informants. With input and guidance from USAID/Liberia’s PPD, CDCS Working Group, and DO M&E focal persons, LSA selected around 25 key informants. The team identified groups and organizations from which to draw key informants, including 38 USAID, GOL, other donors, IPs, and civil society actors, and sought USAID/Liberia’s input on specific individuals to interview. The team selected informants based on their specialized knowledge and unique perspectives on the strategic approach of the current CDCS for FY 2013 – FY 2018, the Liberian development context, aid effectiveness, and the rights and interests of Liberian citizens. LSA and USAID coordinated to arrange KIIs and secure the availability of participants. • Collect data through interviews. The team conducted interviews to collect qualitative data from the key informants. LSA explained the purpose of the interview and the confidentiality of the informant, sequence and phrase questions appropriately, used probing techniques, and maintained neutrality to ensure the KIIs generated useful data to inform judgments about USAID’s future strategic approach in Liberia. The interviewers transcribed notes during and after the interview, entered the notes from each KII as a single excerpt in Dedoose, and applied the same descriptors and codes used in the desk review and numerical review to assist with analysis. Data Analysis Plan Per its mixed-methods approach, the Meta-Analysis analyzed data from its quantitative numerical review method and qualitative desk review and KII methods. This analysis occurred concurrently according to its triangulation design to confirm, cross-validate, and corroborate findings. The table below describes the study’s general procedures for data analysis. Table 2: Data Analysis Procedures Procedure Quantitative Data Qualitative Data Exploring data Descriptive analysis to identify trends and distributions in performance indicators Apply descriptors and codes after reviewing excerpts from desk review studies and KIIs in Dedoose Analyzing data Conduct statistic tests as appropriate, including correlation and potentially inferential statistics Identify frequencies, ratios, and other themes by grouping and disaggregating excerpts in Dedoose Integrating data Consolidate and connect data through the shared codes and descriptors through Dedoose to compare the datasets and their findings, and draw out their summary effect through comprehensive conclusions and recommendations Representing data Depict results through tables, graphs, infographics, impactful quotes, and other data visualizations through Dedoose and other software Utilization-Focused Reporting The Meta-Analysis reported its findings, conclusions, and recommendations using terminology appropriate for the strategic planning process and organized to apply to the milestones of Phase 1 and Phase 2 of the design process, particularly the Concept Note and Results Framework Paper. This intended to promote the utilization of the Meta-Analysis by allowing USAID to easily incorporate its recommendations into the new strategy for FY 2019 - FY 2023 as it is developed. USAID/Liberia’s PPD shared tools and templates with LSA for the CDCS design process throughout August 2017. Table 3: The Meta-Analysis’s Contributions to Milestones in the CDCS Design Process Phase Milestones Meta-Analysis Will Tailor Recommendations For… Phase 1 – Initial Consultations and Parameter Setting • Concept Note • Questionnaire and Consolidation of Washington Operating Unit (OU) Inputs Concept Note 39 • Phase One Digital Video Conference (DVC) • Summary of Conclusions Phase 2 – Results Framework Development • Results Framework Paper • Review of Results Framework Paper • Phase Two DVC • Summary of Conclusions Results Framework Paper Phase 3 – Full CDCS Preparation, Review, and Approval • Draft Full CDCS • Review of CDCS Draft • Phase Three DVC • Summary of Conclusions • Final CDCS Approval • Disseminations of CDCS - Dissemination and Communications Plan To support the use of the Meta-Analysis, the team adhered to the following dissemination and communications plan. Table 4: Communications Plan Type of Information Format of Communication Timeline Person Responsible Meta-Analysis purpose, design, and work plan In-brief presentation with USAID On/around September 11, 2017 Meta-Analysis Team Leader and Monitoring Specialist. Supported by LSA’s CLA/Knowledge Management Advisor. Meta-Analysis preliminary findings De-brief presentation with USAID On/around September 29, 2017 Meta-Analysis Team Leader and Monitoring Specialist. Supported by LSA’s CLA/Knowledge Management Advisor. Meta-Analysis final findings, conclusions, and recommendations Final report On/around October 20, 2017 Meta-Analysis Team Leader CDCS Working Group Learning Event Pause & Reflect Exercise with USAID Mid-November, 2017 CLA/Knowledge Management Advisor 40 ANNEX IV: DISCLOSURE OF ANY CONFLICT OF INTEREST 41 42 43 44 45 46 ANNEX V: DESK REVIEW PARAMETERS Background The Meta-Analysis intends to generate recommendations for USAID’s new strategy for Liberia during FY 2019 - FY 2023 by examining the progress made toward USAID’s strategic goal under the current CDCS for FY 2013 – FY 2018. The LSA team will explore the unintended outcomes and assumptions underpinning USAID’s development assistance during this period. By synthesizing M&E under the current CDCS, and incorporating stakeholder consultations, the LSA team aims to ground USAID’s new strategy in evidence and analysis produced by USAID/Liberia and its partners over the past six years. Research Questions Four research questions will guide the Meta-Analysis, and through this framework the study will offer succinct and actionable recommendations to USAID/Liberia to design a new strategy for FY 2019 - FY 2023. Per guidance received from PPD on August 31, 2017, LSA updated the research questions for the Meta￾Analysis to the following: 1. To what extent has progress been made in achieving USAID’s strategic goal from the current CDCS for FY 2013 – FY 2018? a. What has been achieved b. What has not been achieved? c. What gaps exist? 2. How have USAID’s development objectives contributed to achieving USAID’s strategic goal? 3. To what extent were the programmatic and contextual assumptions identified in USAID’s development hypothesis sufficient to achieve its strategic goal? 4. What were the unintended outcomes of USAID’s development assistance under the current CDCS for FY 2013 – FY 2018? Process & Procedures Please keep the following process and procedures in mind as you conduct your desk review. Please feel free to innovate as you streamline your process. Please share your challenges and innovation with the other team members in a shared Skype conversation. 1. As background, familiarize yourself with USAID/Liberia’s CDCS for FY 2013 – FY 2018, ADS 201, and the mandatory reference to ADS 201 for CDCS on SharePoint here. 2. Review the evaluations, assessments, reports, studies, and documents for your DO from USAID/Liberia on Google Drive here. a. Note the organization by DO, by activity, and by type of report. b. Important: Remember to focus on the types of reports that apply to the Desk Review. i. For example, the Desk Review should focus on performance evaluations, assessments, final reports, etc. You will not need to review AMELPs and site visits, as these will be used by the Numerical Review (Stephanie) c. Important: Remember to save time by not reading duplicate reports. i. For example, if an activity has a final report and annual reports, please only review the final report as it will summarize the preceding annual reports. 47 3. Download the documents that you will read and review, and upload them into Dedoose. 4. Write down each document in the Desk Review List here. Use the appropriate citation style per MLA. a. In general MLA style for a report is: i. Author(s). Title of Report. Place of Publication: Publisher, Year of Publication. b. The Desk Review List will be an annex for the Final Report. 5. Apply the appropriate descriptors to each document, after uploading it to Dedoose. a. Complete a set of General Descriptors b. Link the document to a specific activity in the Activity Descriptors Set 6. Read and review the document. a. Focus on the Meta-Analysis’s research questions as you read the document. 7. Apply the appropriate codes to each document in Dedoose as you read. a. Note the organization of codes, sub-codes, and sub-sub-codes along the lines of the research questions. b. Apply the pre-set codes as necessary. These codes have been labeled as they apply to Research Questions 1, 2, 3, and 4. c. Create emergent codes as you identify themes i. Important: Remember not to worry about standardizing emergent codes used by others. The Team Leader will do this as part of his quality assurance review. d. The codes and descriptors will be used during the analysis phase. 8. Complete the Desk Review Summary Spreadsheet available here after reading all the applicable documents for an activity. a. Summarize your review of all the applicable documents for an activity in the Summary Spreadsheet b. The Summary Spreadsheet intends to summarize your conclusions about the activity and Research Question 1, 2, and 4. i. Important: Carefully consider all documents for an activity, the types of documents available, and the conclusions of these documents. For instance, a Performance Evaluation by a third party may provide more persuasive evidence that a IR was achieved than an Annual Report from an implementing partner. ii. Important: Please feel free to selected any of the options, including the “Mixed Achievement” or “Unclear Evidence” options, when making these normative statements. Remember this is your educated view considering all data available to you through the Desk Review. c. If you encounter challenges with the Summary Spreadsheet (ex. Not applicable questions or lack of appropriate answers), please contact the Team Leader immediate via email or Skype. d. The Desk Review Summary Spreadsheet will be used during the analysis phase. 9. Repeat. 48 10. Notify the Team Leader when you conclude your desk review. Descriptors Descriptors are information that describes the source of the data at a particular level of analysis. There are two sets of descriptors in Dedoose: • Activities - You will only need to link the documents that you review to the activities that they apply to through the Activities Descriptors. Information on the activities are already in Dedoose. • General – You will need to enter data for the documents that you review using these General Descriptors. Information on each of the documents for the desk review are not yet in Dedoose. You will need to complete these six fields for each document that you upload to Dedoose and review. General Descriptors Title Description Options ID Unique identifier Numeric field type. Please describe each desk review document with a four-digit number. Begin that number with 1 for DRG, 2 for EG, 3 for health, and 4 for education, per USAID/Liberia’s DOs. For example, the first DRG document reviewed should be: 1001, etc. DO Development objective or thematic area • Democracy, Rights and Governance • Economic Growth • Health • Education • Cross Cutting • Other Date Date or year published N/A Method Method Used – In your case, always select Desk Review • Desk Review • Indicator Review • KII Source Author or source of a study • Implementing partner • USAID directly • Other Donor • GOL • Civil Society Partner • LSA • Other Type Type of study • Assessment • Performance Evaluation • Impact Evaluation • Standalone Survey • Final Report • Annual Report • PPR • PIDS • KII • Other Pre-Set Codes 49 The team has uploaded the following pre-set codes to the Meta-Analysis project in Dedoose. New codes may be added later as emergent-codes (see below). Code Title Sub-Code Title Description GREAT QUOTE Great quote Used to denote a conclusion or recommendation that is particularly impactful and may wish to be quoted directly in the Meta-Analysis. DO1: Democracy, Rights, and Governance: More effective, accountable, and inclusive governance IR1.1 Public resources managed more transparently and accountably The excerpt contains evidence or a clear statement that the activity achieved this DO or IR. Remember: please only apply this code to a statement or piece of evidence that an activity has clearly achieved its IR or supported its DO. IR1.2 Improved policies, models and providers; increase access to justice The excerpt contains evidence or a clear statement that the activity achieved this DO or IR. IR 1.3 Strengthened capacity to implement election process The excerpt contains evidence or a clear statement that the activity achieved this DO or IR. IR 1.4 Civil society and media exercise their advocacy and oversight roles The excerpt contains evidence or a clear statement that the activity achieved this DO or IR. DO2: Economic Growth: Sustained market-driven economic growth to reduce poverty 2.1 Food security enhanced The excerpt contains evidence or a clear statement that the activity achieved this DO or IR. 2.2 Natural resources managed sustainably The excerpt contains evidence or a clear statement that the activity achieved this DO or IR. 2.3 Enabling environment supports private enterprise growth The excerpt contains evidence or a clear statement that the activity achieved this DO or IR. DO 3: Health: Improved health status of Liberians 3.1 Increased utilization of quality reproductive, maternal, newborn & child health services The excerpt contains evidence or a clear statement that the activity achieved this DO or IR. 3.2 Increased effectiveness of health system at national and county level The excerpt contains evidence or a clear statement that the activity achieved this DO or IR. 3.3 Improved GOL capacity to control infectious diseases The excerpt contains evidence or a clear statement that the activity achieved this DO or IR. 3.4 Increase access to safe water and sanitation The excerpt contains evidence or a clear statement that the activity achieved this DO or IR. DO 4: Education: Better￾educated Liberians 4.1 Increased equitable access to education The excerpt contains evidence or a clear statement that the activity achieved this DO or IR. 4.2 Improved quality of education The excerpt contains evidence or a clear statement that the activity achieved this DO or IR. 4.3 Improved governance of the education system The excerpt contains evidence or a clear statement that the activity achieved this DO or IR. Assumptions from Development Hypothesis Civil Society Organizations Please apply this code as you note conclusions or recommendations that pertain to the role of CSOs and the activity. 50 Use the child codes (Did not lock in reforms / Locked in Reforms) to indicate whether this assumption proved accurate. Community-based structures Please apply this code as you note conclusions or recommendations that pertain to the role of community-based structures (ex. Village committees). Use the child codes (Did not lock in reforms / Locked in Reforms) to indicate whether this assumption proved accurate. Decentralization Please apply this code as you note conclusions or recommendations that pertain to decentralization of government services. Use the child codes (Did not work / Worked) to indicate whether this assumption proved accurate. Environmentally sound approach Please apply this code as you note conclusions or recommendations that pertain to ensuring environmentally sound approaches to development. Please feel free to use emergent codes to further describe whether this assumption proved sufficient for activities’ design. Feedback loops between DOs Please apply this code as you note conclusions or recommendations that pertain to critical feedback loops among DOs (i.e. Progress toward one DO can be inhibited by shortcomings toward another DO). Please feel free to use emergent codes to further describe whether this assumption proved sufficient for activities’ design. Gender inequality Please apply this code as you note conclusions or recommendations that address gender inequalities. Please feel free to use emergent codes to further describe whether this assumption proved sufficient for activities’ design. Geography Please apply this code as you note conclusions or recommendations that comment on the geographic focus in development corridors. Use the child codes (Challenges geographic concentration / supports geographic concentration) to indicate whether this assumption proved accurate. Improved access to goods and services Please apply this code as you note conclusions or recommendations that comment on USAID’s two track approach. Use the child codes (Did not work / Worked) to indicate whether this assumption proved accurate. 51 Strengthen capacity to sustain development Please apply this code as you note conclusions or recommendations that comment on USAID’s two track approach. Use the child codes (Did not work / Worked) to indicate whether this assumption proved accurate. USAID Forward Please apply this code as you note conclusions or recommendations that comment on the USAID Forward principles Use the child codes (Innovation / Results orientated / sustainability) to indicate components of USAID Forward. Please feel free to use emergent codes to further describe whether this assumption proved sufficient for activities’ design. Unintended Outcomes N/A Please use this code to mark an unintended outcome. Please feel free to develop emergent codes within the “Unintended Outcomes” parent code as you notice trends in unintended outcomes emerging. Recommendation Trend Please create emergent codes as you notice common trends in the recommendation sections of documents. Please apply the parent code “Recommendation Trend” and create an appropriate child subcode when you notice common trends in the recommendation sections of documents. Please only apply this code if the recommendation appears in documents to at least three separate activities. For example, if reports for three different activities make recommendations to promote sustainability through genetically modified crops, please: • Apply the Recommendation Trend code • Create a subcode for genetically modified crops The LSA team will label particularly impactful recommendations with the code “great quotes.” These may be recommendations that appear similarly in multiple data sources and make for a persuasive recommendation for the new CDCS. The Meta-Analysis may quote these recommendations directly in the report. Emergent Codes During the desk review, the LSA team will establish a set of codes from reading and analyzing the data. These emergent codes may include ideas, concepts, actions, relationships, and meanings that come up in the data and are different than the pre-set codes. Emergent codes form the basis of interesting stories and may become a major part of the line of inquiry. Review & Quality Control When the Desk Reviewers commence reading the studies collected by LSA, they should first review the methodology section. The Desk Reviewers should confirm the identified studies employ they employ 52 rigorous, empirical investigative methods prior to collecting data from them for the Meta-Analysis. This review will ensure the studies are scientifically valid to be included in the Meta-Analysis. The LSA team will review the data collected from the studies during a series of team meetings and quality control checks. Team meetings will also provide a forum for Desk Reviewers to ask questions and ensure they are using a systematic and standard approach. The Team Leader will review the data uploaded and excerpted in Dedoose as a quality assurance checkpoint, and will refine and revise emergent codes to ensure a streamline codebook. Data analysis will occur per the plan outlined in the Inception Report. 53 ANNEX VI: INTERVIEW GUIDE Location of interview: Name(s): ________________________________ Position(s): _______________________ M/F Name(s): ________________________________ Position(s): _______________________ M/F Name(s): ________________________________ Position(s): _______________________ M/F Date of Interview:___________________ Time of Interview:___________________ Name of Interviewer:__________________ Name of Note-taker:___________________ Context of the Meta-Analysis • Brief introduction to interviewers o The Meta-Analysis intends to generate recommendations for USAID’s new strategy for Liberia during FY 2019 - FY 2023 by examining the progress made toward USAID’s strategic goal under the current CDCS for FY 2013 – FY 2018. The LSA team will explore the unintended outcomes and assumptions underpinning USAID’s development assistance during this period. By synthesizing monitoring and evaluation (M&E) under the current CDCS, and incorporating stakeholder consultations, the LSA team aims to ground USAID’s new strategy in evidence and analysis produced by USAID/Liberia and its partners over the past six years. • Purpose of the analysis and the interview o Collect qualitative data from key stakeholders, including individuals serving with USAID, GOL, other donors, implementing partners, and civil society actors, who have been active during the current CDCS for FY 2013 - FY 2018. o Focus on achievements and shortcomings for USAID from 2013 until present, and areas for improvement, challenges, and opportunities in Liberia for the next five years. • Informant confidentiality is strict o Your name will not be used in the report or its annexes o You may be referred to by your organization but not by your position or department (ex. "USAID", but "not EG team of USAID/Liberia"), nor your sex o A number may be assigned to you as a unique identifier but there will be no link between your identity and this number o The focus is on the content of our discussion and never who said what LSA Team Member 1. What was the nature of your role / involvement with USAID/Liberia from 2013 until present? a. PROBE: What year did that begin? And end? 2. What can you tell me about the achievements of USAID during this period? a. FOLLOW-UP: What evidence have you seen of these achievements? b. FOLLOW-UP: What activities contributed to these achievements? Any outliers? c. PROBE: Tell me why you think these achievements have occurred. 3. On the other hand, what can you tell me about the shortcomings of USAID during this same period? a. FOLLOW-UP: What evidence have you seen of these shortcomings? b. FOLLOW-UP: What activities faced challenges from these shortcomings? Any outliers? 54 c. PROBE: Tell me why you think these shortcomings have occurred. 4. Are you aware of any unintended outcomes of USAID’s assistance between 2013 and now? a. PROBE: How do you know that? b. FOLLOW-UP: What do you think this occurred? 5. In what areas do you feel USAID could improve its development assistance in Liberia? a. PROBE: How can USAID best do that? b. PROBE: Tell me why you think this hasn’t happened yet. 6. What programmatic areas should USAID prioritize over the next five years? a. PROBE: Why do you say that? b. FOLLOW-UP: What would be the best way for USAID to support these program areas? c. PROBE: For example, what type of activities? 7. What opportunities do you foresee occurring or continuing in Liberia over the next five years? a. PROBE: Why is this an opportunity for USAID? b. FOLLOW-UP: How can USAID take advantage of this opportunity? 8. What challenges do you foresee occurring or continuing in Liberia over the next five years? a. PROBE: Why is this a challenge for USAID? b. FOLLOW-UP: How can USAID overcome this obstacle? 9. USAID’s development hypothesis for Liberia for the past five years has been [read below]. If you were designing this hypothesis for the next five years, would you change this? a. Development Hypothesis: Liberia must establish and strengthen inclusive, indigenous institutions that enable it to effectively mobilize its own resources and efficiently manage those resources. b. PROBE: If not, why? If yes, how would you make it different? 10. Is there any other information regarding your experience and USAID’s strategic approach that you think would be useful for me to know? Thank you very much for speaking with me. Your time is very much appreciated and your comments have been helpful. Observations of the interview context: 55 ANNEX VII: PERFORMANCE INDICATOR DATA RF Indicator Source Baseline Endline Achieved Progress Year (FY) Amount Year (FY) Target Result Goal World Bank Country Policy and Institutional Analysis (CPIA) public sector management and institutions cluster average CDCS/PMP 2012 2.80 2016 2.90 2.90 Yes Increase in value Goal CPIA policies for social inclusion/equity cluster average CDCS/PMP 2012 3.10 2016 3.20 3.00 No Increase in value Goal Gender Inequality Index Ranking (United National Development Programme Human Development Report) CDCS/PMP 2013 143 2015 142 150 No Decrease in value DO 1 Corruption perceptions index CDCS/PMP 2012 41 2016 42 37 No Increase in value DO 1 Government Effectiveness Index Score CDCS/PMP 2012 -1.10 2015 -1.20 -1.40 No Increase in value DO 1 Economist Intelligence Unit Democracy Index Secondary Source 2012 4.95 2016 5.10 5.31 Yes Increase in value IR 1.1 Public Expenditure and Financial Accountability Assessment: Grade in "multi-year perspective in fiscal planning, expenditure policy, and budgeting" Secondary Source 2012 3.33 2016 3.40 3.00 No Increase in value IR 1.1 Number of USG supported anti-corruption measures implemented Secondary Source 2012 6 2016 16 19 Yes Increase in value IR 1.1 Public Expenditure and Financial Accountability Assessment: Grade in "Effectivness of internal audit" Secondary Source 2012 2.33 2016 2.50 3.33 Yes Increase in value IR 1.2 Averaged score for "civil justice" and "criminal justice" factors from the World Justice Project Rule of Law Index Secondary Source 2013 0.340 2016 0.360 0.355 No Increase in value IR 1.2 Score from World Justice Project Rule of Law Index, "ADRs are accessible, impartial, and effective" Secondary Source 2013 0.22 2016 0.47 0.48 Yes Increase in value IR 1.2 Percent of Community Justice Advisor-resolved cases that are satisfactorily resolved by a CJA in keeping with best practice in legal and alternative dispute resolution techniques Secondary Source 2015 70% 2017 83% 69% No Increase in value IR 1.3 Number citizens reached through civic and voter education CDCS/PMP 2015 177 2017 1,250,000 1,236,857 Yes Increase in value IR 1.3 Number of domestic election observers and/or party agents trained with USG assistance Secondary Source 2015 0 2017 88 95 Yes Increase in value IR 1.3 Number of elections officials trained with USG assistance Secondary Source 2015 48 2017 158 636 Yes Increase in value 56 IR 1.4 Number of CSOs receiving USG assistance engaged in advocacy interventions Secondary Source 2012 34 2016 54 109 Yes Increase in value IR 1.4 Score on the Freedom of the Press Index CDCS/PMP 2012 56 2016 57 60 No Decrease in value IR 1.4 World Press Freedom Index Secondary Source 2015 30.78 2017 30.66 31.12 No Decrease in value DO 2 Daily per capita expenditures (as a proxy for income) in USG￾assisted areas CDCS/PMP 2012 $ 1.73 2015 $ 2.15 $ 1.93 No Increase in value DO 2 Annual GDP growth rate CDCS/PMP 2012 8% 2016 0% -2% No Increase in value DO 2 Agriculture as a share of GDP CDCS/PMP 2014 39% 2016 36% 34% No Increase in value DO 2 Aggregate Ease of Doing Business Rankings CDCS/PMP 2012 149 2016 178 174 Yes Decrease in value DO 2 Prevalence of poverty: percent of people living on less than $1.25/day CDCS/PMP 2012 34% 2015 48% 40% Yes Decrease in value DO 2 Ratio of female to male labor force participation rate (%, modeled ILO estimate) Secondary Source 2012 90.18% 2016 90.80% 90.55% No Increase in value IR 2.1 Prevalence of underweight children under 5 CDCS/PMP 2012 14.70% 2015 14.00% 15.20% No Decrease in value IR 2.1 Prevalence of households with moderate or severe hunger CDCS/PMP 2013 45% 2015 40% 20% Yes Decrease in value IR 2.1 Women's Empowerment in Agriculture Index (average of three % under 'Resources' indicator) Secondary Source 2012 50% 2015 54% 34% No Increase in value IR 2.1 Gross margin per unit of land, kilogram, or animal of selected product (crops/animals/fisheries) CDCS/PMP 2013 555 2015 638 1049 Yes Increase in value IR 2.2 Number of hectares in areas of biological significance or natural resources showing improved biophysical conditions as a result of USG assistance CDCS/PMP 2012 13,369 2016 48,500 79,192 Yes Increase in value IR 2.2 Economist Intelligence Unit Democracy Index CDCS/PMP 2015 3 2016 20 5 No Increase in value IR 2.2 Number of people with increased economic benefits derived from sustainable NRM and conservation activities as a result of USG assistance Secondary Source 2015 301 2016 420 612 Yes Increase in value IR 2.2 Number of institutions with increased capacity to improve environmental awareness, NRM, biodiversity conservation, and environmental compliance Secondary Source 2014 1 2015 6 1 No Increase in value IR 2.3 Number of beneficiaries with improved modern energy services as a result of USG assistance CDCS/PMP 2012 0 2015 54,500 5,848 No Increase in value 57 IR 2.3 Km of roads constructed or repaired through USG assistance Secondary Source 2012 11 2016 185 190 Yes Increase in value IR 2.3 World Bank Doing Business Score - Starting a Business ranking Secondary Source 2012 38 2016 36 37 No Decrease in value IR 2.3 World Bank Doing Business Score - Registering Property ranking Secondary Source 2012 178 2016 176 179 No Decrease in value DO 3 Lifetime risk of maternal death (%) Secondary Source 2012 3.99% 2015 3.49% 3.55% No Decrease in value DO 3 Mortality rate, under -5 (per 1000 live births) CDCS/PMP 2012 80.3 2015 77.3 69.9 Yes Decrease in value DO 3 Mortality rate, infant (per 1000 live births) CDCS/PMP 2012 59.4 2015 56.4 52.8 Yes Decrease in value DO 3 Total fertility rate (births per woman) CDCS/PMP 2012 4.87 2015 4.77 4.65 Yes Decrease in value IR 3.1 Percent of deliveries in health facilities assisted by skilled attendants CDCS/PMP 2013 66% 2014 68% 58% No Increase in value IR 3.1 Number of children under 5 years of age with pneumonia receiving antibiotics by trained facility or health workers in USG - assisted programs Secondary Source 2014 117,576 2017 187,257 118,039 No Increase in value IR 3.1 Proportion of children under 5 years treated for malaria within 24 hours of onset of fever Secondary Source 2014 42% 2017 50% 65% Yes Increase in value IR 3.1 Number of pregant women with known HIV status (includes women who were tested for HIV and received their results) Secondary Source 2014 53,257 2017 88,611 121,936 Yes Increase in value IR 3.2 Tuberculosis case detection rate (%, all forms) Secondary Source 2012 63% 2015 40% 42% Yes Increase in value IR 3.2 Cause of death by communicable diseases, females aged 15 -34 (%) Secondary Source 2010 75.10% 2015 71.35% 71.70% Yes Decrease in value IR 3.2 Percent of target facilities that have at least a one month supply of the requisite minimum supplies to allow for IPC standard adherence Secondary Source 2015 65% 2017 100% 94% No Increase in value IR 3.2 Percent of facilities that receive quarterly visit from the CHT (Joint Integrated Supportive Supervision) CDCS/PMP 2013 64% 2014 70% 64% No Increase in value IR 3.3 Immunization, measles (% of children aged 12 -23 months) Secondary Source 2012 80% 2017 70% 80% Yes Increase in value IR 3.3 Percent of children with fever receiving antimalarial drugs (% of children under age 5 with fever) Secondary Source 2011 57% 2013 58% 56% No Increase in value IR 3.3 Percent of MCSP - supported health facilities reporting meeting 80% of IPC/SQS minimum standards Secondary Source 2015 46% 2016 100% 69% No Increase in value IR 3.4 Percentage of population in USAID target counties using an improved drinking water source CDCS/PMP 2012 7% 2016 39% 5% No Increase in value 58 IR 3.4 Percentage of households using an improved sanitation facility CDCS/PMP 2013 20% 2014 26% 27% Yes Increase in value IR 3.4 Improved water source (% of population with access) Secondary Source 2012 73% 2015 75% 76% Yes Increase in value DO 4 Proportion of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade -level text CDCS/PMP 2013 6% 2015 10% 14% Yes Increase in value DO 4 Number of repeaters, primary (national, both sexes) Secondary Source 2011 43,865 2015 83,822 77,957 Yes Decrease in value DO 4 Primary net enrollment rate CDCS/PMP 2012 42% 2016 30% 48% Yes Increase in value DO 4 Gross primary completion ratio (net) CDCS/PMP 2014 63% 2015 64% 59% No Increase in value DO 4 Girls' gross enrollment in secondary Secondary Source 2011 37% 2015 37% 33% No Increase in value DO 4 Gender parity index, primary, total Secondary Source 2011 92% 2015 94% 90% No Increase in value DO 4 Gender parity index, secondary, total Secondary Source 2011 82% 2015 84% 78% No Increase in value IR 4.1 Number of teachers in USG supported programs trained on how to support learners' psychological well - being CDCS/PMP 2012 1,451 2016 7,007 10,754 Yes Increase in value IR 4.1 Number of learners reached by USG suported reading interventions at the formal and non -formal primary level CDCS/PMP 2015 25,253 2016 368,289 25,943 No Increase in value IR 4.1 Number of individuals from underserved or disadvantaged groups accessing tertiary education programs (that align with development priorities) CDCS/PMP 2012 281 2016 815 1,174 Yes Increase in value IR 4.2 Number of textbooks and other teaching and learning materials provided with USG assistance CDCS/PMP 2012 428,478 2016 3,539,011 1,382,277 No Increase in value IR 4.2 Pupil teacher ratio (primary) Secondary Source 2011 26% 2016 24% 30% No Decrease in value IR 4.2 Pupil teacher ratio (secondary) Secondary Source 2011 15% 2016 13% 18% No Decrease in value IR 4.2 Number of USG -funded tertiary education programs that include experiential ad/or applied learning opportunities CDCS/PMP 2012 75 2014 100 100 Yes Increase in value IR 4.2 Number of teachers/educators/teaching assitants who successfully completed in -service training or received intensive coaching or mentoring with USG support CDCS/PMP 2012 6,639 2015 22,183 16,795 No Increase in valu e 59 IR 4.2 Number of primary or secondary educators who complete professional development activities with USG assitance Secondary Source - - 2017 46,204 12,659 No Increase in value IR 4.2 Number of administrators and officials who complete professional development activities with USG assistance CDCS/PMP 2012 1,710 2017 24,779 8,113 No Increase in value IR 4.3 Number of laws, policies, regulations or guidelines developed or modified to improve primary reading programs or increase equitable access) CDCS/PMP 2012 0 2015 9 5 No Increase in value IR 4.3 Number of PTAs or similar school governance structures supported CDCS/PMP 2012 800 2015 7,200 5,603 No Increase in value IR 4.3 Percentage of PTA members and teachers who can state five or more breaches of the Teachers' Code of Conduct Secondary Source 2014 76% 2017 80% 91% Yes Increase in value 60 ANNEX VIII: BIBLIOGRAPHY FROM DESK REVIEW The following is a list of documents included in the desk review organized by DO. DO1: Democracy, Rights, and Governance: More effective, accountable, and inclusive governance • Alloway, J. Lance and Pewee S. Flomoku. Access to Justice in Liberia (A2J) Annual Report January 1 to September 30, 2016. Revised Submission April 15, 2017. • Bowen, Nina. Approval of Public Sector Governance PAD. May 8, 2015. • Bruins, Amy, David Felson, Megan Huth, and Mark Marquardt. Liberia Land Governance Support Activity Monitoring, Evaluation, and Learning Plan. June 2017. • Checchi and Company Consulting. Legal Professional Development and Anti-Corruption Program in Liberia (LPAC) – Year One Annual Report. October 2016. • Collin, Cecile and Angelance Browne. Mid-Term Review Liberia Decentralization Support Programme UNDP Liberia. November 2016. • Concord Consulting. Final Evaluation of the Carter Center Access to Justice Program in Liberia. April 2017. • Consortium for Elections and Political Process Strengthening (CEPPS). CEPPS Annual Report: January 1, 2015 to September 30, 2015. October 2015. • Consortium for Elections and Political Process Strengthening (CEPPS). CEPPS Annual Report: January 1, 2016 to September 30, 2016. October 2016. • Consortium for Elections and Political Process Strengthening (CEPPS). CEPPS Liberia Elections and Political Transition Monitoring and Evaluation Narrative. November 2016. • Development Alternatives Inc. (DAI). LEGIT Knowledge Management and Learning Strategy 05.12.17. November 2016 • Development Alternatives Inc. (DAI). Liberia Accountability and Voice Initiative (LAVI) Annual Report: Year 1, Fiscal Year 2016, November 2015 – September 2016. October 2016. • Development Alternatives Inc. (DAI). Liberia Accountability and Voice Initiative (LAVI) Activity Monitoring and Evaluation Plan – Draft 25 November 2015 – 30 September 2016. January 2016. • Development Alternatives Inc. (DAI). Local Empowerment for Government Inclusion and Transparency (LEGIT) Activity Monitoring, Evaluation and Learning Plan, June 2016. • Development Alternatives Inc. (DAI). Local Empowerment for Government Inclusion and Transparency (LEGIT) Knowledge Management and Learning Strategy, November 2016. • Development Alternatives Inc. (DAI). Revenue Generation for Governance and Growth (RG3) Monitoring, Evaluation and Learning Plan. September 16, 2016. • Development Alternatives Inc. (DAI). USAID/Liberia – Local Empowerment for Government Inclusion and Transparency (LEGIT) Fiscal Year 2016 Activity Implementation Report. October 2016. • Flomoku, Pewee S. Monitoring and Evaluation Plan, Strengthen peace in Liberia by enhancing citizen ownership of government and the provision of justice. January 29, 2016 • Government of Liberia. Liberia Decentralization Support Program (LDSP) Annual Progress Report 2016. October 2016. • Gray, Dylan and Jan McArthur. Internews Network. Liberia Media Development (LMD) Program – Fourth Quarter Narrative Report. October 2016. 61 • IBI International. USAID/Liberia Administrative and Systems Strengthening (LASS) Annual Report Fiscal Report 2015. October 2015. • IBI International. USAID/Liberia Administrative and Systems Strengthening (LASS) Annual Report Fiscal Report 2016. October 2016. • IBI International. USAID/Liberia Administrative and Systems Strengthening (LASS) Monitoring and Evaluation Plan. November 2016. • McArthur, Jan and Brice Rambaud. Liberia Media Development (LMD) Monitoring and Evaluation Plan. January 31, 2017. • Project Appraisal Document Land Governance Project 2014-2019. March 2014. • Project Appraisal Document, Rule of Law and Access to Justice Project. • Reporting Secretariat. Annex 4: Results Framework and Monitoring, Public-Sector Modernization Project (PSMP). July 2016. • Reporting Secretariat. Up-Date Report on the Public-Sector Modernization Project (PSMP). October 2016. • Social Impact. Electoral Security Assessment, Liberia: Final Report. August 2016. • Social Impact. Liberia Elections and Political Transition (LEPT): Pause and Reflect Project Review. August 2017 • Tetra Tech. Land and Governance Support Activity Annual Report Year One (August 2015 – September 2016); September 2016. • USAID Mission to Liberia. Liberia Public Sector Modernization Project – Aide Memoire. November 2016. • USAID. Data Quality Assessment – Standard Indicators Access to Justice Cooperative Agreement. April 21, 2017. • USAID. Liberia Land Governance Support Activity (LGSA) Site Visit Report. April 2017. • USAID/Liberia Legal Professional Development and Anti-Corruption (LPAC) Program Monitoring and Evaluation Plan. October 14, 2015. • USAID/Liberia Office of Democracy and Governance. Project Appraisal Document (PAD) Liberia Civil Society and Media (CSM) Project 2015-2020. February 26, 2015. DO2: Economic Growth: Sustained market-driven economic growth to reduce poverty • ACDI/VOCA. Smallholder Oil Palm Support II Program: FY16 Annual Report. November 2016. • ACDI/VOCA. Smallholder Oil Palm Value Chain Market Analysis Report. August 2015. • ACDI/VOCA. Liberia Smallholder Oil Palm Support Program (SHOPS) II: Baseline Survey Report. June 30, 2015. • Alexander, Frank and Lucy Heady. Seizing Potential: An overview of the Liberian marketplace. Building Markets. April 2012. • Bickel, Allison and Paolo Cerutti. Liberia: Domestic Timber Value Chain Analysis. Building Markets. January 2017. • Bishop, Catherine. Smallholder Oil Pal Support Midterm Impact Evaluation. Winrock. May 2013. • Bruins, Amy. Upwardly Mobile: An Overview of the Liberian Economy. Building Markets. July 2013. • Cardno Emerging Markets USA, Ltd. Road Inventory and Condition Survey Report: Bong, Grand Bassa, Lofa and Nimba Counties. Draft. Feeder Roads Alternative and Maintenance Program (FRAMP). March 3, 2017. • Cardno Emerging Markets USA, Ltd. Monitoring, Evaluation & Learning Plan: Feeder Roads Alternative and Maintenance Program (FRAMP). September 29, 2016. 62 • Cardno Emerging Markets USA, Ltd. First Annual Progress Report: Start of Project through September 2016. Feeder Roads Alternative and Maintenance Program. October 31, 2016. • Casavant, Kayla and Earnest Musinamwana. Leaving a Trace: A Report on the Potential Economic Impacts of Mining in Liberia. Building Markets. July 2014. • CDM Smith, Inc. Engineering Services for Rural Road Rehabilitation (ES3R): Final Report, September 2013-April 2016. July 31, 2016. • Cultivating New Frontiers in Agriculture (CNFA). FY16 Annual Report. Liberia Agribusiness Development Activity (LADA). November 5, 2016. • Cultivating New Frontiers in Agriculture (CNFA). Feed the Future Liberia Agribusiness Development Activity (LADA): Monitoring and Evaluation Plan. September 21, 2016. • Development Alternatives, Inc. Food and Enterprise Development (FED) Program for Liberia: Fiscal Year 2015 Annual Report. October 30, 2015. • Development Alternatives, Inc. Food and Enterprise Development (FED) Program for Liberia: Fiscal Year 2014 Annual Report. November 25, 2014. • Development Alternatives, Inc. Food and Enterprise Development (FED) Program for Liberia: Fiscal Year 2013 Annual Report. October 31, 2013. • Development Alternatives, Inc. Food and Enterprise Development (FED) Program for Liberia: Annual Report: October 2011-September 2012. October 30, 2012. • Food and Enterprise Development Activity for Liberia. Final Report, 2011-2016. DAI. March 11, 2017. • International Development Group LLC. Mid-term Performance Evaluation: Liberia Investing for Business Expansion Program and Sustainable Markets Initiative-Liberia. February 2016. • International Development Group LLC. Food and Enterprise Development Project Impact Survey. February 2017. • Liberia Strategic Analysis. Smallholder Oil Palm Support Project Final Performance Evaluation: Final Report. August 2017. • Liberia Strategic Analysis. Performance Evaluation of People, Rules, and Organizations Supporting the Protection of Ecosystem Resources. July 2016. • Liberia Strategic Analysis. Qualitative Assessment of Farm-to-Market Road Rehabilitation. August 2016. • Loranger, Kim and Timothy Melvin. The Sustainable Marketplace Initiative: Second Annual Report, October 2012-September 2013. Building Markets. October 2013. • Loranger, Kim and Zurab Eprikashvili. The Sustainable Marketplace Initiative: Annual Report, October 1, 2014-September 30, 2015. Building Markets. October 2015. • Maillard, Emmanual and Jennifer P. Hold. Final Activity Report: Sustainable Market Initiative – Liberia. Building Markets. April 2017. • Maillard, Emmanual. The Sustainable Markets Initiative: Annual Report, October 1, 2015-September 30, 2016. Building Markets. October 2016. • Mbevi, Lydia. Gender Strategy: Smallholder Palm Oil Support II. ACDI/VOCA. November 2015. • Michigan State University. Borlaug Higher Education for Agricultural Research and Development (BHEARD): Performance Monitoring and Evaluation Plan. June 2015. • Musinamwana, Earnest and Daniel F. Togba. Gathering Competitive Momentum: Overview of the Liberian Economy. Building Markets. November 2014. • NRECA International. Performance Monitoring Plan. Beyond the Grid (BTG). August 1, 2017. 63 • NRECA International. Semi-Annual Report-Semester One. Beyond the Grid (BTG). July 3, 2015. • NRECA International. Semi-Annual Report-Semester Two. Beyond the Grid (BTG). January 31, 2016. • Perez, Francisco and Benedict Quaqua. Performance Management Plan. Engineering Services for Rural Road Rehabilitation (ES3R). Revised June 24, 2015. • Performance Management Plan. Food and Enterprise Development Program for Liberia (FED). April 2015. • Power Africa Transactions and Reforms Program. Second Annual Report. December 12, 2015. • Shelmerdine, Howard M. and Michael Kpange. Annual Report – Year 1. Engineering Services for Rural Roads Rehabilitation (ES3R). November 20, 2014. • Tetra Tech ARD. People, Rules and Organizations Supporting the Protection of Ecosystem Resources (PROSPER): Updated Performance Management Plan – FY 2016 (Draft). October 2015. • Tetra Tech ARD. People, Rules and Organizations Supporting the Protection of Ecosystem Resources (PROSPER): Annual Report, FY 2013 (Draft). November 2013. • Tetra Tech ARD. Annual Report, FY14: People, Rules and Organizations Supporting the Protection of Ecosystem Resources (PROSPER). October 2014. • Tetra Tech ARD. Annual Report, FY15: People, Rules and Organizations Supporting the Protection of Ecosystem Resources (PROSPER). October 2015. • Tetra Tech ARD. Land Rights and Community Forestry Program: Final Report: 14 January 2007-30 October 2011. December 2011. • Tetra Tech ES Inc. Loss Reduction Program for the Liberia Electricity Corporation. Power Africa Transactions and Reforms Program (PATRP). January 26, 2016. • USDA Forest Service Office of International Programs. Liberia-Environmental Threats & Opportunities. 118/119 Assessment. Final Report. December 2014. • Wallace, Eric. Smallholder Oil Palm Support II (SHOPS II) Program: Monitoring, Evaluation and Learning Plan. ACDI/VOCA. Revised May 12, 2017. DO 3: Health: Improved health status of Liberians • Hughes, Jacob. Liberia Fixed Amount Reimbursement Agreement: Costing Study Report. Liberia Strategic Analysis. November 2016. • Intermedia. Assessment of FARA 1.0 Outcomes on the Capacity of MOH and CHTs, and the Quality of Service Provision at Health Facilities in Liberia. April 25, 2017. • Learn Evaluation and Analysis Project - II. Liberia Municipal Water Project Mid-Term Performance Evaluation. Final Evaluation Report. March 31, 2015. • Liberia Collaborative Support for Health (CSH) Program. Annual Summary Report for Fiscal Year 2016 (October 1, 2015 – September 30, 2016) Quarterly Report for Quarter 4 and (July 1 – September 30, 2016). Management Sciences for Health (July 1 – September 30, 2016). October 30, 2016. • Liberia Strategic Analysis. A Summative Report of USAID/Liberia’s Government -to￾Government Funding Mechanism: the Fixed Amount Reimbursement Agreement (FARA) 2011-2015. June 2017. • Liberia Strategic Analysis. Fixed Amount Reimbursement Agreement Project Quarters 11 (October – December 2016) Deliverables Verification Draft Report. June 14, 2017. • Liberia Strategic Analysis. Liberia Fixed Amount Reimbursement Agreement Final Evaluation: Health Outputs Report. May 2017. • Liberia Strategic Analysis. USAID/ Liberia Collaborative Support for Health (CSH): Summary Draft Performance Evaluation Report. June 2017. 64 • Liberia Strategic Analysis. USAID/Liberia Partnership for Advancing Community-Based Services Midterm Performance Evaluation. August 2017. • Maternal Child Survival Program (MCSP); Restoration of Health Services in Liberia. Quarterly Report. Quarter 4 – July 1, 2016 to September 30, 2016 (Fiscal Year 2016). • Maternal Child Survival Program (MCSP); Restoration of Health Services in Liberia. Quarter 5 – October 1, 2016 to December 31, 2016 (Quarter 1, Fiscal Year 2017). • Maternal Child Survival Program (MCSP); Restoration of Health Services in Liberia. Quarterly Report (Project Year 2, Quarter 2, January 1, 2017-March 31, 2017, FY17Q2). • Maternal Child Survival Program (MCSP); Restoration of Health Services in Liberia. Quarterly Report (Project Year 2, Quarter 3, April 1, 2017-June 30, 2017, FY17Q3). • Maternal Child Survival Program (MCSP); Human Resources for Health in Liberia. Project Year 1, Quarter 2 – October 1, 2016 to December 31, 2016 (Quarter 1, Fiscal Year 2017). • Maternal Child Survival Program (MCSP); Human Resources for Health in Liberia. Quarterly Report (Project Year 1, Quarter 3, January 1, 2017-March 31, 2017, FY17 Q2). • Maternal Child Survival Program; Human Resources for Health in Liberia. Quarterly Report (Project Year 1, Quarter 4, April 1, 2017-June 30, 2017, PY1 Q4). • MCSP Restoration of Health Services (RHS) in Liberia. Midline Assessment Findings – In Brief. November 2016. • MCSP Restoration of Health Services (RHS) in Liberia. GHET Quarterly Report. April – June 2017. • Mid-Term Evaluation of the USAID/Liberia Health Sector Fixed Amount Reimbursement Agreement (FARA). 2013. • Office of Inspector General. Audit of USAID/Liberia’s Malaria Interventions. Audit Report No. 7-669-13-002-P. May 14, 2013. • Partnership for Advancing Community-based Services – PACS. Annual Report (FY 2015). • Partnership for Advancing Community-based Services (PACS). Annual Report (FY 2016). • Postma, Sjoerd. Liberia Collaborative Support for Health (CSH) Program. Annual Summary Report for Year 1 (February 27 – September 30, 2015) and Quarterly Report for Quarter 4. Management Sciences for Health (July 1 – September 30, 2015). October 30, 2015. • Tetra Tech. Liberia Municipal Water Project II. Year 1 Quarterly Report 1: October - December 2016. January 30, 2017. • Tetra Tech. Liberia Municipal Water Project II. Year 1 Quarterly Report 2: January - March 2017. April 27, 2017. Revised May 23, 2017. DO 4: Education: Better-educated Liberians • Auten, Sarah, Amit Dabla, and Mark Bassie. Advancing Youth Project Performance Evaluation: Final Report. Liberia Strategic Analysis. May 2016. • Concern Worldwide. Education Quality and Access in Liberia (EQUAL) Annual Narrative Report (FY 2015). October 30, 2016. • Concern Worldwide. Education Quality and Access in Liberia (EQUAL) Annual Narrative Report (FY 2016). October 30, 2015. • Concern Worldwide. Education Quality and Access in Liberia (EQUAL) Report on the Baseline Knowledge Attitude and Practices (KAPs) on gender equitable attitude and school violence rolled out in September and October 2016. February 20, 2017. 65 • Concern Worldwide Liberia. Education Quality and Access in Liberia (EQUAL) Program: Performance Monitoring Plan. June 2016. • Education Crisis Response for Liberia: Second Progress Report. December 2015. • Education Development Center, Inc. Liberia Advancing Youth Project: Performance Monitoring Plan. May 29, 2012. • Education Development Center, Inc. Year 3 Quarter 4 and Year 3 Annual Report. Advancing Youth Project. October 2014. • Education Development Center, Inc. Year 1 Quarterly Report 4 and Annual Report. Advancing Youth Project. October 2012. • Education Development Center, Inc. Year 2 Quarterly Report 4 and Annual Report. Advancing Youth Project. October 2013. • Education Development Center, Inc. Year 5 Annual and Quarter 4 Report. Advancing Youth Project. October 2016. • Education Development Center, Inc. Year 4 Quarter 4 and Year 4 Annual Report. Advancing Youth Project. October 2015. • Liberia Monitoring & Evaluation Program. Mid-term Assessment of the Excellence in Higher Education Liberian Development (EHELD) Project. May 2014. • Liberia Strategic Analysis. Education Quality and Access in Liberia (EQUAL) Final Performance Evaluation: Final Report. July 2017. • Liberia Strategic Analysis. Excellence in Higher Education for Liberian Development (EHELD) Final Performance Evaluation: Final Report. June 2017. • The Liberia USAID Education Crisis Response Programme. Monitoring and Evaluation Plan. December 2017. • RTI International. Examining Barriers and Persistence Factors for Female Students in the Engineering and Agriculture Degree Programs at the University of Liberia and Cuttington University. July 2013. • RTI International. Excellence in Higher Education for Liberian Development (EHELD): Performance Management Plan. February 24, 2016. • RTI International. Excellence in Higher Education for Liberian Development (EHELD): Annual Report. October 15, 2011. • RTI International. Excellence in Higher Education for Liberian Development (EHELD): Annual Report. October 15, 2012. • RTI International. Excellence in Higher Education for Liberian Development (EHELD): Annual Report (Fiscal Year 2013). October 22, 2013. • RTI International. Excellence in Higher Education for Liberian Development (EHELD): Annual Report (Fiscal Year 2014). October 15, 2014. • RTI International. Excellence in Higher Education for Liberian Development (EHELD): Annual Report (Year 5). October 15, 2015. • RTI International. Excellence in Higher Education for Liberian Development (EHELD): Annual Report (Year 6). October 2016. 66 ANNEX IX: INTEGRATION IN STRATEGIC PLANNING The Meta-Analysis recommendations USAID/Liberia develop an integrated DO in its new strategy for FY 2019 – FY 2023. It suggests the Mission consider gender, youth, and STIP as potential focuses and notes the importance of securing Mission leadership, systematized cooperation, and broad commitment for an integrated DO to succeed. Integrated DOs Address Different Topics The Meta-Analysis found 30 percent of CDCS’s available on www.usaid.gov have integrated DOs (see diagram to the right). Eight percent of CDCS’s have an integrated DO supporting youth development. Four percent have an integrated DO on STIP, and three percent have an integrated DO supporting gender equality. Other integrated DOs include: focusing on a specific geographic region; combining two technical areas, like education and health, into a single DO; human capital, which involves health, education, and EG; social development, which involves health and education; and resilience, which may involve DRG, EG, health, and education, depending on the anticipated type of crises. Examples of Integrated DOs Please find below examples of integrated DOs from the CDCS’s of eight Missions. Integrated DO Mission (Click link to the CDCS) DO Name Description Youth Rwanda DO 4: increase opportunities for Rwandan youth to succeed in schooling and modern workplace USAID/Rwanda’s DO 4 combines education and EG to support youth. The Mission aims to support to the Ministry of Education in its efforts to provide children basic foundational skills at the primary school level and to strengthen workforce readiness training systems both in the public and private sector for in and out-of-school youth. The DO has two IRs: 4.1 Improve literacy outcomes for children in primary grades; and 4.2 Increase stable employment for vulnerable youth. Gender Jordan DO 4: enhance gender equality and female empowerment USAID/Jordan’s DO 4 merges DRG, EG, health and education to support gender equality. The Mission intends fort female empowerment and gender equality to be enhanced as changes in discriminatory social norms and practices are promoted, advocacy and policy reforms for women’s issues are enhanced, and access to female-centered services are expanded. The DO has three IRs: IR 4.1 Changes in discriminatory social norms and practices promoted; IR 4.2 Advocacy and policy reformed enhanced; and IR 4.3 Access to women- and girl￾centered services expanded. STIP Indonesia DO 4: increase collaborate achievement in science, technology, and innovation USAID/Indonesia’s DO 4 focuses on the development of science, technology, and innovation. The Mission aims to improve research quality and productivity, linkages to broader scientific communities, education opportunities and standards especially for women students and academics, and increase the technical and management capacity of research institutions to foster more sustainable programs and approaches to development issues. The DO has three IRs: IR 4.1 Academic capacity and scientific research strengthened; IR 4.2 Evidence-based decision-making enhanced; and IR 4.3 Innovative approaches to development utilized. Geographic Focus Guatemala DO 2: improved economic growth and social development in Western Highlands USAID/Guatemala concentrated its DRG, EG, health, and education programming in the country’s Western Highlands, taking advantage of funding from Feed the Future and the Global Health Initiative. The Mission phased out programs not in this geographic focus. The DO has three IRs: IR 2.1 Broad￾based economic growth and food security improved; IR 2.2 Access to and use of sustainable quality health care and nutrition services expanded; and IR 2.3 67 Education quality and access improved. Combination of Two Traditional DOs Guinea DO 2: strengthen democratic governance and economic processes USAID/Guinea’s DO 2 explicitly combines EG and DRG in a single DO. The Mission intends to increase participation by the Guinean people in their own governance to provide a basis for deepening and eventually consolidating democracy and economic growth in Guinea. The DO has three IRs: IR 2.1 Transparent, competitive, accountable governance strengthened; IR 2.2 Conflict prevention and mitigation promoted in the forest region; and IR 2.3 Agricultural education and rural micro-enterprises strengthened. Human Capital Zambia DO 3: improve human capital USAID/Zambia’s DO 3 combines education, health, and EG to develop the human capital that people need for life and work. The Mission aims to foster an educated populace that is capable of embracing new technologies while able to make sound decisions that affect the health and welfare of families and economic livelihoods. The DO has two IRs: IR 3.1 Educational achievement in reading and math improved; and IR 3.2 Health status improved. Social Development Malawi DO 1: social development improved USAID/Malawi’s DO 1 combined health and education to improve citizens’ social development. The Mission intends to improve the quality basic services to improve health and education services. The DO has two IRs: IR 1.1 Availability of essential social services expanded; and IR 1.2 Quality of essential social services improved. Resilience Zimbabwe DO 2: Resilience – Adaptive Capacity of Vulnerable Communities and Households Improved USAID/Zimbabwe merges DRG, EG, and health to reduce the risk of recurrent climate shocks and stresses. The Mission intends to mitigate drivers of conflict, diversify and improve livelihoods, and strengthen human capital to manage the risks faced by vulnerable communities and households. The DO has four IRs: IR 2.1 Risk from recurrent climate shocks and stresses reduce; IR 2.2 Drivers of conflict mitigated; IR 2.3 Economic resilience improved; and IR 2.4 Human capital strengthened. How an Integrated DO Works Missions implement integrated DOs differently, but their approaches share certain roles and responsibilities, and organizational changes. Roles & Responsibilities Organizational Changes • Front Office leadership establishes integration as a priority • Program Office facilitates and champions integration • Technical Offices establish a vision for integration and manage the effort • Design integration into new projects and activities, and integrate funding streams • Hold implementing partners (IPs) accountable for their integration efforts • Engage the host country government in integration Below are examples of how USAID/Malawi and USAID/Guatemala implemented the integrated DOs from their CDCS’s. USAID/Malawi’s 3C Approach to Integration USAID/Malawi established integration as a priority in its CDCS for FY 2013 – 2018. It has an integrated DO entitled “DO 1: social development improved”, which merges health and education technical areas. Furthermore, the Mission envisioned a “3C approach” for operationalizing integration: • Co-location of USAID interventions/activities • Coordination within USAID and with other development partners • Collaboration between USAID and the Government of Malawi (GOM), district authorities, other development partners, civil society organizations, and community-based organizations A stakeholder analysis conducted by Social Impact of USAID/Malawi in 201614 recommended the Mission continue its integration efforts. The analysis confirmed the importance of establishing roles and responsibilities to implement 14 Mandel, Jennifer, Alison Smith, etc. 2016 Malawi Stakeholder Analysis: Integrated Development in Malawi. Jan. 2017. Link. 68 integration. The Mission’s Front Office set integration as a priority, giving it a prominent role in the CDCS. It encouraged different offices to take a leadership role in identifying opportunities for integrated development, rather than relying on IPs to do it. USAID/Malawi’s Program Office facilitated integration by organizing stakeholder meetings and all-partner meetings. The Technical Offices played the critical role of establishing a vision and managing the integration effort. They identified potential integration activities by vetting ideas from all-partner meetings. The Mission facilitated integration through organizational changes, including designing integration into new activities, developing a definition and guidance for integration, and holding IPs accountable for prioritizing integration in their work plans. The Program Office assisted the Technical Offices with developing indicators for IPs to measure integration. The Mission also established a mechanism to engage GOM in district-level meetings in priority zones of influence. USAID/Guatemala’s Western Highlands Integrated Program USAID/Guatemala set integration as a priority in its CDCS for FY 2012 – 2016. It put forth an integrated DO with a geographic focus entitled “DO 2: improved economic growth and social development in Western Highlands”. The Mission took a systematic approach to establish collaboration between Technical Offices, government entities, and IPs, which is documented in a blog on USAID’s Learning Lab15 and a report by USAID/Guatemala.16 The Mission focused its Feed the Future (FTF) and Global Health Initiative funding in the Western Highlands, which has the country’s highest poverty and malnutrition. It established an intra-Mission working group for the Western Highlands with representatives from each Technical Office. This working group collaborated on the design of projects and implementing mechanisms to ensure consistency with FTF principles and met regularly to promote integration and communication among Technical Offices. USAID/Guatemala overcame several challenges to institutionalizing integration. It found large-scale coordination required resources and time from staff who were already busy. Some staff perceive it was easier to implement activities in isolation rather than cooperate on large complex problems. Furthermore, IPs and Technical Offices were reluctant to collaborate when it meant surrendering some ownership of their activities. The Mission identified leadership, cooperation, and commitment as critical to success. The Mission Director prioritized integration by having the intra-Mission working group report directly to him. USAID/Guatemala wrote the principles of collaboration into implementing mechanisms and letters of understanding between IPs. The Mission allowed integration efforts to evolve over several years and adapted structures and models to overcome challenges that emerged. 15 How USAID/Guatemala Is Using Collaboration and an Integrated Approach to Achieve Development Objectives. Dec. 2013. Link. 16 Integration of USAID in the Western Highlands. May 2013. Link. Malawi: Integrated Funding An obstacle to integration is different funding streams. USAID/Malawi’s Contracts Offices and Technical Offices worked together to create integrate funding. They found the most effective time to do this was at an activity’s inception. For example, the Mission commenced a local governance project funded 50% by DRG and 50% by other sector offices. The Mission commenced five jointly-funded activities in two years, and most staff felt they had saved money by avoiding a duplication of resources. Guatemala: Integrated Funding The Mission directed FTF resources to finance local governance and nutrition activities managed by the DRG, health, and education offices. This ensured a shared level of accountability, interest, and collaboration among these Technical Offices, linking them to EG. Additionally, performance reports and portfolio reviews required teams to work together on shared inputs. 69 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523