META-EVALUATION OF USAID/LIBERIA'S PORTFOLIO 2016–2020 July 2020 This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by Social Impact, Inc., under the Liberia Strategic Analysis activity. PHOTO: ALEXANDRA DRYER What's in this report? 1. Guide to this Report Abstract, Background, and Limitations 2. How do we learn? The LSA Process 3.What did we learn? Examples of Learning Tools in Action 4. The Learning Process A Roadmap for Learning after Evaluations 5. Learning is Not Reading a Report Conclusions and Utilization 6. How-To Selecting a Learning Tool 7. Acronyms Abstract The meta-evaluation synthesizes 31 evaluations and assessments that were conducted between 2016 and 2020. These reports evaluate $464 million USD in investment by USAID/Liberia and review the context and activities of seven major sectors and 20 subsectors. As a meta-evaluation, this report does not evaluate a particular activity, but rather reflects on the evaluation process and high-level findings that emerged across reports. What emerges is guidance for both future implementation and evaluations. To begin, the report illustrates a step-by-step process into evaluations, from research design to data collection to learning events. We provide examples of learning in action—specifically, from findings and challenges identified in evaluations and assessments conducted by Liberia Strategic Analysis (LSA). Each of the 31 reports is included in key highlights, examining first the main cross-cutting lessons. The Development Objectives (DOs) for Economic Growth, Health and Education, and Democracy, Rights, and Governance are further examined alongside standalone reports on gender and stakeholder consultation. However, findings do not necessarily generate learning unless the information is interacted with and processed. The report outlines a roadmap for learning—from Development Experience Clearinghouse (DEC) publication to hosting a learning event to a post-activity review. LSA shows how its own evaluations have been utilized and concludes that learning is more than reading a report. Selecting a learning tool—whether a performance evaluation or assessment—can be complicated. The “How-To” section leads through the learning tool selection process so that main factors are considered. In sum, our goal is to present and package all data and findings in a way that is accessible, actionable, and useful beyond a single activity's scope. A Note on Study Limitations This report contains a variety of research methods and tools. The research was conducted by different groups, the designs vary, and research questions are different. Because of this, the evaluations and assessments are not meant to be generalized together. The findings for each report are distinct. The meta￾evaluation brings together these evaluations and assessments to highlight the processes of learning and key findings. The purpose is not meant to combine specific report findings as a way to illustrate new evidence on a specific activity. A Heartfelt Thank You! We dedicate this report to the evaluators, Government of Liberia, civil society, community leaders, learners, entrepreneurs, and citizens. Without your insights and reflections, this level of learning would not be possible. We thank you for the miles you traveled and hours you set aside to speak with us. We thank you for your openness. Our hope is that this report leverages the Liberian voices and your lived experiences to promote the change you envision for your communities and for your country. Thank you to USAID/Liberia for collaborating and taking risks with us to explore and push the boundaries of how evidence can be used to support learning. Sincerely, Carla Trippe Senior Evaluation Advisor, Liberia Strategic Analysis Social Impact Brooke Hill Program Manager, Impact Evaluations Social Impact A Guide to this Report LSA Meta-Evaluation 2020 This report serves as a guide into evaluations and decision-making, with the aim of helping you and your activities benefit from the Liberia Strategic Analysis (LSA) experience. Frequently, the learnings and institutional knowledge gained from data and undertaking evaluations are lost and limited to the lifespan of a single activity. This meta-evaluation seeks to prevent this from happening and enable continuous learning. We bring together the lessons learned, findings, and recommendations from all assessments and evaluations conducted since 2016, synthesizing LSA’s experience into replicable tools. LSA conducted its first evaluation in 2016 and ultimately conducted 31 evaluations and assessments on activities, including 27 activity evaluations and seven contextual assessments. The evaluations and assessments span across seven main sectors and 20 subsectors and examine over $464 million USD in activity investment by USAID/Liberia. LSA, implemented by Social Impact, supports USAID/Liberia by providing innovative monitoring, evaluation, and learning (MEL) assistance to facilitate data-driven decision-making. LSA’s goal is to improve the effectiveness and sustainability of USAID/Liberia’s portfolio by ensuring that USAID can learn from and adapt to its operating environment. USAID/Liberia has used this body of evidence to design its new 2019–2024 Country Development Cooperation Strategy (CDCS); engage in Mission portfolio reviews; undertake course correction of activities; design new activities and Project Appraisal Documents (PADs); and strengthen partnerships with the Government of Liberia (GOL), institutions, and civil society. LSA’s approach emphasized utilization and ownership from the design to dissemination phase of each analytical exercise. LSA sought to go beyond the Automated Directives System (ADS) 201 and treated every analytical exercise as an opportunity to test innovative learning approaches and develop tailored designs for technical teams. Each research activity served a unique purpose to USAID staff that was thoughtfully reflected in the scope, analytical tools, and engagement strategy. Together with USAID/Liberia, LSA tested multiple approaches to build the base of its current evaluative toolkit, including activity reviews, midterm stocktakings, and pause and reflect exercises. Think of this report as a toolkit with each section as a tool. These tools are: 1. How do we learn?: The LSA Process 2. What did we learn?: Examples of Learning Tools in Action 3. The Learning Process: A Roadmap for Learning after Evaluations 4. How-To: Selecting a Learning Tool Present and package all data and findings in a way that is accessible, actionable, and useful beyond a single activity's scope. Our Goal. How do we learn? The LSA Process We developed the LSA process by asking, “How do you do utilization-focused evaluation and adaptive activity adaptation in a post-conflict context like Liberia?” To begin, LSA supported USAID in prioritizing stakeholder inclusion in the evaluation approach. We emphasized building buy-in and ownership from the design to the dissemination phases so that all development outcome stakeholders use the evidence produced. As explained above, LSA’s process begins with learning and incorporates learning throughout. During each step, you can also add stakeholder meetings. This engagement ensures each step is contextually relevant and can bolster relationships and local buy-in, encouraging sustainability. LSA fine-tuned this process with USAID/Liberia to ensure we are at the forefront of stakeholder engagement, facilitating collaboration at critical moments. Prolonged engagement and collaboration is crucial to development effectiveness and supporting self-reliance. Step 1: Learning Agenda Step 2: SOW & Design Step 3: Data Collection Step 4: Analysis & Debrief Step 5: Report Step 6: Learning Step 7: Utilization Survey Determine when information is needed at all levels of decision￾making. Anticipate evaluation questions that feed into CDCS and allow the Program and Project Development (PPD) Office to provide input on the most strategic use of funds for evaluations that reach beyond solely activity￾level learning. Use the statement of work (SOW) development and evaluation design step to engage stakeholders linked to the success of your implementation. Hold planning meetings with the government and implementer to allow them to shape the evaluation approach before finalizing. Ongoing engagement fosters long term self-reliance by helping local stakeholders think through how they can use the data. Data collection tools and processes should leverage local voices and contexts. Work closely at the county and district levels to follow protocols that align with your implementing partner (IP) and the intervention that will be evaluated. Participatory analysis workshops allow for deep engagement in findings and create opportunities for quicker evidence use. Shorter debriefs can be used for USAID/ Liberia staff and management teams unable to set aside the time for a workshop. The analyzed output, in draft report form, should be shared with relevant stakeholders and government partners. This is done to continue engaging stakeholders in shaping the final product. Create a two-page summary of the final report to support dissemination to a wider audience. Produce tailored events that engage all relevant stakeholders to develop an action plan that operationalizes the recommendations. Use a tracking survey to analyze the value of evaluations and uptake of recommendations. This should inform ongoing adaption of evaluation approaches. How to use this tool: The LSA Process You’ve decided to conduct an evaluation or assessment, but don’t know where to begin? Start here. This section guides you on a step-by-step generalized evaluation process from figuring out timelines and agendas to close-outs. What did we learn? Learning Tools in Action What did USAID/Liberia learn from utilizing this approach? In this section, we will draw out the key findings from the 31 evaluations and assessments since 2016. The findings are in three parts: 1. A list of the five common findings that cut across each Development Objective (DO) and sector. 2. Findings on gender and stakeholders. 3. Findings specific to each of the three DOs, as well as the systematic challenges faced. In the process of applying capacity assessments and verification tools, partners become more aware of the donor’s expectations or international best practices for delivering a certain service. We use them to measure improvement in capacity, but they provide an added level of learning for CSOs or government partners because they begin to understand the steps required to achieve their desired organizational capacity goals. It gives them the tools to plan for how to achieve their strategy. 1. Capacity assessments and verification tools serve as capacity development opportunities. 3. Liberians care about what is relevant to them, and progress is better if it matters to them. 2. Working with a government agency requires benchmarks and consistent presence. 4. Irregular pay critically affects civil servants' ability to deliver services. 5. Poor internet access inhibits basic work. Cross-Cutting Findings Across the 31 evaluations and assessments, we found five cross-cutting lessons: Activity design should require Memorandums of Understanding (MOUs) with government institutions that define the scope of activity contributions and set clear, written expectations between implementing partners (IPs) and government entities. Long-term embedded advisors may be able to improve institutional capacity, but must maintain a physical on-site presence to minimize unmet expectations from government partners. Solutions must be driven by beneficiaries or stakeholders using evidence-based design. IPs often did not achieve targeted results because Liberians first needed to meet their basic needs. For example, immediate access to food or income is often a priority over sending children to school, going to the clinic for treatment, sustainably harvesting natural resources, or investing in agribusiness approaches that produce higher yields. Liberia lacks the financial infrastructure needed to disburse civil servant salaries outside Monrovia. Paydays vary, salaries fluctuate considerably, and civil servants face significant costs to collect their salaries. In 2018, education and health workers spent an average of 35 hours per month traveling to the bank to collect their salary (MSTAR). They are subsequently absent from schools and health facilities, leading to undelivered public services. The GOL often cited lack of access to the internet as limiting their ability to do basic work, both at central and local levels. As Liberia is moving toward improving fiber optic internet access, follow-up research should assess the impact on ministry-level work. What did we learn? Learning Tools in Action How to use this tool: Learning Tools in Action Each DO will begin with a table. The table includes links (when applicable) to the report with the "Learning Tool" and Sector. Use this table to see how learning tools are used in action! Agriculture Agriculture Date Learning Tool Sector Qualitative Assessment of Farm-to-Market Road Rehabilitation August 2016 Assessment Infrastructure The Smallholder Oil-Palm Support Final Evaluation September 2017 Performance Evaluation Agriculture Liberia Agribusiness Development Activity Midterm Evaluation July 2018 Performance Evaluation Agribusiness Informal Cross-Border Trade Survey Final Report July 2018 Assessment Trade and Infrastructure Natural Resource Management & Biodiversity Date Learning Tool Sector Peoples, Rules, and Organizations Supporting the Protection of Ecosystem Resources Final Performance Evaluation July 2016 Performance Evaluation Natural Resource Management Forest Incomes for Environmental Sustainability Midterm Evaluation June 2018 Performance Evaluation Natural Resource Management Biodiversity Sector Assessment January 2020 Assessment Biodiversity DO 1: Market-driven, inclusive economic growth supporting increased job creation To holistically strengthen value chains, implementers applied vertical integration approaches, providing training, site-based assistance, financial support, and inputs to various actors within a value chain. In the Smallholders Oil Palm Support Project (SHOPS) II, trainings focused on best business practices, marketing, and financial record-keeping to improve business management and decision-making skills. SHOPS I and II provided smallholder farmers with Tenera oil palm seedlings and built up manufacturers of manual and motorized oil palm processors (called Freedom Mills). LADA contributed to a modest increase in private sector investment in agricultural input systems and in post-harvest handling, storage, packing, transport, marketing, and auxiliary services. Most economically sustainable progress was seen in the vegetable, cassava, and rice value chains. Vegetables: Activities contributed to production and marketing, mainly through support to agricultural input providers. Vegetable growers were the main customers for agricultural inputs such as fertilizer, pesticide, and fungicide (compared to other value chains). Cassava: There is a small but growing demand for value-added gari products. Some cassava processors reported increased sales following LADA cassava value chain support, while others reported losses from cassava processing. Targeting the small but growing demand for value-added gari products may offer sustainable market opportunities. Rice: LADA increased purchases specifically in large-scale private sector processors in Lofa, including diversification of sales into domestic rice markets and linkages with wholesale rice dealers in Monrovia. While two private enterprise Lofa County rice processors were found to increase purchases, in general, LADA activities were not yet significantly increasing investment by the private sector through inputs, processing, or marketing services. Value Chain Strengthening What did we learn? Learning Tools in Action Agriculture DO 1: Market-driven, inclusive economic growth supporting increased job creation Road rehabilitation efforts in Bong, Lofa, and Nimba improved access to service delivery, including transportation for economic activities, health services, and education services. Road rehabilitation increased the availability of transportation and led to reductions in travel time and costs. Reduced transport costs had broader economic effects, such as reducing transaction costs for economic actors and increasing the number of buyers and sellers along value chains. Road rehabilitation improved access to health care, including ambulance services and increased health facility visits. Increased availability of transportation also affected the delivery of education services, such as increased student attendance and enrollment. Specific to youth and vulnerable groups, youth boasted about selling farm produce to save money for major purchases, mainly motorbikes, which are an important mode of commercial passenger transport. Physically disabled respondents reported increased access to transportation due to increased availability of cars on rehabilitated road segments. At the borders between Liberia, Cote d’Ivoire, and Guinea, 70 percent of traders are women, and most traders use informal routes. Eighty-one percent use informal routes to avoid costs of formal borders, including fees and bribes. About half of traders do not have a registered business. Despite the popularity of informal borders, the majority of traders felt the informal routes were less safe but did note they thought these routes were shorter. In particular, women said formal borders put them at risk of sexual harassment. This harassment manifests through detaining women and arbitrary enforcement of rules and regulations. The most common imported goods include dried pepper, clothes, slippers, peanut butter, and ground nuts. The most common exports are local rum (cane juice), palm oil, frozen chicken, and kola nuts. Incremental progress has made agriculture more inclusive to women. For example, women received training to manufacture Freedom Mills. However, most activities at the field level have been insufficient to move traditional gender roles beyond unskilled labor. Although activities may be meeting targets on gender participation, they will probably not achieve high-level gender results implied in their Theories of Change (such as making agriculture more inclusive and attractive to women) without targeted support approaches. o The cost of agricultural equipment remains prohibitive, especially in the absence of available financing. o The sustainability of working through rice cooperatives, which USAID has invested in for many years, is challenged by persistent production, management, and marketing capacities. Cooperatives in Liberia have a history of ineffective governance and donor support in the face of these ongoing challenges. o The limited number of entrepreneurs and private sector businesses engaged in agriculture poses difficulties for USAID in terms of partners to work with and build up. o The use of free input distributions, often by other donors, reduces farmer demand for agricultural inputs and, in turn, challenges the ability to improve value chains as well as agricultural dealer sustainability. o Corruption and sexual harassment at formal borders by border officials inhibit their use by traders. Fees for business registration and financing applications are prohibitive to many traders. Systemic Challenges Gender Trade Infrastructure What did we learn? Learning Tools in Action Natural Resource Management & Biodiversity DO 1: Market-driven, inclusive economic growth supporting increased job creation Biodiversity Government Management Community Forest Management An Overview USAID’s technical approach to natural resource management has three pillars: governance, biodiversity, and livelihood. The overarching approach is threefold: 1. A substantial, comprehensive, and long-term commitment to strengthening Forestry Development Authority's (FDA's) capacity. 2. Financial and technical support for the institutions implementing community forestry, until planned sources of revenue are functional and local technical capacity develops further. 3. Development or strengthening of systems for oversight and accountability of community forestry governance bodies. USAID implementers used an inclusive, bottom-up adaptive management approach to create the text for the legal framework of the 2009 Community Rights Law, which serves as the foundation of community forestry in Liberia. This ensured strong buy-in by community members and governance structures. By creating this foundation, communities are given greater security of tenure of their own resources. The way that community forest management bodies (CFMBs) are created mirrors Liberia’s customary system for governance, making the CFMBs effective and sustainable. The model has been refined by working with the same pilot communities throughout the implementation of multiple USAID activities. USAID activities made strides by supporting women and youth to play roles in CF governance, though they have not increased the knowledge and access to information required to exert influence on CFMBs’ decisions. USAID’s close work with FDA to develop the nine-step process made it an institutionalized path for the authorization and management of community forests. Each step of the process engendered the whole community in decision-making, creating high levels of awareness and respect for the management of community forests across Liberia. Additionally, communities benefited from boundary dispute resolution in the process. Most community members expect FDA to fulfill its responsibilities under the law and generally express a high level of trust in their local county-level authorities. They also recognize that FDA and local authorities do not currently have the financial or human resources, or the technical expertise, to support the authorization process and community forestry governance bodies. Community outreach by implementers has raised awareness of community ownership of forests, damaging forest practices and the principle of sustainable use across all stakeholder groups. Community Forest Management Plans are essential mechanisms to guide sustainable production of forest products; however, USAID activities made little if any progress in producing plans. The People, Rules, and Organizations Supporting the Protection of Ecosystem Resources (PROSPER) activity developed a manual for monitoring forest biodiversity and supported biodiversity monitoring training; however, systematic biomonitoring of community forests is not occurring. What did we learn? Learning Tools in Action Natural Resource Management & Biodiversity DO 1: Market-driven, inclusive economic growth supporting increased job creation Systemic Challenges Alternative Livelihoods Alternative livelihood interventions aim to ensure that forest-dependent communities do not suffer adverse livelihood impacts from community forestry. Some progress has been made, but generally, the support through USAID activities does not offset illegal or unsustainable forest uses. Livelihood interventions in their current form only benefit a small percentage of community members, and new and unfamiliar livelihoods are not sustained beyond the intervention. Community members benefited from technical knowledge of agricultural practices consistent with biodiversity protection. Forest Incomes for Environmental Sustainability Activity (FIFES) supported the establishment of Women Operating Resources Together (WORTH) groups, through which women learned to keep track of finances and to read and write. These skills improved their self-confidence in starting small-scale businesses. The main challenges include: o A need for communities to understand the true commercial value of their forests. o A need for communities to understand the consequences of their decisions when agreeing to changing livelihood strategies, including setting aside community forests for conservation. o A need for a Community Rights Law 2009 amendment to add social safeguards that protect communities from exploitation by powerful actors in the commercialization of forest resources. Since the community forests have produced few benefits for the communities so far, members are urging the CFMBs to negotiate timber sale agreements with logging companies. The FDA is actively helping logging companies and communities make these agreements (likely due to the 45 percent of the income from sales of community forests timber that the government receives). Of the 11 community forests, some have already signed logging agreements, and most of the others are interested in doing so. The CFMBs are unprepared to negotiate fair, technically sound logging agreements, or to monitor and enforce compliance by logging companies within the terms of such agreements. CFMBs lack capacity in management of forest resources, understanding of forest value, and power dynamics with commercial companies. It appears likely that logging contracts will fail to provide mechanisms for silviculture, low-impact logging, and forest management practices that protect community forests’ long-term sustainability. At the same time, the CFMBs are unprepared to manage the potential income from timber sales from logging companies, and disagreement over the management of funds is likely to weaken community forest governance. Interventions implemented to protect an ecosystem require behavioral changes of all stakeholders around a community forest and/or Protected Area. Where interventions only engage few community members in alternative livelihood efforts, they are unable to change the behavior of the whole community. As such, these efforts are unable to reduce the pressures posed by the majority of the population on forests. What did we learn? Learning Tools in Action Good Governance DO 2: Effective and inclusive governance catalyzed through reforms and greater accountability Public Administration Good Governance Date Learning Tool Sector Local Empowerment for Government Inclusion and Transparency Midterm Evaluation August 2018 Midterm Strategy Stocktaking Governance Revenue Generation for Governance and Growth Midterm Evaluation October 2018 Midterm Strategy Stocktaking Tax Administration Evaluation of the Pay and Payroll Process in the Government of Liberia April 2018 Performance Evaluation Public Financial Management Government of Liberia Pay and Payroll Process Mapping Study May 2020 Process Mapping Public Financial Management Rule of Law Date Learning Tool Sector Land Governance Support Activity Midterm Activity Review April 2018 Pause and Reflect Land Governance Reform Legal Professional Development and Anti￾Corruption Midterm Evaluation December 2018 Performance Evaluation Legal Reform Civil Society and Media Date Learning Tool Sector Electoral Security Assessment August 2016 Assessment Elections Electoral Process After-Action Review May 2018 After-Action Review Elections Civil Society and Media Whole-of-Project Evaluation June 2019 Whole-of-Project Evaluation Civil Society and Media Under the previous CDCS, the biggest lesson of the DO on Democracy, Rights, and Governance (DRG) was learning how to pivot in the face of stalled government laws and policies. This led to many implementers adapting to emphasize bottom-up approaches and generating a demand for reform among citizens, CSOs, and local government officials. Flexibility in contracting mechanisms was critical to allowing implementers to adjust. This also enabled improved coordination with donors, as IPs could adapt to avoid overlap as new donor activities came online. The current pay and payroll reform initiative resulted in one consolidated payroll system for all personnel in the GOL workforce. Groundbreaking results include: o Merit-based and transparent pay (compared to allowances being at the discretion of ministers before). o Salary reductions between 40 and 70 percent for many of the higher-income earners. Despite several rounds of payroll cleaning, the payroll is still not entirely clean. The GOL also needs comprehensive institutional rationalization to determine the number of persons needed to effectively deliver services. The effective delivery of public goods and services is not dependent upon decentralization, but rather, it is dependent upon good governance practices such as dependable and transparent public administration and citizen input into decision-making. Local Empowerment for Development Inclusion and Transparency (LEGIT) made most of its progress by taking a bottom-up approach working with city and county officials and CSOs to begin to provide better service delivery. Revenue Generation for Governance and Growth (RG3) made strides to improve Domestic Resource Mobilization through a more efficient and transparent tax administration and processes. A critical lesson is to transition the Technical Assistance approach from modeling to transferring knowledge and coaching Liberia Revenue Authority staff to steward processes in line with their institutional priorities. What did we learn? Learning Tools in Action Civil Society Engagement Judicial System DO 2: Effective and inclusive governance catalyzed through reforms and greater accountability Rule of Law In terms of land governance, USAID learned that when the passage of a law stalls, progress should be demonstrated by supporting institutions in their ability to implement the law and supporting community-level advocacy to generate demand for the law. In USAID/Liberia’s specific experience, this was with the passage of the Land Rights Act. In doing so, community land governance is strengthened, which, in turn, strengthens a community’s ability to articulate its interests in local and national reform processes. In the formal justice sector, USAID missed an opportunity for needed co-creation with legal and regulatory institutional partners during the design of its formal justice sector strengthening activity. This resulted in inaccurate assumptions during the design and led to implementation of technical support that did not match institutional priorities. USAID used the dissemination of midterm evaluation findings to hold workshops to internally realign the goal of the activity and externally set a joint vision with institutional partners that defined success and responded to USAID’s and other institutions’ strategies. The design of the contract necessitated time-consuming modifications to shift the technical assistance approach, identifying the need for more flexibility in future designs to respond to shifting contextual needs. Through its implementer, USAID achieved increased women’s participation as legal service providers and recipients in both urban and rural communities. Specifically, these include training female magistrates, targeting female law students through scholarships, supporting female law professors teaching new gender law and family law courses, and targeting women and family law issues in pro bono clinics outside Monrovia. This portfolio demonstrated that thoughtfulness and intentionality in the Civil Society and Media (CSM) and Electoral designs were critical to their success. Having well-performing CSM activities meant there was not a need for activity-level evaluations to steer course correction. Instead, USAID used the evaluation as an opportunity to reflect on the strategy-level gains that were made in this sector to inform new CDCS, PAD, and activity designs. Conducting one Whole-of-Project Evaluation is now an ADS 201 requirement for the new CDCS. At the time, LSA designed the evaluation as a Whole-of-Project Evaluation (or Whole-of-Intermediate Result Evaluation) as a best practice in tailoring evaluation approaches to match decision-making needs. The evaluation showed that many of the assumptions included in the 2015 PAD continue to hold, particularly with space existing for civil society and media sectors to engage and report on democracy and government accountability issues. The USAID approach of “working with the grain” focused on realistic results and finding entry points for political will on policy reforms. By intentionally establishing national coalitions to coordinate advocacy efforts for reforms, the new PAD directly improved USAID’s approach to achieving the IR. Dividing civil society and media between two implementers helped partners effectively adapt to USAID priorities, such as shifting emphasis during 2017 to elections. Additionally, the evaluation identified critical gaps that are shaping the new PAD, so there are effective feedback loops connecting political leaders to citizens, using CSO coalitions and media as enabling channels. Regarding elections, USAID/Liberia took an inter-agency approach to conduct an after-action review following the 2017 presidential elections to assesses how the U.S. Embassy in Monrovia supported the electoral process. This captured invaluable lessons, including: o Plan early. USAID started planning for the 2017 elections more than four years in advance with a comprehensive suite of activities. Capacity building activities are generally most effective in the period between elections. It is harder to strengthen institutions 12 to 18 months before elections because they are focused on organizing elections, conducting civic and voter education, and observing elections. o Think holistically about election support. Other DRG activities in media development, local governance, and conflict mitigation were leveraged in support of the elections. It is important to consider, from the design stage, how activities in the mission portfolio could be leveraged to amplify support of other results. o Coordinate with other donors early and share as much information as possible when designing activities to avoid gaps and overlaps. Civil Society & Media What did we learn? Learning Tools in Action Health Systems Strengthening DO 3: Foundation for growth strengthened through a healthy, productive, and educated population Health Health Date Type Sector Fixed Amount Reimbursement Agreement Final Evaluation Costing Study November 2016 Performance Evaluation Health Financing, G2G Fixed Amount Reimbursement Agreement Final Evaluation Capacity Assessment April 2017 Performance Evaluation Health Systems Strengthening, G2G Fixed Amount Reimbursement Agreement Final Evaluation Health Outputs Study May 2017 Performance Evaluation Health Systems Strengthening, G2G Collaborative Support for Health Midterm Evaluation June 2017 Performance Evaluation Health Systems Strengthening Partnership for Advancing Community Services Midterm Evaluation September 2017 Performance Evaluation Community Health Partnership for Advancing Community Services Final Evaluation September 2019 Performance Evaluation Community Health Joint Capital Improvement Project Final and Liberia Municipal Water Project Midterm Evaluation January 2019 Performance Evaluation Water Utility County Health Team Capacity Assessment November 2019 Assessment Health Systems Strengthening At the national level, MOH stewardship of basic health service delivery was strengthened through the Government to Government (G2G) agreement. Also known as the Fixed Amount Reimbursement Agreement Activity (FARA), it provided on-budget support and put MOH in the driving seat to manage service delivery. While the FARA is one of the most management-intensive mechanisms for USAID staff, it is one of the strongest for modeling self-reliance enabling structures within government partners. At the county level, FARA support facilitated decentralization in health sector decision-making, though improvements are still needed in this area. At the community level, performance-based contracting was found to be highly successful in motivating facility staff and increasing their sense of ownership, leadership, and governance over facilities. Though facilities and County Health Teams (CHTs) saw large increases in the capacity of key administrative functions, many faced issues surrounding compliance with financial and human resource management protocols. These weaknesses were largely due to a lack of training and inadequate staffing. Out of 15 CHTs, only three—Lofa, Gbapolu, and Rivercess—had viable management systems that met basic compliance with MOH policies and guidelines, although each also had significant weaknesses, especially in service delivery and supply chain. The central-level MOH controls staffing, pay, performance reviews, drug supply, and trainings with little or no delegation to CHTs. Because of this, CHTs have limited autonomy to develop systems that adhere to the guidelines set out in MOH's shift into contracting-in. The central MOH-CHT organizational structure has clear lines of communication, supervision, and support. However, in practice, supervision has not been effective in maintaining overall management and identifying critical issues at the CHT level. Data quality is particularly problematic, with many CHTs lacking data quality assurance processes. Data is rarely cleaned or analyzed and, at times, inaccurate, with no direct or continuous oversight at the central MOH level. The integrated Human Resource Information System (iHRIS) was introduced in the MOH to help the Ministry track health staff training activities, track management performance, and enroll competent and qualified staff on the GOL payroll. This system offers excellent potential in clear and effective system processing. To make the system usable, the MOH needs to coordinate the use of the system with the Civil Service Agency, which oversees enrolling new civil servants on the GOL payroll. What did we learn? Learning Tools in Action Systemic Challenges WASH Health DO 3: Foundation for growth strengthened through a healthy, productive, and educated population Water treatment plants, distribution systems, and kiosks in Robertsport, Sanniquellie, and Voinjama have been completed, handed over to the Liberia Water and Sewer Corporation (LWSC), and are operational. This is providing close to 90 percent of the population in those cities with clean, affordable water. Health Systems Strengthening o CHTs lack the capacity, materials, and training to maintain proper documentation. Lack or poor quality of documentation prevents CHTs from complying with MOH protocols and inhibits effective monitoring. o Many drugs and commodities are out of stock, and even if available, the infrastructure does not enable compliance for drug storage. o Funding is often delayed. Due to this delay, many basic and required functions of CHTs are jeopardized, including trainings, operations, supervision, and maintaining qualified staff. With continuous and reliable funding, CHTs would be better equipped to address some of their foundational challenges. WASH o LWSC’s full ownership and financial sustainability of the institutional framework is difficult, with changing senior management. Changes in leadership create a lack of ownership and familiarity with the institutional framework and develop resistance to working with other stakeholders, such a private contractors. o From a capacity standpoint, LWSC will likely be capable of managing, monitoring, operating, and maintaining the water systems in the three cities, but they are unable to do so unless the Ministry of Finance and Development Planning disburses subsidies per the MOU. o In terms of cost recovery, achieving financial sustainability of the three water systems is highly unlikely under the present circumstances. The absence of sufficient subsidies may soon result in inferior operation and maintenance practices and demotivated operating staff, which, in either case, would likely affect service provision to end users. 90 percent of the population in Robertsport, Sanniquellie, and Voinjama has clean, affordable water What did we learn? Learning Tools in Action DO 3: Foundation for growth strengthened through a healthy, productive, and educated population Education Date Type Sector Education Quality and Access in Liberia Final Evaluation July 2017 Performance Evaluation Basic Education Excellence in Higher Education for Liberian Development Final Evaluation June 2017 Performance Evaluation Higher Education Advancing Youth Project Final Performance Evaluation May 2016 Performance Evaluation Alternative Education Accelerated Quality Education for Liberian Children Midterm Evaluation September 2019 Performance Evaluation Alternative Education Learning Links Closeout Learning Event February 2020 Pause and Reflect Alternative Education Education Enablers Education Implementers demonstrated progress toward increasing access to education by providing low-cost schooling that incorporates high￾quality teaching and learning materials. Community engagement efforts were consistently cited as the most effective method to encourage parents to provide children and youth access educational opportunities, whether formal or informal. What: The most effective methods were back-to-school campaigns, community awareness, dialogues with traditional leaders, and house-to-house mobilization. Who: Principals, parent teacher associations (PTAs), and facilitators/change makers were effective, since these stakeholders are key community members who can be influential in generating support for education. Coordination with village elders and traditional leaders helped ensure that traditional school activities are held outside of the approved school calendar. In the formal setting, Education Quality and Access in Liberia (EQUAL) demonstrated that helping principals improve record keeping enabled the principals and PTAs to track student and teacher attendance and, thus, provide a basis for follow-up work when data indicated problems. In the informal setting, Learning Links’ Social-Emotional Learning component provided a more holistic approach to understanding, breaking down barriers, and building self-esteem for girls to access educational opportunities. Institutionalization was most effective when the implementer emphasized engaging the Ministry of Education (MOE) from the development of tools through implementation processes. Engagement has to allow for dialogue and joint problem solving (e.g., FGDs or reflective retreats) and include the MOE in every activity and decision-making process. This includes developing curricula, having MOE staff facilitate trainings (e.g., Training of Trainers), jointly monitoring the implementation of curricula, and collaborative work-plan development. Institutionalization at the county, district, and school levels was hampered by lack of equipment (e.g., computers) and difficulty to visit schools (e.g., fuel for motorbikes and bad roads during the rainy season). Sustainability of any education activity is premised on government ownership. Ultimately, the goal is to support MOE approaches and have national standardized curricula, tools, and processes driven by the MOE. However, this level of institutionalization has yet to be achieved, owing to bureaucratic challenges, lack of resources at the MOE, and gaps in how government is engaged. For alternative education interventions, a limitation to understanding their impact and sustainability lies in the difficulty of tracing beneficiaries after participation. This type of measurement is critical but was rarely prioritized. MOE Sustainability What did we learn? Learning Tools in Action DO 3: Foundation for growth strengthened through a healthy, productive, and educated population Systemic Challenges Education Broader factors such as hunger, farming, rainy seasons, early marriage, and traditional schools inhibit access to education. While these factors affect both boys and girls, families often pull girls out of school or alternative education at an earlier age to contribute economically to their families or to attend traditional schools in preparation for marriage. The biggest barriers for youth were their livelihoods conflicting with educational opportunities. There is a cooperative agreement between the MOE and Ministry of Internal Affairs stating that traditional schools are not to be held when the MOE schools are in session, but the experience of implementers reflected that this is not widely enforced. Students continued to attend traditional schools during the school year, missing a significant number of formal and alternative education session days. The attendance of teachers also influenced access to education. When teachers were asked about reasons for being absent, responses centered around the need to travel to get their salary, conducting other livelihoods, or illness. The effect of remuneration on civil servants is discussed in more detail under Cross-Cutting Lessons above. What did we learn? Learning Tools in Action: Gender and Stakeholders Assessments Date Type Sector CDCS Stakeholder Consultation Assessment May 2019 Assessment Stakeholder CDCS Gender Assessment June 2018 Assessment Gender Gender Gender & Stakeholder Assessments In 2012, 28 percent of ministerial-level positions in the GOL were filled by women, which just misses the 30 percent female staff goal. Even though women are engaged in the political system, women continue to suffer deep inequities in multiple sectors, primarily due to a lack of education and the gender education gap. Very few women work in professional, managerial, technical, or skilled labor roles. Instead, women are largely considered “unemployed” even when they work in the agricultural or service sectors. Women are hindered by insecure land ownership, lack of access to credit, and general financial obstacles. Notably, the impact of the Ebola outbreak in 2014 to 2015 on women’s economic empowerment continues to linger, namely by worsening access to village savings and loan activities and cross-border trade. Health data remained the same between the 2009 and 2018 Gender Assessments. Fertility rates, maternal mortality, and the percentage of women who give birth by age 15 did not change significantly. Sexual-based violence, most prominently rape, is one of the most frequently reported crimes. Liberia has one of the highest rates of sexual violence in the world and widely practices female genital cutting, despite national-level laws making this practice a misdemeanor. Although women are becoming literate at a faster rate than their male counterparts, a large gap remains between men and women in terms of overall education. This is especially true for girls’ retention at the secondary and tertiary levels. Sexual and gender￾based violence are one of the main reasons for girls leaving school. Since 2009, the Ministry of Gender and Development began incorporating children and social protection into its mandate and became the Ministry of Gender, Children, and Social Protection in 2014. The GOL established the National Gender Policy 2016–2021 in addition to developing the Liberia National Action Plan in line with the UN Resolution. In 2018, President Sirleaf signed the Executive Order on Domestic Violence, which broadens the definition of domestic violence. Challenges: Poverty and lack of formal employment opportunities, especially in rural areas, lead youth to perpetuate gender roles that continue to marginalize women. Employment for women, though available, is often informal and provides little to no job security. Nutrition continues to be a challenge for many Liberians, with one-third of Liberian children experiencing stunting. Boys are more likely to experience malnutrition than girls. The overarching goal of the stakeholder consultation was to identify successes that can be built upon in future strategies and to improve the design and implementation of the least impactful activities, thereby improving the lives of marginalized populations directly affected by USAID activities. Overall, stakeholders were pleased with USAID/Liberia’s work. Participants were satisfied with procurement and contracting processes, as well as adaptive management techniques. Education sector activities were the most impactful, especially in the areas of improving quality of and access to education. Even so, education governance needs to continue to improve. In the health sector, stakeholders expressed the most gratitude for increased access to safe water and sanitation through the provision of water pumps. Stakeholders also emphasized improvements in maternal and child health services due to USAID's support of physical infrastructure and CHT capacity. Despite these successes, the health sector had the largest number of activities considered least impactful. In particular, communities cited that although hand pumps are helpful, some were malfunctioning or no longer in use because of servicing and maintenance issues. A second major issue is the lack of access to drugs, which hinders the effectiveness of community health management systems. USAID/Liberia is already implementing many of the participatory development “best practices,” including assessments and informing participants prior to implementation. The main factors for influencing success were local ownership, capacity, strong accountability systems, and integration of activities. Implementation success can be enhanced through USAID local representation and adequate monitoring of implementers. Challenges: Community members are not always engaged extensively as high-level stakeholders, which can impact ownership and sustainability. More specifically, youth are sometimes unintentionally excluded or invalidated. Of activities that are not sustained, many fail simply because USAID/Liberia did not follow up. Local ownership requires continuous engagement. Stakeholder Consultation 1 2 Plan Learning Event Now that the report is published, invite USAID to meet to plan the learning event. When you meet, consider the scope and purpose of the learning event. This will dictate timing and location. Depending on the purpose, the learning event can take place anywhere. It can be in the implementer’s field office, a bigger venue in Monrovia to accommodate beneficiaries and GOL stakeholders, or your conference room. Host Learning Event Host a learning event! This event usually takes place one to six months after the report is finalized, pending USAID availability. Timing also depends on how the event is going to be used—sometimes, it needs to inform the drafting of the next work plan or another pivotal moment. 3 Report is Finalized 4 After-Action Review The purpose of the learning event will inform how you shape the after-action review component. You can take many approaches to validating recommendations and operationalizing them, but at the end of the day, it should be documented in an action plan that is shared with everyone after the learning event. Utilization Survey Finally, consider measuring the value and usefulness of your evaluation or assessment. This could be done by sending a survey to USAID and the implementer on utilization of the recommendations about 3–6 months after the learning event (or publishing of the report if a learning event will not take place). This gives USAID and the implementer a little time to implement the recommendations before surveying them. The data can feed into your own performance indicators and help you to adapt your evaluation approach over time. 5 Congratulations on submitting your final report to USAID! First, you need to receive clearance to publish the report to the DEC. Consider creating a visually engaging two-page summary of the final report to support dissemination to a wider audience. The Learning Process A Roadmap for Learning After Evaluations How to use this tool: The Learning Process Congratulations on finalizing your evaluation or assessment! Let's make sure all this work is useful to USAID and your stakeholders. Follow this roadmap to make learning continuous and engaging. We believe learning is not reading a report. As the roadmap indicates, the learning process ends with a utilization survey. LSA conducted a survey to measure the efficacy and general perceptions of its evaluations. Of all the evaluation recommendations (total of 169), 72 percent were accepted. In fact, only three percent of recommendations were not accepted. Eighty-two percent of respondents also felt that LSA evaluations were most impactful on implementation effectiveness. With collaborative learning and continuous stakeholder engagement, 86 percent of these evaluations were deemed timely for decision￾making, and 91 percent were relevant to decision￾making. What does learning look like? We believe learning is not simply reading a report. That’s why LSA put a lot of thought over the years into how evaluations could give USAID staff the information they need—packaged in the right way and at the right time—for decision-making. We created more interactive analysis sessions earlier in the evaluation process for USAID staff to engage with the data and created two-pagers and truncated versions of evaluations through more collaborative workshop models that brought together stakeholders to reflect on progress and ways forward. These resulted in a shorter timeframe for USAID staff to get the information they needed. Of course, we recognize that USAID and implementers do not work in a vacuum, and we need stakeholders to realize the development objectives. That’s why learning events are so important and have been successful tools to improve how USAID collaborates with implementers and the GOL toward adaptive management. Results in Action: How were LSA recommendations utilized? How will all this information be used in decision-making? Findings can be used to inform: o Country Development Cooperation Strategies o Mission Portfolio Reviews o Project Appraisal Documents o Activity course correction and designs o Strengthen partnerships with governments, institutions, and CSOs o Identify best practices 16% Partially Accepted 9% Pending 3% Not Accepted 72% Accepted Percent of LSA Recommendations Accepted by USAID 1 2 3 4 How-To: Selecting a Learning Tool How to use this tool: Selecting a learning tool If you don't know where to begin, start by asking why you want to learn in the first place. Once you've answered this question, you can look through which tools are available. These include simple monitoring and more rigorous processes like performance evaluations. To determine which tool you should select, you need to review your resources. Each tool requires different resources. Finally, select your learning tool. What will you select from the toolkit? The Why. The Tools. The Resources. The Toolkit. Why do we want to learn, and what do we want to learn about? What are the tools that we can use to learn? What resources do you have to consider before selecting a learning tool? How do you select a learning tool? Learning is the intentional process of generating, capturing, sharing, and analyzing information and knowledge from a wide range of sources to inform decisions and adapt programs to be more effective. —USAID Learning Lab The Why. To begin, we ask ourselves, what is the reason we want to learn in the first place?The answer can inform what type of tool we select.We may be looking to do one or more of the following: Gather Evidence Information gathered from learning tools can be used as evidence for determining option years or advocating for funds or policy changes from USAID/Washington. It can also be used as a policy advocacy tool with local stakeholders, including government counterparts. Inform Current or Future Activities Evaluations, assessments, and other tools can be used to provide information about current and future activities. For example, an endline performance evaluation (conducted at the end of an activity) can illuminate how an activity performed in comparison to a baseline performance evaluation. An assessment of a market can inform a future activity regarding a supply chain. Meet Requirements Often, USAID and other contracts require learning to be incorporated into activities. What are your contract's requirements? Ensure you are familiar with ADS 201. Foster Sustainability Learning tools are critical to identifying the best methods of operationalizing sustainability. Local buy-in and stakeholder engagement are crucial for activity outcomes to persist beyond the end of activity funding. Learning tools can provide opportunities to engage stakeholders throughout the learning process, creating buy-in and engendering comprehensive support for the activity. Optimize Efficiency Learning tools can identify productivity gaps, as well as ways to maximize resources. Investing in learning tools can lead to increased cost efficiency and more effective use of resources. How-To: Selecting a Learning Tool Step 1 TheTools. You've determined why you want to utilize a learning tool. Now, what are the tools in our toolkit? Performance Evaluation Performance evaluations encompass a broad range of evaluation methods. They often incorporate before-after comparisons but generally lack a rigorously defined counterfactual. Performance evaluations may focus on what a particular project or program has achieved (at any point during or after implementation); how it was implemented; how it was perceived and valued; and other questions that are pertinent to design, management, and operational decision-making. (defined by ADS 201) Activity Review An activity review can include both qualitative and quantitative primary data collection. This approach usually takes a hands-on form and includes aspects such as workshops, consultations, and focus group discussions (FGDs) or key informant interviews (KIIs), sometimes combined with short quantitative surveys or secondary data.Activity reviews allow for USAID staff to engage with implementers and beneficiaries in an interactive manner. Assessment An assessment is a forward-looking process that may be designed to examine country or sector context to inform strategic planning or project design, or be an informal review of a strategy project or activity. It is distinct from an evaluation. (defined by ADS 201) Monitoring Performance monitoring is the ongoing and systematic collection of performance indicator data and other quantitative or qualitative information to reveal whether implementation is on track and whether expected results are being achieved. Performance monitoring includes monitoring the quantity, quality, and timeliness of activity outputs within the control of USAID or its implementers, as well as the monitoring of project and strategic outcomes that are expected to result from the combination of these outputs and other factors. Performance monitoring continues throughout strategies, projects, and activities. (defined by ADS 201) Desk Review Desk reviews utilize existing published data, evaluations, and studies, as well as government or secondary data made available by stakeholders.The information is analyzed and synthesized to address context or theory-based research questions. Other Options While comprehensive documentation of learning tools goes beyond the scope of this report, other options of particular relevance include Impact Evaluations and Developmental Evaluations. These evaluations tend to require significantly more time and rigor and answer different questions than the tools described above. How-To: Step 2 Selecting a Learning Tool 3. Contract Requirements Sometimes, contracts require specific learning tools, research questions, or a certain level of rigor or independence. These requirements must be considered when selecting a tool. 4. Independence & Existing Skills Do you want or are you required to have an independent review or evaluation? This means that another party must conduct the research. The benefit of independence is that it allows your activity to be reviewed without your bias, yet a corresponding drawback is that learnings from conducting the evaluation itself are not internalized. You also need to assess your team's internal capacity, availability, and skills. Can staff rigorously analyze quantitative data? Is anyone trained in facilitating or developing research tools? Depending on the answer, you may need to utilize a consultant, research firm, or other technical staff. 5. Rigor What are your research questions? How will the results be used? Does the research need to be representative of a certain geographic area? The answers will determine the rigor required and lead to a certain learning tool. For example, if you want to review an activity, an activity review (focusing on interactive consultations with stakeholders) will likely not allow for information that is representative across a region; however, it can enable relationships, buy-in, and capacity building. Remember the Forest. Things are busy! We have calendar alerts, an unending flow of emails, and a growing to-do list with competing priorities. Sometimes the tree right in front of us is all we can see. But don't forget about the forest! Selecting the right learning tool from the toolkit is both a short-term (tree) and long-term (forest) decision. As you select your tool, don't forget to think about your long￾term goal and needs. The findings from your learning tool can be used to inform the entire program cycle, including the CDCS, PMP, and PADs. The right tool can be used for the CDCS, PMP, PAD, and more! How-To: Selecting a Learning Tool Step 3 The Resources. What are the inputs that you and your team have at your disposal? Do you have a limited budget but team members with rigorous statistical skills? Do you have specific contractual limits or flexibility? The answers to these questions will determine which learning tool you select from the toolkit. Some resources to consider are: 1. Cost Each tool has associated costs. For example, if you have a small budget, a desk review is likely the cheapest tool; however, you lose the ability to compare baseline and endline results. If you need to collect a lot of data, you may need to hire a data collection firm— with costs that vary depending on location and method. 2. Time Similar to costs, each tool requires a gradient of time. For a before￾after performance evaluation, you will need time before and after an activity to conduct data collection. In addition to the time of an evaluation, you also need to consider requirements for research that are country specific. This can include government clearance requirements, local-level permissions, and institutional review board (or similar research ethic association) approvals. Evaluations should also be timed around implementer performance indicator data collection, data quality assessments, and studies, if being used as secondary data. Geography and seasonal changes are also important to consider when thinking about time. Is it rainy season? Is it harvest season? Are people's incomes affected during a certain time? Will enumerators be available? Will respondents be available? The Toolkit. What are you trying to learn about: an activity, the local context, or pre-existing information on a topic? We need to establish what resources are at your disposal, based on the explanations on the preceding page. Do you have a large budget, lots of time, high level of skills and/or a high level of rigor? Do you have a flexible contract? Can you combine different tools? Activity or Project Cost Time Contract Independence Skill Rigor Resources Performance Evaluation Activity Review Depending on your resources, you may choose to conduct a baseline, midline, and endline evaluation in all locations or in a representative sample of locations. The evaluation can include qualitative tools, such as FGDs, KIIs, or observations, in addition to a rigorous quantitative survey tool. You could hire a data collection firm with trained enumerators. If you are limited in some areas, you may choose to only conduct a baseline and endline evaluation or remove the qualitative component. If independence is not required, you may choose to develop the tools internally. Perhaps you need to learn about your activity, but you are limited in terms of time or do not have the budget to hire a data collection firm. Maybe your team is interested in interactive learning, and your activity would benefit from input from many stakeholders. You can develop a learning tool in which USAID staff consult with stakeholders and other beneficiaries in a facilitated workshop or group discussion. If you have a little more time or budget, you can conduct small-scale data collection to supplement your workshops, FGDs, and consultations. Performance Evaluation o Liberia FARA Evaluation: Health Outputs o Legal Professional Development and Anti￾Corruption Midterm Evaluation o Civil Society and Media Whole-of-Project Evaluation Activity or Project Review such as Pause & Reflects and After-Action Reviews o Land Governance Support Activity Review o Learning Links Closeout Events Step 1: Step 2: Select your tool. Note that many of the tools can be combined to create the most comprehensive picture. Step 3: What are your inputs? What are you reviewing? As noted in "The Tools" section, other tools include impact evaluations and developmental evaluations. These specific tools require more time, a higher level of rigor, and an environment that enables the research. Although they require a large amount of effort, they have large payoffs. For example, an impact evaluation can determine the causal impact of an activity (rather than a correlation). This means that an impact at the end of activity can be linked directly to an activity alone, rather than other factors such as government or policy changes, environmental differences, etc. A developmental evaluation fosters iterative adaption and is especially useful for complex interventions or new innovations (e.g., human-centered design, systems change). Other Activity Evaluations How-To: Step 4 Selecting a Learning Tool Cost Time Contract Independence Skill Rigor Assessment o Electoral Security Assessment o Biodiversity Sector Assessment The Toolkit. What are you trying to learn about: an activity, the local context, or pre-existing information on a topic? We need to establish what resources are at your disposal, based on the explanations in the preceding page. Do you have a large budget, lots of time, high skills and/or a high level of rigor? Do you have a flexible contract? Can you combine different tools? Local Context Resources Assessment Step 1: Step 2: Select your tool. Note that many of the tools can be combined to create the most comprehensive picture. Step 3: What are your inputs? What are you reviewing? An assessment is used to understand a context better. This is often done prior to designing an activity or project. Assessments are especially beneficial in complex environments to ensure a human-centered design process that responds to actual needs, rather than perceived needs. Assessments can utilize both quantitative and qualitative tools including surveys, FGDs, KIIs, and observations. Depending on your resources and desired intervention, you could develop a method so that your assessment would be representative of an entire geographic region or group of people. FGDs and KIIs could be used to identify areas that may not be on USAID's radar, while quantitative data can be used to best develop the scope of future designs. Theory of Change Don't forget to test your theory of change! Assessments allow us to test our theories and, subsequently, reduce the chance of negative unintended outcomes or wasted resources. Understand Existing Information Cost Time Contract Independence Skill Rigor Resources Desk Reviews o Food Security Desk Review for Liberia Monitoring Data o Liberia Health Events GIS mapping Desk Review A desk review is a comprehensive review of available evidence. This can be as simple as a judicious internet search (just make sure your sources are credible!). Or, it can be as extensive as compiling existing literature and information from stakeholders, such as government data. A desk review is especially helpful in developing a theory of change, so that you develop an evidence-based theory and evidence-based design. Monitoring Data Whether you are interested in an ongoing activity or trying to understand a context, data likely exists. This can include implementer indicator, government, and open source data from other stakeholders. This data could be geographical information, third party monitoring, or performance indicators such as the total number of students enrolled. Alone, monitoring should be combined with literature from a desk review, assessments, or evaluations to provide the most thorough picture. From a resource standpoint, monitoring can be low cost and time; however, keep in mind how data can be accessed. Is the data highly sensitive? Is there a cost to this data? Does the data require particular software? Often, governments may require approval or payment for certain data. These uncertainties should be part of your resource calculation. How-To: Step 4 Selecting a Learning Tool Acronyms o ADS: Automated Directives System o CDCS: Country Development Cooperation Strategy o CFMB: Community Forest Management Body o CHT: County Health Team o CSM: Civil Society and Media o CSO: Civil Society Organization o DEC: Development Experience Clearinghouse o DO: Development Objective o DRG: Democracy, Rights, and Governance o EQUAL: Education Quality and Access in Liberia o FARA: Fixed Amount Reimbursement Agreement o FDA: Forestry Development Authority o FGD: Focus Group Discussion o FIFES: Forest Incomes for Environmental Sustainability o G2G: Government to Government o GIS: Geographic Information System o GOL: Government of Liberia o iHRIS: integrated Human Resource Information System o IP: Implementing Partner o IR: Intermediate Results o KII: Key Informant Interview o LADA: Liberia Agricbusiness Development Activity o LAVI: Liberia Accountability and Voice Initiative o LEGIT: Local Empowerment for Development Inclusion and Transparency Acronyms o LOE: Level of Effort o LPAC: Legal Professional Development & Anti-Corruption o LRA: Liberia Revenue Authority o LSA: Liberia Strategic Analysis o LWSC: Liberia Water and Sewer Corporation o MEL: Monitoring, Evaluation, and Learning o MOE: Ministry of Education o MOH: Ministry of Health o MOU: Memorandum of Understanding o MSTAR: Mobile Solutions Technical Assistance and Research o NGO: Non-Governmental Organization o PAC: Partnership for Advancing Community-Based Services o PAD: Project Appraisal Document o PIDS: Performance Indicator Database System o PMP: Performance Management System o PROSPER: People, Rules, and Organizing Support for the Protection of Ecosystem Resources o PTA: Parent Teacher Association o RG3: Revenue Generation for Governance and Growth o SHOPS: Smallholder Oil Palm Support o SI: Social Impact, Inc. o SOW: Statement of Work o USAID: United States Agency for International Development o USD: United States Dollar o WORTH: Women Operating Resources Together