Final MELS Report 2017-2022 AID-OAA-I-15-00022 / AID-612-TO-17-00001 Submitted: March 17, 2022 Resubmitted: March 31, 2022 International Business & Technical Consultants, Inc. 8618 Westwood Center Drive, #405 Vienna, VA 22182 USA DISCLAIMER This Final MELS Report was prepared by International Business & Technical Consultants, Inc. (IBTCI) for review by the United States Agency for International Development (USAID). The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. MONITORING, EVALUATION AND LEARNING SUPPORT PROJECT (MELS) TABLE OF CONTENTS ACRONYMS.............................................................................................................................iii 1. INTRODUCTION .................................................................................................................1 2. SUMMARY OF COMPLETED ACTIVITIES ........................................................................2 2.1. MELS Evaluations, Assessments and Studies .................................................................................2 1. SEG Meta-Analysis: 2022..............................................................................................................3 2. WALA Rapid Assessment: 2018 ....................................................................................................3 3. AgDiv Mid-Term Performance Evaluation: 2019/2020..................................................................4 4. PERFORM Final Performance Evaluation: 2019.............................................................................4 5. FISH Final Performance Evaluation: 2019/2020 ............................................................................5 6. SANE Mid-Term Performance Evaluation: 2018/2019...................................................................5 7. MISST Mid-Term Performance Evaluation: 2018 ..........................................................................6 8. PERFORM Mid-Term Performance Evaluation: 2017 ....................................................................6 9. FISH Mid-Term Performance Evaluation: 2017 .............................................................................7 10. MCHF Verification Activity: 2022................................................................................................7 11. AgDiv Verification Activities: 2017-2021.....................................................................................7 12. AgDiv Annual Surveys of Participants: 2018-2021.......................................................................9 13. AgDiv Baseline Survey and Listing Operation: 2017 ....................................................................9 2.2. MELS Capacity Building ............................................................................................................... 10 The Kauniuni Forum Formation...................................................................................................... 12 3. LESSONS LEARNED AND RECOMMENDATIONS ........................................................12 3.1. Performance Evaluations and Special Studies.............................................................................. 12 3.2. Verification Activities.................................................................................................................. 13 3.3. Annual Surveys........................................................................................................................... 14 3.4. Capacity Building ........................................................................................................................ 15 3.5. Adaptation to COVID-19.............................................................................................................. 16 4. MELS PERFORMANCE INDICATORS..............................................................................17 ANNEXES ...............................................................................................................................18 Annex 1. The MELS Personnel in 2017-2022....................................................................................... 18 Annex 2. Complete list of MELS documents uploaded to the Development Experience Clearinghouse (DEC)................................................................................................................................................. 20 iii ACRONYMS AgDiv Agriculture Diversification ASFNS Agriculture Sector Food and Nutrition Strategy ASP Area Stakeholder Panel BVC Beach Village Committee CAPI Computer-Assisted Personal Interviews CBA Cost-Benefit Analysis CDCS Country Development Cooperation Strategy CDM Center for Development Management CLA Collaborating, Learning, and Adapting COP Chief of Party COR Contract Officer’s Representative CRS Catholic Relief Services DAECC District Agricultural Extension Coordinating Committees DAESS District Agricultural Extension Services System DEC Development Experience Clearinghouse DQA Data Quality Assessment DSP District Stakeholder Panel EA Enumeration Area EPA Extension Planning Area FGD Focus Group Discussion FISH Fisheries Integration of Society and Habitats FTF Feed the Future FY Fiscal Year GAP Good Agricultural Practices GIS Geographic Information System GCC Global Climate Change GOM Government of Malawi GPS Global Positioning System GVH Group Village Headman HACCP Hazard Analysis Critical Control Point IBTCI International Business & Technical Consultants, Inc. IDIQ Indefinite Delivery Indefinite Quantity ISO International Organization for Standardization KII Key Informant Interview LQAS Lot Quality Assurance Sampling M&E Monitoring and Evaluation MCHF Modern Cooking for Healthy Forests ME&L Monitoring, Evaluation and Learning MELS Monitoring, Evaluation, Learning, and Support MISST Malawi Improved Seed Systems and Technologies MT Metric Ton MTE Mid-Term Evaluation NGO Non-Government Organization NRM Natural Resource Management OCR Organizational Capacity Review OFSP Orange Fleshed Sweet Potato PE Performance Evaluation PERFORM Protecting Ecosystems and Restoring Forests in Malawi PFM Participatory Fishery Management QC Quality Control RFP Request for Proposal SANE Strengthening Agricultural and Nutrition Extension SEG Sustainable Economic Growth SOW Scope of Work SOQ Statement of Qualifications SPSS Statistical Package for the Social Sciences STTA Short-Term Technical Assistance USAID United States Agency for International Development WALA Wellness and Agriculture for Life Advancement ZOI Zone of Influence 1 1. INTRODUCTION The United States Agency for International Development (USAID) contracted International Business & Technical Consultants Inc. (IBTCI) to provide monitoring, evaluation and learning support (MELS) to USAID/Malawi under the MELS contract (AID-612-TO-17-00001). Specifically, MELS supported the Mission’s Development Objective 2 activities authorized through the Sustainable Livelihoods Project and the Feed the Future (FTF) flagship Agricultural Diversification Project, which directly aligned and supported the Mission’s Country Development Cooperation Strategy (CDCS). The original period of performance for this task order was 5 years, February 1, 2017 through January 31, 2022. The activity received a two-month extension on December 14, 2021 and will close out on March 31, 2022. The purpose of the MELS activity was to implement performance evaluations (PE) and assessment services under the Monitoring and Evaluation (M&E) Indefinite Delivery Indefinite Quantity Contract (IDIQ). MELS provided support to the FTF and Environment (Global Climate Change (GCC) and biodiversity) activities that were managed by the USAID/Malawi’s Sustainable Economic Growth (SEG) Office. MELS activities fell under four primary objectives: ● Objective 1: Performance evaluations of FTF and environment activities and of the Sustainable Livelihoods Project designed and implemented. ● Objective 2: Assessments of FTF Malawi Agriculture Diversification (AgDiv) activity performance designed and implemented. ● Objective 3: Studies and analyses on selected topical issues developed and conducted. ● Objective 4: Local capacity to undertake evaluations and assessments strengthened. The MELS project involved a wide range of personnel that contributed to successful implementation of the activity. In total, not less than 50 evaluation specialists and subject matter experts were involved, both international and local. The core team included Jim Phillips, Project Director, Roman Pogojev, M&E Specialist, Madan Singh, Statistician, Lynne Workman, Senior PA, Mkuwanda Mtimuni, Local MEL Coordinator, Dickson Kazembe, Local M&E Specialist, Victor Mbamba, Local MELS Specialist, Chikondi Chikoya, Local Logistics Coordinator, and Thomas Carson, Quality Assurance. This report covers all activities of the MELS project, supported by the USAID/Malawi SEG Office, and implemented by IBTCI from February 1, 2017 to March 31, 2022. IBTCI's work under this contract included seven performance evaluations, two special studies, six verification activities, five baseline and annual surveys, and numerous capacity building activities. The report concludes with lessons learned and recommendations for future similar activities. 2 2. SUMMARY OF COMPLETED ACTIVITIES 2.1. MELS Evaluations, Assessments and Studies The MELS project conducted two special studies, seven performance evaluations, six verification activities, one baseline, and four annual surveys. Table 2 below provides the list of evaluations, assessments, and studies the MELS completed since 2017. Table 1. MELS Evaluations and Studies in 2017-2022 MELS EVALUATIONS, ASSESSMENTS, AND STUDIES Activity Title Timeframe Brief Description 1. Special Studies SEG Meta-Analysis Q2, FY2022 A meta-analysis of three activities (PERFORM, AgDiv, and MISST) implemented by USAID/Malawi’s SEG office. WALA Rapid Assessment Q1-Q2, FY2018 An independent assessment of the Wellness and Agriculture for Life Advancement project (WALA) implemented by Catholic Relief Services (CRS), which ended in 2014. 2. Performance Evaluations AgDiv Mid-Term Performance Evaluation Q4, FY2019- Q3, FY2020 Mid-term performance evaluation of the FTF AgDiv project in Malawi. PERFORM Final Performance Evaluation Q3-Q4, FY2019 Final performance evaluation of the Protecting Ecosystems and Restoring Forests in Malawi (PERFORM) project. FISH Final Performance Evaluation Q3, FY2019- Q1, FY2020 Final performance evaluation of the Fisheries Integration of Society and Habitats (FISH) project in Malawi. SANE Mid-Term Performance Evaluation Q4, 2018- Q1, FY2019 Mid-term performance evaluation of the FTF Strengthening Agriculture and Nutrition Extension Activity (SANE) project. MISST Mid-Term Performance Evaluation Q3-Q4, FY2018 Mid-term performance evaluation of the FTF Malawi Improved Seed Systems and Technologies (MISST) project. PERFORM Mid-Term Performance Evaluation Q3-Q4, FY2017 Mid-term performance evaluation of the PERFORM project in Malawi. FISH Mid-Term Performance Evaluation Q3-Q4, FY2017 Mid-term performance evaluation of the FISH project in Malawi. 3. Verification Activities MCHF 2021 Verification Activity Q1, FY2022 Verification of an indicator reported by the Modern Cooking for Healthy Forests (MCHF) project to USAID in FY2021: Percent change in annual conviction rate for illegal charcoal and other forestry crime activities. AgDiv 2021 Verification Q1, FY2022 Verification of nonfarm outcome indicators and output indicators reported by the FTF AgDiv project to USAID from FY2017 to FY2021. This AgDiv 2020 Verification Q1, FY2021 AgDiv 2019 Verification Q1, FY2020 3 MELS EVALUATIONS, ASSESSMENTS, AND STUDIES Activity Title Timeframe Brief Description AgDiv 2018 Verification Q1, FY2019 task included verification of calculations, source AgDiv 2017 Verification Q3, FY2018 documents, and spot checks in the field. 4. FTF AgDiv Assessments AgDiv Year 5 Annual Survey of Participants Q4, FY2021- Q1, FY2022 Annual surveys of AgDiv participants for farm-level outcome indicators in the Zone of Influence (ZOI): Balaka, Blantyre, Dedza, Lilongwe, Machinga, Mangochi, Mchinji, and Ntcheu. The annual surveys utilized a stratified two-stage clustered probability sampling design. Data in the field was collected by a local data collection firm – the Center for Development Management (CDM) applying tablet assisted personal interviews. AgDiv Year 4 Annual Survey of Participants Q4, FY2020- Q1, FY2021 AgDiv Year 3 Annual Survey of Participants Q4, FY2019- Q1, FY2020 AgDiv Year 2 Annual Survey of Participants Q4, FY2018- Q1, FY2019 AgDiv Baseline Survey and Listing Operation Q3, FY2017- Q1, FY2018 A baseline survey for farm-level outcome indicators for the AgDiv project in Malawi. The survey began with a listing operation to list all farmers who grew the targeted value chain crops in the AgDiv ZOI. The MELS team brought the evaluation and assessment reports into compliance with the 508 requirements and submitted it to the USAID Development Experience Clearinghouse (DEC). 1. SEG Meta-Analysis: 2022 The MELS team completed a meta-analysis of three SEG Office activities (PERFORM, AgDiv, and MISST). The primary purpose of this meta-analysis was to assess the efficacy and scalability of some selected key strategies and approaches that were employed in the implementation of activities managed by the SEG Office to deliver interventions towards the CDCS 2013-2019. The evaluation team completed all fieldwork and submitted a final meta-analysis report to USAID in Q2 FY2022. The MELS team applied the following methods in this assessment: a desk study, data mining, and key informant interviews (KIIs) with key stakeholders. The findings were validated with representatives of the three assessed projects as well as with the USAID SEG office. 2. WALA Rapid Assessment: 2018 MELS conducted an independent assessment of the Wellness and Agriculture for Life Advancement project implemented by CRS, which ended in 2014. The assessment aimed to understand the resilience of the communities who benefitted from the watershed development activities of this project. The Lingoni community in Machinga district, which participated in WALA, appeared to be an extremely successful community, and USAID was interested in knowing how other WALA beneficiary communities were doing with respect to resilience. The assessment was conducted in Q1-Q2 FY18 by a two-person team during a very intensive 2-week period of field work. The assessment team visited 24 of the WALA watershed communities including Lingoni and found that some of the communities were very resilient and others were not. Overall, there appeared to be a correlation between good community leadership and community resilience. Of the 24 sites visited only five required food aid during the 2015/16 El Nino event. The findings were well received by both USAID and CRS and resulted in additional presentations by the team leader to communicate learnings from this assessment. The report and analysis were filed and uploaded to the DEC. 4 3. AgDiv Mid-Term Performance Evaluation: 2019/2020 The purpose of this mid-term evaluation (MTE) was three-fold: 1) to review, analyze, and evaluate the effectiveness of the AgDiv activity in achieving objectives; 2) to identify major constraints in achieving project results; and 3) to provide specific recommendations about strategies and approaches that USAID/Malawi should pursue in its future economic growth activities. Within that framework, USAID/Malawi was also interested in addressing five specific questions related to tobacco vs non￾tobacco farmers: 1) to what extent have farmers diversified their production as a result of the project?; 2) to what extent has AgDiv strengthened the capacity of the private sector?; 3) to what extent are farmers better connected to markets?; 4) to what extent is the adoption of new technologies by smallholder farmers growing in a sustainable manner?; and 5) what elements of AgDiv should be continued in future private sector-oriented agricultural development? MELS deployed a team of two international experts and two local consultants, who spent five weeks in the field visiting many areas of the project. The team convened thirty focus group discussions (FGDs). The total number of participants in focus groups was 330, of whom 190 were women, 58 percent. The team also carried out twenty-nine key informant interviews (KIIs). USAID’s AgDiv activity has demonstrated a number of strengths: • Addressing Malawi’s national deficit of quality seed by helping farmers move away from reliance on tobacco; • Addressing significant limitations in agricultural financing at the second tier and at the village level; strengthening agricultural input markets by working with private sector multi-nationals; • Successfully addressing the long-overlooked issue of post-harvest food losses; • Introducing and scaling up inoculant use, drip irrigation, and solar pumping; attempting to address backyard garden production of nutritious crops and introducing innovative household soy processing equipment; • Addressing women’s empowerment, financial literacy, and economic decision making; and experimenting with soil conservation activities. 4. PERFORM Final Performance Evaluation: 2019 The purpose of the final performance evaluation of the Protecting Ecosystems and Restoring Forests in Malawi project was to assess the effectiveness of the PERFORM design and implementation processes, and to inform management regarding effective implementation strategies that support reducing emissions from deforestation and forest degradation and effective natural resource management in Malawi. The PERFORM evaluation team used a mixed-methods approach to generate both qualitative and quantitative primary data. In addition to a desk review of PERFORM documents and data, the evaluation team used 18 KIIs in Lilongwe and the three focal districts of Mzimba (north), Ntchisi (central), and Machinga (south) to target key stakeholders. In order to capture the experiences of community members and the performance of the PERFORM activities implemented, 194 project participants were engaged through nine FGDs representing nine different groups of villages. An additional 36 individuals were randomly sampled for one-on-one interviews as part of a mini survey that generated quantitative findings. The evaluation team also conducted a gender assessment of PERFORM activities and applied a gender lens to each of the final performance evaluation question in order to determine any differential impacts of the project on women. The evaluation team also collected age data in order to determine the extent of youth engagement (10-35 years old) in PERFORM. 5 The PERFORM final performance evaluation team found evidence of some progress toward key outcomes and identified strategic options for future USAID/Malawi programming. 5. FISH Final Performance Evaluation: 2019/2020 The purpose of the final performance evaluation of the Fisheries Integration of Society and Habitats project in Malawi was to explore how the project had succeeded in addressing the primary objectives of FISH, which were to increase resilience to climate change and to improve biodiversity conservation through effective sustainable fisheries co-management. This FISH final evaluation covered the full period of implementation of the project. The evaluation used a participatory mixed-method approach using both quantitative and qualitative techniques in gathering reliable primary and secondary data and valid evidence of project outcomes and impacts. The team was tasked to provide detailed answers to five questions and applied participatory methodology to enable all stakeholders to take active roles in generating knowledge to test the fundamental assumptions underlying project design, implementation, and capture lessons learned from FISH. The evaluation assessed the impacts of FISH’s approaches under each of its four components in each of the four key ecological freshwater lake ecosystems over four weeks in Malawi and interviewed a majority of the key organizations and individuals involved in the project. 6. SANE Mid-Term Performance Evaluation: 2018/2019 The purpose of the mid-term performance evaluation of the FTF Strengthening Agriculture and Nutrition Extension Activity project was to determine what was working and what was not working and to provide recommendations to the SANE team for mid-course corrections. USAID intended SANE to strengthen agricultural and nutrition extension services through policy work, capacity development, extension service provider coordination, and improved research extension linkages. USAID commissioned this mid-term evaluation to determine: • SANE "learnings” were being applied outside the FTF Extended ZOI; • If SANE was achieving its policy, capacity development and coordination strengthening, and research-extension linkage objectives; • What implementation challenges the project faced, and how they were addressed; • If project outputs and outcomes were sustainable; and, • How to ensure greater adoption and scaling up in non-ZOI areas of SANE promoted practices. The MELS team conducted this evaluation using three methods, which included document review, KIIs, and FGDs. The evaluation results show that SANE has made good achievements regarding strengthening District Agricultural Extension Services System (DAESS) as a system. SANE piloted an effective model for DAESS upper echelon development (i.e., the District Agricultural Extension Coordinating Committees (DAECCs) and Area Stakeholder Panels (ASPs), and less so for the District Stakeholder Panels (DSPs). In pursuing output/outcome sustainability, SANE supported NAEASS and Agriculture Sector Food and Nutrition Strategy (ASFNS) development. SANE integration with other USAID activities, especially the MISST activity, improved resource use efficiency, but likely had little impact on long term sustainability. 6 7. MISST Mid-Term Performance Evaluation: 2018 The purpose of the mid-term performance evaluation of the FTF Malawi Improved Seed Systems and Technologies project was to assess to what extent the MISST project was on track, in terms of progress and outcomes; to what extent has MISST improved functionality of input and output markets of its focus value chains; what challenges MISST faced in implementing its activities and how has MISST responded to those challenges; to what extent would the outcome of MISST interventions likely result in improved seed production, quality and productivity; and how MISST integrated and worked with other partners to achieve its results. The MELS deployed a team of two expats and two local experts to evaluate the project. Based on the evaluation findings, the MISST project has made the following progress against key indicators for each of its strategic objectives: • In order to improve functionality of input chains of selected value chains for seed, MISST has facilitated the production of 17,500 MT of cereal and legume seed and over 200,000 Orange Fleshed Sweet Potato (OFSP) bundles in total, and has during the last two years provided development assistance to 1,512 and 1,990 organizations, respectively (averaging 111 percent of the target). • In relation to capacity building, the number of stakeholders directly benefitting from MISST interventions has exceeded the target in each of the last three years (120 percent of the target). • Regarding adoption and uptake, new crops, varieties, and technologies have been tried by 140,000 farmers (on target) and the number of organizations applying improved organization￾level technologies or management practices was 1,741, or 133 percent of the target. 8. PERFORM Mid-Term Performance Evaluation: 2017 The primary purpose of the PERFORM mid-term performance evaluation was to assess the effectiveness of this USAID funded activity, considering both its design and implementation processes. The evaluation also sought to inform management of course corrections, if needed. Additionally, through this evaluation, USAID sought strategic and programmatic options for future engagement to support climate change mitigation and adaptation in Malawi. Consistent with USAID’s Evaluation Policy, the primary goal of this mid-term evaluation was to provide evidence so that USAID/Malawi could determine whether PERFORM was on track to achieve its desired results or not. PERFORM has built upon prior efforts and has moved REDD+ readiness in Malawi forward through support for local scale forest inventories and carbon stock assessments, capacity building, operationalizing the MRPAP, and improving policy and legal frameworks. The Government of Malawi (GOM) has been regarded as slow to develop what they referred to as the REDD+ action plan (REDD strategy). PERFORM was working to accelerate the timeline for the development of the National REDD+ Strategy. Creating an opportunity to "affect positive transformational change in the forestry and natural resource management sector" would require a greater commitment from the GOM in significantly advancing its preparatory work. The main (and important) contribution of the PERFORM’s agricultural interventions have been to increase food security, which helped to reduce the negative effects from 2015 El Niño drought. There was insufficient evidence to show that the income obtained from these activities could replace the potential income from illegal charcoal sales. The introduction of cookstoves has been greatly successful, with significant penetration and replication potential, reducing the need for fuel wood, with implications for decreased pressure on surrounding forests. 7 9. FISH Mid-Term Performance Evaluation: 2017 FISH completed approximately 2.5 years of its period of performance; therefore, USAID wished to evaluate the performance of the project at the mid-point of its five-year program. Consistent with USAID’s 2011 Evaluation Policy (and its more recent updates), the primary goal of this mid-term evaluation was to provide evidence so that USAID/Malawi could determine whether FISH was on track to achieve its desired results or not. Other goals of the evaluation were to inform what course corrections, if any, were needed, identifying specific project interventions that could be scaled up further and to determine best ways to ensure sustainability of the activities, institutions and capacities promoted by the activity. The FISH project made significant progress in the first 2.5 years. The PACT consortium of NGO partners worked well across four lake communities to implement projects related to fishery and watershed management and adaptation to climate change. FISH worked to implement Participatory Fishery Management (PFM), or co-management, by Beach Village Committees (BVCs) to monitor fishing activities in local communities. These efforts were off to a good start and BVCs have developed constitutions and by-laws to manage fishery and coastal resources. FISH conducted an effective governance program that informed and influenced important stakeholder groups. By integrating climate change into the program design, FISH added agriculture and forestry activities to those for fishery management. In this context, FISH represented more of a coastal zone management program rather than a strictly fishery management project. In a larger context, FISH represented an experiment in fishery co-management for freshwater African lakes. 10. MCHF Verification Activity: 2022 The Modern Cooking for Healthy Forests Verification Activity was completed in Q2 FY22. The MCHF verification report was reviewed and confirmed by the MCHF team including its Chief of Party (COP) Ramzy Kanaan. Overall, despite some revealed inconsistencies, taking into account both overreported and underreported locations, and applying the Lot Quality Assurance Sampling (LQAS) technique, the MELS team found the indicator values reported by MCHF to USAID for FY21 to be acceptable. 11. AgDiv Verification Activities: 2017-2021 In addition to the farm-level outcome indicators, which were collected via annual surveys of participants, the AgDiv project also reported on nonfarm outcome indicators and output indicators. The MELS team conducted annual verification exercises on these indicators from 2017 to 2021. The purpose of the verification activities was to verify the numbers that AgDiv reported to USAID for all output and all non-farm outcome indicators. Table 2. AgDiv Verification Activities in 2017-2021 2017 2018 2019 2020 2021 Total number of indicators assessed 11 11 12 12 12 Number of indicators accepted 6 8 11 11 12 8 The MELS team had some difficulties with the verification of indicators during the first two years (FY17, FY18). There were some data quality issues that occurred primarily because AgDiv partners lacked well established MEL systems. Key issues included a lack of data, untimely data submission from AgDiv’s partners, lack of source documents, and lack of electronic records. The lack of electronic records required that the verification team utilize hard copies of source documents to verify the numbers for indicators reported by AgDiv. For example, the training indicator required manual counting of signatures from source documents. However, for the training indicator these source documents were largely missing. AgDiv was well aware of these issues and worked closely with the MELS verification team to assist in locating farmers visited during the verification process/spot checks. AgDiv took corrective measures and substantially improved data quality in FY19 compared to the previous two reporting years, and the quality of data remained high in FY20 and FY21. The MELS team did not have any major difficulties with the verification process. Partner record keeping improved and almost all data were digitized and available in an electronic database. Besides some issues with the data quality during the first couple of years, two indicators created additional difficulties with the verification process almost every year: 1. The MELS team had issues with the following indicator and did not find sufficient evidence to approve it: Number of agro-processors, cooperatives, care groups, and farmers' associations that practice safe food processing for the targeted value chains (indicator #3.1-1). There was sufficient evidence to confirm that all reported organizations participated in safe food processing training organized by AgDiv and its partners. The training included safety measures that agro￾processing firms should apply in practice including washing hands before and after using soy kits, cleaning all utensils regularly, wearing protective gear (gloves, head caps) and some other general cleanliness principles. However, participation in training is not the same as application of safe food processing in practice. There was insufficient evidence to confirm that it was, indeed, applied. Moreover, observations made by the evaluation team revealed that the soya milk producers did not follow even simple hygiene rules, such as wearing masks to cover the nose and the mouth, wearing gloves and head caps. During the fifth year (FY21) AgDiv addressed this issue and changed this indicator to the Number of agro-processors, cooperatives, care groups, and farmers' associations that received training on safe food processing for the targeted value chains. The MELS team could verify and confirm this indicator in FY21 after the change was introduced. 2. The MELS team had some difficulties with verification of sensitive data, such as investments, loans, and sales. Often, the only source of verification for the sensitive data was email communication. The MELS team recommended to find ways of securing necessary documents for such data i.e., if copies of official records were not available (even under confidentiality agreements), it could have been formal letters from recipients summarizing received technical and financial assistance. This issue has not been addressed yet, but USAID allowed to apply relaxed criteria to verification of this indicator and to accept email documentation as proof for loans and private sector investments. Throughout the verification process, the AgDiv team was always cooperative and transparent and provided the MELS team with all necessary assistance. 9 12. AgDiv Annual Surveys of Participants: 2018-2021 The MELS team completed surveys for the farm level outcome indicators for the AgDiv project once per year: from 2018 to 2021. A baseline survey was conducted in year 1 (FY17), and four follow up annual surveys of AgDiv participants were conducted in year 2 (FY18), year 3 (FY19), year 4 (FY20), and year 5 (FY21). IBTCI selected CDM to conduct computer assisted personal interviews utilizing tablets with AgDiv farmers. Under supervision of the MELS team, CDM recruited all necessary data collectors and supervisors, piloted the survey toolkits, completed thorough training of the field teams, and conducted the surveys. The training sessions included pilots in the field (with participation of all enumerators), and follow up review sessions before the field work commenced. Complete lists of AgDiv participants in the ZOI (Balaka, Blantyre, Dedza, Lilongwe, Machinga, Mangochi, Mchinji, and Ntcheu) served as sample frames every year. AgDiv provided updated lists of participants for each year that contained all information necessary for sampling. Table 3. Annual Survey of AgDiv Participants in 2018-2021 2018 2019 2020 2021 Total number of AgDiv participants 56,665 132,091 148,698 160,668 Annual survey sample size 1,200 1,200 2,000 2,000 Total number of indicators assessed 11 12 12 12 Every year, the base sample size was calculated using four topline indicators; then, adjustments were made to the base sample size to reflect the design effect and anticipated non-response rate. The MELS team applied the USAID Sampling Guide for Beneficiary-Based Surveys 1 to calculate the sample size and sampled 1,200 participants in 2018 and 2019. In 2020, the MELS team increased the sample size to 2,000 participants to allow for stronger power calculations when indicators were heavily disaggregated. The annual surveys utilized a stratified two-stage clustered sampling design, with the first stage sampling being the group village headman (GVH) from the Districts and Extension Planning Areas (EPA) where AgDiv partners were operating. The second stage randomly sampled the beneficiary farmers from these GVHs. For each district the team used a proportion to size sampling methodology. The annual survey field works were usually conducted during Q4, and the survey findings were available in Q1 of the following fiscal year. 13. AgDiv Baseline Survey and Listing Operation: 2017 The AgDiv baseline survey was not a typical baseline survey as it began with a listing operation to list all farmers who grew the targeted value chain crops in the AgDiv zone of influence. In theory, these were target beneficiaries. The purpose of this listing operation was to establish a sample frame from which to draw a sample for the baseline survey. In order to carry out this listing operation MELS obtained the focus EPAs from the AgDiv project. These were the EPAs where Ag Div was going to be operating. Once the EPAs were provided, USAID matched these EPAs with the Enumeration Areas (EA) including maps from the previous household census carried out in 2008. Once MELS obtained this list of EAs, the 1 USAID Sampling Guide for Beneficiary-Based Surveys for Select Feed the Future Agricultural Indicators, USAID, 2016 10 team selected a sample and used maps to develop a farmer list for the sample frame for the baseline survey. Farmers in these target EAs who grew the targeted value chain crops of soybean, groundnuts, or OFSP made up this list. The farmer listing took about 10 days with an additional two days to clean up the maps and fill gaps, and listed a total of 22,660 farmers, of which 12,692 grew one of the targeted crops. The team chose the final list of eligible farmers included in the second stage sampling frame from the list of farmers who grew one of the targeted crops and lived in a residence. This represented about 56 percent of the total listed farmers. The baseline survey utilized a stratified two-stage clustered sampling design, the first stage sampling the EAs to be used for the listing operation. The second stage randomly sampled the eligible farmers from these EAs, with approximately 30 farmers in 60 targeted EAs, resulting in a total sample of 1,794 farmers selected from the planned sample. 2.2. MELS Capacity Building Since 2017, the MELS team conducted 20 capacity building activities that included various workshops and webinars. It also conducted a capacity needs assessment of MEL firms in Malawi in 2018 prior to the 2018 workshop, and formed the Kauniuni Forum in 2019 during the 2019 workshop, just before the COVID-19 broke out. In total, the MELS training reached out to over 100 participants in Malawi. Table 4. MELS Capacity Building Activities in 2017-2022 MELS CAPACITY BUILDING Activity Title Timeframe Brief Description 1. Special Activities Formation of the Kauniuni Forum Q3, FY2019 The MELS team conducted a three-day workshop for 40 participants in June 2019 that resulted in formation of the Kauniuni Forum, a community of MEL practitioners. The Forum aimed at enhancing learning between MEL practitioners in Malawi. The MELS team strongly supported the Forum with various workshops and webinars and additional materials and manuals devoted to MEL. A capacity development needs assessment of MEL firms in Malawi Q1-Q2, FY2018 The capacity development needs assessment of MEL firms in Malawi was done through self-assessment. The MELS team designed a special capacity assessment tool, conducted the assessment, and developed a capacity building plan based on the assessment findings. 2. Webinars Cost-Benefit Analysis Part 1 & Part II Q4, FY2021 The workshop took place in two sessions, the first session was an introduction to cost benefit analysis and the methodology utilized for calculation. The second session was a problem-solving activity and practical exercise that utilized the methodology. Qualitative Methods Q2, FY2021 Participants of this webinar learned the purpose of qualitative methods, its importance to the M&E field, different methodological approaches, and various qualitative methods software and their functions. 11 MELS CAPACITY BUILDING Activity Title Timeframe Brief Description Data Quality Assessments (DQAs) Q1, FY2021 This webinar explained the purpose of DQAs and their importance to the M&E field. The instructor also provided a DQA checklist and examples. Sampling methods Q4, FY2020 A webinar for survey specialists, enumerators, and supervisors on sampling techniques and approaches. Data Visualization Part 1, Part II & Part III Q3, FY2020 The data visualization capacity building activity consisted of three consecutive webinars devoted to data visualization with practical applications using MS Excel and Power BI. GIS Mapping Q1, FY2020 A webinar with specific examples using GPS and GIS showcased a success story from a USAID partner. 3. In-Person Workshops Fundamentals of MEL Q3, FY2019 MEL principles; a MEL plan; examples of good MEL practices; implementation of robust MEL plans; data quality assurance; DQAs; Theory of Change – the logic underlying the activity; results frameworks; monitoring indicators; data management; and evaluations. Performance Evaluations Q3, FY2019 Performance evaluations; USAID requirements for performance evaluations; examples of past evaluations. Qualitative Methods Q3, FY2019 Basics of qualitative methods; FGDs; and Outcome Harvesting. Cost-Benefit Analysis Q3, FY2019 Cost Benefit Analysis approaches and calculations; analytical thinking; CBA examples. Impact Evaluations Q3, FY2019 Randomized Control Trials; Difference-in-difference, Regression Discontinuity; Propensity Score Matching. GIS, GPS, and Mapping Q3, FY2019 GIS/GPS data collection and mapping; practical applications of GIS/GPS Analytics Frameworks Q3, FY2019 Analytics Frameworks; USAID requirements for this. CLA fundamentals Q3, FY2018 This session focused on Collaborating, Learning, and Adapting (CLA) and the importance of learnings with monitoring and evaluation. Impact Evaluations Q3, FY2018 This session introduced impact evaluations and a discussion on the difference between experimental design and quasi-experimental design studies. Data Security Guidance Q3, FY2018 Protecting beneficiary data – this session looked at USAID’s open data policy and how it could be followed while protecting important beneficiary information. ADS Updates Q4, FY2018 This workshop covered updates to the ADS, the USAID Monitoring and Evaluation policies, overview of new terminology, sharing of resources, and question and answers. 12 The Kauniuni Forum Formation One of the objectives of the capacity building activity organized in Q3 FY2019 was to bring together Malawi M&E practitioners and create an M&E community of practice. The primary goal of this objective was to increase the pool of firms providing MELS services to USAID and other donors in Malawi. During the closing activity, Maciej Chmielewski did a thorough walkthrough of some of the most popular learning products that USAID has been using in its own activities. The prompt was for each of the small groups to take ownership of a learning product, develop it by keeping the community of practice (COP) in mind and produce elements that would be needed to promote the COP. Through the activity, workshop participants were able to develop a COP promotional video, an infographic of the current COP’s membership statistics, mission statement of the COP, about text for the COP, and a set of learning questions to be used to drive the COP agenda. The infographic team was responsible for analyzing the pre-workshop survey results and for coming up with an attractive an eye-catching infographic that communicated the demographics of the workshop participants. The Kauniuni Forum united 40 local MEL practitioners who participated in the initial training; later, additional practitioners were invited to join. From 2019 to 2021, The MELS delivered nine additional webinars to the Forum participants. The team could not conduct additional in-person training because of COVID-19 that broke out in 2020. This slowed down the Kauniuni Forum development a bit, but the MELS team managed to keep core participants and is ready to jump start the Forum again with new initiatives and training and attract more people to it. 3. LESSONS LEARNED AND RECOMMENDATIONS 3.1. Performance Evaluations and Special Studies During its period of performance, the MELS team conducted seven performance evaluations and two special studies. Here are the main lessons learned: 1. Objectivity and triangulation: evaluations should be objective and independent; evaluation designs should incorporate approaches that carefully integrate a range of contrasting primary and secondary data sources, assorted methods of data collection (interviews, surveys, FGDs, KIIs, OCRs, observations, etc.) and assorted analytical techniques. Data collected should be triangulated against one another. 2. Team composition: evaluation teams should consist of both expat and local consultants and subject matter experts. Expat specialists usually bring different perspective to evaluations, and local consultants contribute with the local knowledge, networks, and language skills. 3. Project management: useful tools include, but are not limited to, more frequent check-ins with evaluation teams, validation of findings with key stakeholders, and thorough reviews of final deliverables (quality assurance) before they are submitted to the client. Drafts should be prepared as early as possible to allow for additional revisions after the initial review process. 13 4. Special studies: MELS has conducted only two special studies during the past five years. This includes an independent assessment of the WALA project aimed at understanding the resilience of the communities who benefitted from the watershed development activities, and a meta￾analysis of three SEG Office activities (PERFORM, AgDiv, and MISST) to assess the efficacy and scalability of selected strategies and approaches. Both studies were very useful. The IBTCI team believes that USAID could benefit from additional special studies that utilize in-depth qualitative methods and approaches, such as gender studies, assessments directed to analysis of specific improved technologies and management practices, and ad hoc studies aimed at revealing specific issues (i.e., for instance, why OFSP yield was very high in 2021, or why women’s dietary diversity dropped down in 2020). AgDiv expressed high interest in such studies as well. 3.2. Verification Activities During its period of performance, the MELS team conducted five verifications of AgDiv indicators and one verification of the MCHF project indicator. Here are the main lessons learned: 1. LQAS Methodology: the MELS team adapted techniques from the Lot Quality Assurance Sampling methods to do the verification activities. This approach was extremely useful and cost￾effective and became a backbone of the verification process. Various sources tested LQAS for small sample sizes (i.e., N = 19) and found it to be a useful tool for estimating binomial classification based on a binary indicator between acceptable and unacceptable. With a 95 percent confidence interval and a small sample size of 19 the classifications determined that four or more participants not receiving a service were deemed unacceptable (so acceptable was slightly over 84 percent, or at least 15 that received a service). The team applied 84 percent to the verifications, i.e., the data was acceptable if the source documents existed for 84 percent or more of the data. The approach included three stages: (1) checking calculations from the digital datasets and checking that data combined from different sources is free of errors, (2) looking at the hard copy source documents and comparing it with the digital data to make sure that the data integrity remains, and (3) conducting data quality spot checks in the field using LQAS sampling methodology. IBTCI found this process quite robust. 2. Access to sensitive data: the team experienced some difficulties with getting access to such data as loans and private sector investments. The team needed such information to verify corresponding indicators. Often, the only source of verification due to privacy issues for sensitive data was email communication. MELS recommended AgDiv to find ways of securing necessary documentation for such data, i.e., when copies of official records were not available (even under confidentiality agreements), AgDiv could collect formal letters from recipients summarizing received technical and financial assistance. This issue has not been addressed yet, but USAID allowed to apply relaxed criteria to verification of this indicator and to accept email documentation as proof for loans and private sector investments. There should be a way to better document such information. 3. Safe Food Processing Indicator. The MELS team had issues with the following indicator and did not find sufficient evidence to approve it: Number of agro-processors, cooperatives, care groups, and farmers' associations that practice safe food processing for the targeted value chains. There was sufficient evidence to confirm that all reported organizations participated in safe food processing training organized by AgDiv and its partners. The training included safety measures that agro-processing firms should apply in practice including washing hands before and 14 after using soy kits, cleaning all utensils regularly, wearing protective gear, and some other general cleanliness principles. However, participation in a training is not the same as application of safe food processing in practice. There was insufficient evidence to confirm that it was, indeed, applied. If USAID wants to track this indicator, it should be thoroughly described in a performance indicator reference sheet, and a definition of safe food processing should be provided with a check list of criteria used for evaluation. If the preference is to apply one of the international safe food processing standards, such as HACCP, ISO, or Global GAP, evaluation of this indicator will require hiring an additional quality standards specialist, who will perform this task. 4. Team composition: for verification activities, it is strongly recommended to have a local team knowledgeable in the subject matter and geography of the verified project that can contribute with the local knowledge, networks, and language skills. MELS had a knowledgeable experienced team in the ground for the whole duration of the MELS activity that significantly contributed to the success of the verification activities. 5. DQAs: the MELS team did not conduct any DQAs during the period of the project performance. It is strongly recommended to add this instrument to the pool of MEL evaluation instruments used by USAID in Malawi. 6. Qualitative Information – during the past two reporting periods (FY20, FY21) the MELS team started collecting anecdotal evidence from farmers to support main findings from the verifications and surveys. Such information was collected with minimal additional effort, because the team was already in the field collecting or verifying indicator data. At the same time, the anecdotes helped to explain some quantitative data and illustrate findings with field stories. Going forward, this approach is strongly recommended. 3.3. Annual Surveys Overall, the MELS team has conducted one baseline and four annual surveys. Here are the main lessons learned: 1. Adjustment of the Sample Size: The MELS team sampled 1,200 AgDiv participants for the annual surveys in 2018 and 2019 using the USAID Sampling Guide for Beneficiary-Based Surveys to calculate the sample. In 2020, the MELS team increased the sample size up to 2,000 participants to allow for stronger power calculations when indicators were heavily disaggregated. IBTCI strongly recommends keeping a sample size of at least 2,000 participants in the future annual surveys. 2. Physical Measurements of Land Plots: In order to compensate for potential farmer recall bias, the survey team physically measured a small sample of land plots annually and compared it to the size the farmers gave using their recall. The results found that larger farmers tended to underestimate the size of their plots and smaller farmers tended to overestimate their plots. Therefore, the team calculated and utilized a regression coefficient for each type of farmer (soy, groundnut, and OFSP) to adjust the plot sizes that were obtained via farmer recall. This approach proved to be very useful in mitigating farmer recall bias. 3. Utilization of SPSS Complex Samples Module: In order to get more rigorous statistical analysis, the team switched from standard SPSS to SPSS Complex Samples Module. Standard 15 SPSS software was more appropriate for analysis of data based on single-stage sampling that did not include clusters and stratification. IBM SPSS Complex Samples software makes computations for samples with complex designs allowing subpopulation assessments to consider other subpopulations. Overall, it increased precision of calculations and improved data processing and analysis. 4. GIS/GPS: The team applied several GIS/GPS instruments in the survey that were quite useful: (a) CDM used GIS/GPS during the baseline survey listing operation; the GPS capability was necessary to locate households in the ZOI and establish a reasonable baseline; (b) the team utilized an area calculator that worked with GPS when the perimeter of a plot was paced by enumerators; (c) GPS was available on all tablets during the survey and exact locations of completed interviews could be traced for mapping and quality control. 5. Team composition: A strong team consisting of expat and local experts was one of the key success factors in the annual survey. IBTCI team had a lot of experience with FTF indicators and design of relevant survey instruments. Three local consultants in Malawi were knowledgeable and delivered high-quality services. All key MELS staff members remained unchanged since the project launch – it led to constant improvements and consistently high quality of deliverables. It should be noted that the local data collection firm, CDM, also conducted the survey since 2017, and its core team members, including most enumerators, remained unchanged. 6. Enhanced Quality Control (QC): MELS had several layers of the quality control – CDM provided a first layer, local IBTCI consultants – a second layer, and IBTCI home office staff – a third QC layer. All surveys started from training, piloting of instruments, and a follow up review session with the enumerators to enhance learning. Constant capacity building and quality control were important for the surveys’ success. 7. Instructions and Manuals. The MELS team designed a wide range of instructions and manuals for the field team that included at least 11 documents: assessment plan, data treatment and analysis plan, supervisor manual, enumerator manual, survey protocol, survey sampling plan, survey questionnaire, survey questionnaire programming guide, cluster allocation procedures, translation protocol, and data cleaning plan. Although development of all mentioned instructions required large time investments during the first year, it paid off in the consecutive years, when all surveys were implemented in accordance with the high standards. Anne Swindale, a senior USAID staff member and survey expert, reviewed all instructions, protocols, and scripts and approved them. Having a good solid USAID analyst upfront was very useful and helpful. 8. Computer Assisted Personal Interviews: The original plan was to implement paper-based surveys, but the MELS team decided to use tablets instead to reduce non-sampling errors. It proved to be the right decision. Although it required additional time for programming, the team managed to control non-sampling errors well through data entry and range checks. 3.4. Capacity Building Since 2017, the MELS team conducted 20 workshops and webinars and provided training to over 100 local participants in Malawi. Here are the main lessons learned: 1. Capacity Development Needs Assessment: The MELS team revealed capacity development needs of local MEL firms before proposing and conducting relevant training to the 16 participants. It helped to ensure that the training addressed needs of local participants and was aligned with their proficiency level and needs. 2. Formation of the Kauniuni Forum: The Forum brought together Malawi MEL practitioners and provided them with an opportunity to take part in the training offered by the MELS project. First, it united 40 participants, and later the MELS team expanded its number inviting new participants. Involvement in a community of practice provided additional benefits compared to a one-off training: (a) involvement in several consecutive workshops that enhanced each other; (b) an opportunity to discuss MEL issues with other practitioners – both expats and Malawi-based; (c) access to additional MEL-related materials and documents; and (d) networking opportunities for joint projects. 3. Shifting from in-person events to webinars – The MELS team started capacity building of local MEL practitioners from a series of in-person events: a one-day event in Malawi was followed by a large three-day training event. However, amid the COVID-19 pandemic, the team had to switch to webinars. Webinars proved to be a valuable modality: a lot of content could be delivered to participants in this manner; more top-notch experts were available to deliver short￾term training, and it was cost-efficient for the MELS project. The MELS team conducted a total of 4 webinars in FY21 and 5 webinars in FY20. The participants highly appraised provided training and found it practical and useful. 4. Learning events – Starting from FY20, the MELS team delivered several presentations that involved various stakeholders presenting findings from the annual surveys, verification activities, and special studies. Participants of the learning events found it insightful and useful. Since this activity supports USAID’ CLA agenda, it should be continued and expanded. 3.5. Adaptation to COVID-19 FY2020-2022 turned out to be challenging years as the coronavirus spread across the world, including Malawi and the United States. The MELS team was able to develop mitigation strategies to continue with the field work for the annual outcome indicator surveys for AgDiv, which included using personal protective equipment and social distancing for conducting the annual surveys. MELS also devised a similar strategy for conducting the annual verification activities using remote tools such as Google Meet, ZOOM, and Skype. The MELS team was able to complete the survey, the verification activities, and the meta-analysis with minimal disruption. Moving the capacity building to a series of webinars was a good solution given that in person meetings could not take place. The MELS team conducted a total of four webinars in FY21 and five webinars in FY20. Overall, the adaptive approaches highlighted above allowed MELS to continue with its activities during this worldwide pandemic without major interruptions and delays. 17 4. MELS PERFORMANCE INDICATORS The following table summarizes the MELS project achievements from 2017 to 2022. Overall, MELS completed 100 percent of its scheduled activities and provided all deliverables on schedule. Performance Indicator Realized as of March 2022 Expected Variance (%) Objective 1: Performance Evaluations Designed and Implemented IR 1.1: Performance Evaluations Designed and Implemented Number of performance evaluations designed and successfully implemented 7 7 100 Objective 2: Assessments of Feed the Future Malawi Ag Diversification Activity Performance Designed and Implemented IR 2.1: Methods for Baseline and Annual Performance Assessments (BPAP) of Key Performance Indicators for FtF Ag. Div Project Determined Number of baseline and annual surveys conducted and approved 5 5 100 IR 2.2 Baseline and annual Beneficiary Based Surveys for FtF Farm Level Outcome Indicators Implemented The number of verification activities designed, conducted and approved 5 5 100 Objective II1: Local Capacity to Undertake Evaluations and Assessments Strengthened IR 4.1: Capacity of Local Institutions/firms to undertake MEL related services strengthened Number of workshops and webinars conducted and successfully implemented 15 15 100 18 ANNEXES Annex 1. The MELS Personnel in 2017-2022 MELS PERSONNEL IN 2017-2022 1. Core team Jim Phillips Project Director, Sr. MEL Specialist Thomas Carson Quality Assurance, Sr. MEL Specialist Roman Pogojev M&E Specialist Mamadou Sidibe M&E Specialist Lynne Workman Senior PA Madan Singh Statistician, Analyst Abigail Price Senior PA Kathryn Hornbeck Senior PA Nicholas Sprout Senior PA Merrill Jordan Senior PA Mkuwanda Mtimuni Local MELS Coordinator Dickson Kazembe Local M&E Specialist Victor Mbamba Local MELS Enumerator Chikondi Chikoya Logistics Coordinator 2. Consultants (STTAs) Assa Maganga Data Management and Economics Specialist Austin Tibu Local Natural Resource Management/Livelihoods and M&E Specialist Beatrice Mtinumi Local Evaluation Specialist Blessings Manda Local Notetaker, Evaluation Specialist Bright Sibale Team Leader, Sr. Evaluation Specialist Charles Masangano Local Extension Governance Specialist, Evaluation Specialist Charles Bakolo Local MELS Enumerator Denis Tweddle Team Leader, Senior Evaluation Specialist David Soroko Team Leader, Senior Evaluation Specialist David Coombs Team Leader, Seed Sector Specialist, Senior Evaluation Specialist Dezio Banda Local M&E Specialist Elizabeth Venable Evaluation Specialist, Capacity Building Emma Gausi Local Ag. Economist/ Gender Specialist, Evaluation Specialist Frank Sullivan Team Leader, Senior Evaluation Specialist Frank Wiebe Cost-Benefit Analysis Specialist, Senior Economist, Capacity Building Francis Matita Local M&E Specialist Geoffrey Kananji Local Agriculture Specialist, Local Evaluation Specialist Gracian Chimseu Local Notetaker, Evaluation Specialist Ivy Gondwe GIS/GPS Specialist, Capacity Building John Haydu Senior Agriculture Specialist, Evaluation Specialist, Capacity Building John Holtzman Team Leader, Ag. Economist, Senior Evaluation Specialist Keith Joseph Forbes Senior Evaluation Specialist, GCC/REDO Specialist Lawrence Mapemba Agricultural Specialist, Evaluation Specialist Maciej Chmielewski Learning and Knowledge Management Specialist Mayamiko Kakwera Local Evaluation Specialist, Local Agriculture Specialist Neveson Lemon Note Taker, Evaluation Specialist Nora Nelson Team Leader, Sr. Evaluation Specialist, NRM Specialist Prince Kaponda Local Natural Resource Management (NRM) Specialist Sandra Medina Qualitative Research Methods Specialist, Capacity Building Sergio Cambronero Team Leader, Senior Evaluation Specialist 19 MELS PERSONNEL IN 2017-2022 Steve Nelson Senior Fisheries Expert, Senior Evaluation Specialist Steve Hansch Senior Evaluation Specialist, Capacity Building Vitowe Batch Nutrition/Resilience Specialist, Evaluation Specialist Vitowe Harazi Local Agricultural Economist, Gender Specialist, Evaluation Specialist Wales Singini Local Fisheries and Evaluation Specialist William Fiebig Senior Agricultural Extension Specialist, Evaluation Specialist Willie Kachaka Field Manager and Quality Control Manager 20 Annex 2. Complete list of MELS documents uploaded to the Development Experience Clearinghouse (DEC) Document Title DEC Link Evaluation Reports FISH Midterm Evaluation (2017) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=MjM1NzM3 FISH Final Evaluation (2020) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTU5NjY3 PERFORM Midterm Evaluation (2017) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=MjM1MjAw PERFORM Final Performance Evaluation (2020) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTYxNzQ3 MISST Midterm Evaluation (2018) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTEyNzQ2 SANE (FtF) Midterm Evaluation (2019) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTUyNjYx AgDiv Midterm Evaluation (2020) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTcxNzI5 Special Studies and Assessments WALA Rapid Assessment (2018) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTAzNzA5 Meta-Analysis To be uploaded after completion and approval. Verification Reports Verification Report FY17 (FtF AgDiv) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkxNDc4&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= Verification Report FY18 (FtF AgDiv) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTE4NzMy Verification Report FY19 (FtF AgDiv) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTU5ODA4 Verification Report FY20 (FtF AgDiv) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkxMzk0&inr=VHJ1 ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= Verification Report FY21 (FtF AgDiv) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTk3Nzcy&inr=VHJ1 ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= Verification Report FY21 (MCHF) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTk3Nzcz&inr=VHJ1 ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Baseline and Annual Survey Reports AgDiv Annual Survey Report 2017 (baseline) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTg4MTcx&inr=VHJ1 ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Annual Survey Report 2018 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkxNTAw&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= 21 Document Title DEC Link AgDiv Annual Survey Report 2019 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkwMDA0&inr=VH J1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Annual Survey Report 2020 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTg3MzQ1&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Annual Survey Report 2021 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTk2ODM2&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Baseline Survey Documents 2017 Baseline and Performance Assessment Plan (FtF AgDiv) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTA4NjA3 Revised Data Treatment and Analysis Plan (FtF AgDiv) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTA4NjA1 Baseline Survey Supervisor Manual (FtF AgDiv) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTA4NjAz Baseline Survey Protocol (FtF AgDiv) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTA4NTY2 Baseline Survey Sampling Plan (FtF AgDiv) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTA4NjA5 Baseline Survey Enumerator Manual (FtF AgDiv) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTA4NTY1 Baseline Survey Questionnaire Translation Protocol (FtF AgDiv) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTA4NjA0 Baseline Survey Questionnaire Programming Guide (FtF AgDiv) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTA5MTY0 Baseline Survey Data Cleaning Plan (FtF AgDiv) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTA4NjA4 Baseline Survey Report (AgDiv) 2017 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTIyMjE3 Baseline Survey Cluster Allocation Procedures (FtF AgDiv) https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTA4NjA2 AgDiv Annual Survey Questionnaires AgDiv Annual Survey Questionnaire 2018 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkxNDc1&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Annual Survey Questionnaire 2019 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkxMjU1&inr=VHJ1 ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Annual Survey Questionnaire 2020 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkwMTI0&inr=VHJ1 ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Annual Survey Questionnaire 2021 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTk2NTYx&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Annual Survey Sampling Plans 22 Document Title DEC Link AgDiv Annual Survey Sampling Plan 2018 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkwNTkz&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Annual Survey Sampling Plan 2019 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkwNTky&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Annual Survey Sampling Plan 2020 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkwNjI1&inr=VHJ1 ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Annual Survey Sampling Plan 2021 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTk2NjQ4&inr=VHJ1 ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Baseline and Annual Survey Enumerator Manuals Baseline Survey Enumerator Manual (FtF AgDiv), FY17 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTA4NTY1 AgDiv Annual Survey Enumerator Manual FY18 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkxMjYx&inr=VHJ1 ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Annual Survey Enumerator Manual FY19 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkxMzQz&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Annual Survey Enumerator Manual FY20 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkxMzE3&inr=VHJ1 ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Annual Survey Enumerator Manual FY21 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTk2NTYz&inr=VHJ1 ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Baseline and Annual Survey Supervisor Manuals Baseline Survey Supervisor Manual (FtF AgDiv), FY17 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTA4NjAz AgDiv Annual Survey Supervisor Manual FY18 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkxMzEy&inr=VHJ1 ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Annual Survey Supervisor Manual FY19 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkxMzEw&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Annual Survey Supervisor Manual FY20 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkxMzQy&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= AgDiv Annual Survey Supervisor Manual FY21 https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTk2NTY1&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= MELS Annual Activity Reports MELS FY17 Annual Activity Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkxMzY2&inr=VHJ1 ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= MELS FY18 Annual Activity Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkxMzY0&inr=VHJ1 ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= MELS FY19 Annual Activity Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTg5ODk1&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= 23 Document Title DEC Link MELS FY20 Annual Activity Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTg5ODcw&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= MELS FY21 Annual Activity Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTk3Nzg0&inr=VHJ1 ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= MELS Quarterly Activity Reports MELS FY17 Q2 Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTgwMjE1 MELS FY17Q3 Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTgwMjE2 MELS FY18 Q1 Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTgwNjY2 MELS FY18 Q2 Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTgwNjY3 MELS FY18 Q3 Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTgwNjY4 MELS FY19 Q1 Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTgwNjY5 MELS FY19 Q2 Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTgwNjcw MELS FY19 Q3 Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTgwNjcx MELS FY20 Q1 Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTgwNjgz MELS FY20 Q2 Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTgwNjg0 MELS FY20 Q3 Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?vID=47&ctID=ODVhZjk 4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTgwNjg1 MELS FY21 Q1 Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkwMDgx&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= MELS FY21 Q2 Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkwMDg0&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= MELS FY21 Q3 Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTkwMDg2&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= MELS FY22 Q1 Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTk2OTI4&inr=VHJ1 ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL= MELS FY22 Q2 Report https://dec.usaid.gov/dec/content/Detail_Presto.aspx?ctID=ODVhZjk4NWQt M2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=NTk5MTA1&inr=VHJ 1ZQ%3d%3d&dc=YWRk&rrtc=VHJ1ZQ%3d%3d&bckToL=