th FY 2017 Côte d’Ivoire Food for Progress Report Final Evaluation September 30, 2021 1 Final Evaluation Report : MOCA Côte d'Ivoire Maximizing Opportunities in Cocoa Activity (MOCA) Final Evaluation Report Program: Food for Progress Agreement Number: FCC-681-2017/029-00 Funding Year: Fiscal Year 2017 Project Duration: September 2017- September 30, 2021 Implemented by: CNFA Evaluation Authored by: Naama Raz-Yaseef for The Khana Group DISCLAIMER: This publication was produced at the request of the United States Department of Agriculture. It was prepared by an independent third￾party evaluation firm. The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States Government Final Evaluation Report : MOCA Côte d'Ivoire 2 Table of Contents Table of Contents.......................................................................................................... 2 List of Tables ................................................................................................................. 5 List of Figures ............................................................................................................... 5 List of Acronyms........................................................................................................... 6 1 Executive Summary................................................................................................ 7 2 Introduction and Purpose .................................................................................... 10 2.1 The Cocoa Sector in Côte d’Ivoire................................................................ 10 2.1.1 Cocoa Production Challenges and Opportunities ...................................................................10 2.1.2 Other Challenges and Opportunities.......................................................................................11 2.2 The MOCA Project.......................................................................................... 12 2.2.1 MOCA Baseline Study ............................................................................................................12 2.2.2 Project Outcomes....................................................................................................................13 2.2.3 Project Outputs: Activities and Partners .................................................................................14 2.2.4 MOCA Timeframe and Budget................................................................................................16 2.2.5 MOCA Organizational Setup and Governance .......................................................................18 2.2.6 Project and Evaluation Area....................................................................................................18 2.3 Final Evaluation Purpose .............................................................................. 19 2.4 The Khana Group’s Approach to the Final Evaluation ............................... 20 3 Evaluation Design and Methodology .................................................................. 21 3.1 Data Collection............................................................................................... 21 3.1.1 Training and Piloting ...............................................................................................................21 3.1.2 Field Data Collection ...............................................................................................................22 3.1.3 Data Collection Challenges and Mediation .............................................................................22 3.2 Sample Framework and Design.................................................................... 23 3.2.1 Quantitative Data Sample Frame............................................................................................23 3.2.2 Qualitative Data Sample Frame ..............................................................................................27 3.3 Data Collection Tools .................................................................................... 29 3.4 Data Analysis.................................................................................................. 29 3.5 Ethical Considerations .................................................................................. 30 3.6 Evaluation Limitations................................................................................... 31 4 Findings................................................................................................................. 33 4.1 Relevance ....................................................................................................... 33 Final Evaluation Report : MOCA Côte d'Ivoire 3 4.1.1 To what extent has the project responded to the actual needs and interests of the target beneficiaries? ......................................................................................................................................33 4.1.2 To what extent does the project integrate or strengthen the national cocoa strategy of Côte d’Ivoire?...............................................................................................................................................37 4.2 Impact ............................................................................................................. 39 4.2.1 Access to inputs and input services ........................................................................................39 4.2.2 Access to finance and financial services ................................................................................42 4.2.3 Access to markets ...................................................................................................................45 4.2.4 Inputs.......................................................................................................................................47 4.3 Effectiveness.................................................................................................. 49 4.3.1 To what extent have the expected results of the project been achieved? ..............................49 4.3.2 Which activity areas were most critical to the achievement or non-achievement of project objectives? ..........................................................................................................................................60 4.3.3 To what extent did performance in assisted cooperatives improve because of MOCA activities?.............................................................................................................................................63 4.3.4 To what extent did the improved performance of assisted cooperatives contribute to positive outcomes and impact for farmers?......................................................................................................64 4.4 Efficiency ........................................................................................................ 64 4.4.1 To what extent have the various units/levels of management and coordination (grants, project staff, M&E) supported the implementation of the project?..................................................................64 4.4.2 How did the leveraging of funds through Public-Private Partnerships add value to the project? 65 4.5 Sustainability.................................................................................................. 65 4.5.1 To what extent did MOCA project management respond to needs and requests for collaboration from cooperatives, farmers, businesses, financing institutions, and the government institutions? .........................................................................................................................................65 4.5.2 Are project outcomes likely to be sustained and/or scaled-up after the project closes? ........66 5 Summary, Conclusions, and Recommendations............................................... 68 5.1 Summary and Conclusions........................................................................... 68 5.2 Project Successes ......................................................................................... 70 5.3 Project Challenges......................................................................................... 70 5.4 Recommendations for Future Programming............................................... 71 6 Appendixes ........................................................................................................... 73 6.1 Appendix 1: Evaluation Matrix...................................................................... 73 6.2 Appendix 2: References ................................................................................ 75 6.2.1 Academic papers and external reports ...................................................................................75 6.2.2 MOCA Data sources ...............................................................................................................75 6.3 Appendix 3: Project Timeline........................................................................ 77 Final Evaluation Report : MOCA Côte d'Ivoire 4 6.4 Appendix 4: List of KIIs ................................................................................. 78 6.5 Appendix 5: Farmer Household Questionnaire (Quantitative Tool) ......... 80 6.6 Appendix 6: Semi-structured FGD’s Questions (Qualitative Tool)............ 94 6.6.1 Farmers Focus Group Discussion Tool...................................................................................94 6.6.2 Cooperative Staff and Managers Focus Group Discussion Tool............................................97 6.7 Appendix 7: Semi-structured KII’s Questions (Qualitative Tool)............. 100 6.7.1 MOCA and SOCODEVI staff.................................................................................................100 6.7.2 Cooperative Management and Farmers Organizations ........................................................101 6.7.3 Financial Institutes, Governmental Representatives, and Exporters ....................................103 6.8 Appendix 8: Statistical Tables of Farmer Survey Results ........................ 107 6.9 Appendix 9: MOCA Final Evaluation Scope of Work ................................ 127 6.10 Appendix 10: Results Framework............................................................... 131 Final Evaluation Report : MOCA Côte d'Ivoire 5 List of Tables Table 1: MOCA timeframe, budget, and key activities .................................................. 17 Table 2: Survey Respondent Type Categories.............................................................. 24 Table 3: Survey Respondent Population by Categories ................................................ 25 Table 4: Collapsed Survey Respondent Categories...................................................... 26 Table 5: Regional Dispersion of Respondents .............................................................. 26 Table 6: Abengourou Endline Survey Respondent by Cooperative .............................. 26 Table 7: Daloa Endline Survey Respondent by Cooperative......................................... 27 Table 8: Soubre Endline Survey Respondent by Cooperative ...................................... 27 Table 9: Survey Respondent Type Categories.............................................................. 28 Table 10: Possession of financial accounts (% of individuals) ...................................... 43 Table 11: Loans received over last 2-3 years for self or other household member (% of individuals) .................................................................................................................... 44 Table 12: Largest amount of loan received (CFA francs).............................................. 44 Table 13: Primary payment method .............................................................................. 46 Table 14: Cocoa production and yields ......................................................................... 48 Table 15: MOCA FFPr Performance Indicators: Targets and Life of Project (LoP) Values ...................................................................................................................................... 50 Table 16: Cocoa income generated from sales............................................................. 54 Table 17: Cocoa production income.............................................................................. 54 List of Figures Figure 1 : Physical Characteristics of MOCA’s Geographical Region ........................... 19 Figure 2: Application of Pesticides (Insecticides and Fungicides) on Cocoa Farms...... 40 Figure 3: Application of Fertilizers (Chemical and Organic) on Cocoa Farms ............... 41 Figure 4: Application of Pruning on Cocoa Farms......................................................... 41 Figure 5: The practice of post-harvesting techniques.................................................... 42 Figure 6: Farmers receiving bonus on premium............................................................ 46 Figure 7: Size of premium ............................................................................................. 47 Figure 8: Number of Primary beneficiaries receiving training and/or support (% out of 664 Primary beneficiaries)............................................................................................. 56 Final Evaluation Report : MOCA Côte d'Ivoire 6 List of Acronyms Acronym Full Term 2QC Quantité, Qualité, Croissance (Quantity, Quality, Growth) ANADER Agence Nationale d’Appui au Développement Rural CCC Conseil Café et Cocoa CNFA Cultivating New Frontiers in Agriculture CNFRA The National Agricultural Research Center CNRA Centre National de Recherche Agronomique COVID-19 Coronavirus pandemic CSSVD Cocoa Swollen Shoot Virus Disease FCIA Fine Chocolate Industry Association FFPr Food For Progress FGD Focus Group Discussion FSC Farm Service Center GAP Good Agricultural Practices GIZ German Agency for International Cooperation KII Key Informant Interview KIT Royal Tropical Institute LoP Life of Project MFI Microfinance Institution MOCA Maximizing Opportunities for Cocoa Activities MoU Memorandum of Understanding NGO Non-Governmental Organization PERSUAP Pesticide Evaluation Report and Safe Use Action Plan PPP Public-private Partnership RFP Request for Proposal SOCODEVI Société de Coopération pour Le Développement International SPU Service Provision Unit SSP Spray Service Professional TKG The Khana Group USDA United States Department of Agriculture Final Evaluation Report : MOCA Côte d'Ivoire 7 1 Executive Summary Cultivating New Frontiers in Agriculture (CNFA) implemented the Maximizing Opportunities for Cocoa Activities (MOCA) project intended to increase the productivity and efficiency of actors in the cocoa value chain in Côte d’Ivoire. MOCA applied a holistic value chain development approach supporting farmers and other value actors with quality, production, and efficiency improvement activities, from input provision to market access and from access to finance to improved cooperative governance. It intended to increase their income through increased trade of and added value to cocoa beans. The Final Evaluation aimed to critically evaluate the project’s performance in terms of project relevance, effectiveness, efficiency, sustainability, and impact. The evaluation used a mixed-methods approach, employing both quantitative and qualitative methods. Quantitative data was collected from a sample size of 1,310 farmer cocoa producers, including farmers that were trained directly by MOCA and their partners (Primary beneficiaries), farmers that belong to cooperatives that MOCA worked with (Secondary beneficiaries), and a Control farmers Group1 . Qualitative data was collected through nine focus group discussions (FGDs) with MOCA Primary beneficiaries (farmers and cooperative management) and 33 key informant interviews (KIIs) with various stakeholders, including cooperative and farmers’ organization management, MOCA partners such as bank representatives, governmental representatives, exporters, and chocolatiers, and MOCA staff. The evaluation used, when possible, information from the Baseline study to evaluate project impact and change over time. Despite the short project time span compared to most development projects in the agriculture sector and the further reduction in the project’s implementation period, MOCA achieved and even overpassed by large most of the project performance targets. The evaluation was able to validate these achievements. Based on internal project monitoring, the project achieved 103% of the life of project (LoP) target for the number of individuals who have applied new techniques or technologies as a result of USDA assistance. The farmer survey results validated this achievement, indicating that 95% of MOCA farmers surveyed also applied at least one of the training they received from MOCA on improved input, agricultural best practices, and post-harvest techniques. In FGDs and KIIs with project beneficiaries, nearly all farmers confirmed that they apply the new techniques and technologies acquired. This was further validated in FGDs and KIIs with cooperative managers that said that these training were practiced by farmers. 1 This report uses the term “Primary” beneficiaries to describe farmers trained directly by MOCA and partners, and “Secondary” beneficiaries to describe the farmers who belong to cooperatives that MOCA worked with. This is to differentiate between the terminology used to report on Indicator FFPr 17: Number of individuals who have benefitted directly as a result of USDA assistance defined for MOCA as all individuals that directly benefited from project activities like training, technical assistance, or access to financial services, and FFPr 18: Number of individuals who have benefitted indirectly as a result of USDA assistance, which is defined as family members of MOCA direct participants. Final Evaluation Report : MOCA Côte d'Ivoire 8 Based on internal project monitoring, the number of hectares of land under improved techniques or technologies as a result of USDA assistance achieved 125% of the LoP target. When compared to Baseline values, the Endline survey results showed a small increase in the area on which fungicide and organic fertilizer were applied and a small decrease in the area on which insecticide and chemical fertilizer were applied since Baseline. Trends for MOCA farmers were similar to those for Control farmers. Based on internal project monitoring, the volume of commodities sold by project beneficiaries achieved 111% of the LoP target, and the value of sales of cocoa products by project beneficiaries achieved 174% of the LoP target. On average, MOCA beneficiaries made 69% more than at Baseline, and 17% more than the income of the Control group at Endline. Moreover, MOCA farmers were able to increase their income from bonuses received on premium more than Control farmers: twice as much MOCA farmers than Control farmers received a bonus, and the size of the average bonus received was larger for MOCA farmers than for Control. While income from bonuses account for only a small portion of the farmers annual income (4-5%), it shows potential to scale-up. 60% of respondents said they would continue applying the new practices in the coming years. In discussions and interviews, it became apparent that farmers need a longer project time span to retain the knowledge gained. Farmers asked that the support provided by MOCA will continue and said they need ‘refreshers’ and more opportunities to grow and further develop their capacities. The farmer's survey showed that the number of farmers that have a bank account has more than doubled since Baseline, and that the increase in the number of individuals that have a bank account increased more for MOCA farmers than for Control Farmers. However, this number is still low, and only 20% of MOCA farmers possess a bank account. In FGDs, more than half of the interviewed farmers said they have attended events with financial institutes arranged by MOCA and received training on how to open bank accounts, yet they did not apply for these financial services. More work is needed to identify bottlenecks and plan strategies to improve finance for producers and cooperatives. MOCA worked to improve the quality of cocoa and quality flavor cocoa production and trade. This activity strongly aligns with Côte d'Ivoire's strategy to add value without increasing cocoa production and high potential to bring change to the sector and increase income for farmers. MOCA’s work on improved production and post-harvesting techniques proved that the country’s producers could produce high-quality cocoa and high-quality flavor cocoa. MOCA’s work to link cooperatives with exporters and chocolatiers that purchased these products provided another proof of concept to the attainability of this activity for future projects. By partnering with sector actors and creating new public-private partnerships, the project was able to leverage USDA’s investment and bring other funders into Côte d'Ivoire's cocoa sector. The value of the new public and private sector investment leveraged by USDA assistance was more than 2.5 times higher than the target. MOCA provided In-kind grants to farmers' producers groups, established Farm Service Centers (FSC), and initiated a network of Spray Service Providers. Because most Final Evaluation Report : MOCA Côte d'Ivoire 9 MoU’s and grants were signed close to project completion, the impact of this activity could not be evaluated at Endline. Similarly, the time that has passed since the opening of the FSC was too short to evaluate their impact or sustainability. These successes can be attributed to the devoted and professional MOCA team, which created effective and efficient training and support activities. Project beneficiaries at all levels and project partners praised the transparency and responsiveness of project staff. The project leadership practiced a result-oriented, adaptive management style that responded to current actual needs and focused on activities that are achievable during the short life span of the project, and activities were tailored to the specific needs of each region, cooperative, and organization MOCA worked with. Acknowledgment The Khana Group (TKG) extends its sincerest gratitude to the Cultivating New Frontiers in Agriculture (CNFA) for having the confidence in our firm to lead the Maximizing Opportunities for Cocoa Activities (MOCA) Program Final Evaluation. We are grateful to MOCA/CNFA’s technical team for their continuous support throughout the different phases of this evaluation, with special recognition to the projects’ Chief of Party and the Director of Monitoring and Evaluation. We appreciate the inputs from all project beneficiaries who contributed to this Final Evaluation. We also greatly appreciate MOCA’s partners' input - financial institute representatives, exporting chocolatiers companies, and Ivorian Government officials who participated in the evaluation. Finally, we extend our appreciation to the United States Department for Agriculture (USDA) for funding this initiative and all local and international stakeholders. TKG also credits all researchers and institutions who offered their support in carrying out this study, including enumerators, group facilitators, translators, and transcribers. Final Evaluation Report : MOCA Côte d'Ivoire 10 2 Introduction and Purpose 2.1 The Cocoa Sector in Côte d’Ivoire Côte d’Ivoire is the biggest producer of cocoa beans globally, producing over 2 million tons of cocoa yearly. The cocoa sector is the backbone of the Ivorian economy, accounting for up to 40% of the export revenue (Kanga, Moussa, and Sanogo, 2018). Cocoa provides an important contribution to the thriving economy of Côte d’Ivoire, which is one of the most robust economies in Africa and worldwide and has grown at an annual average rate of 8% since 20122 . Cocoa is a promising revenue for the country, as the global demand for cocoa products remains high, and the forecast suggests that cocoa demand will continue to grow significantly over the next decade. However, cocoa prices are extremely volatile, impacted by global supply and demand, and production is highly reliant on climatic conditions. In 2016-2017, cocoa production in Côte d’Ivoire increased by 18% due to favorable weather conditions, resulting in an oversupply of cocoa on the world market and a drop in prices. The government of Côte d’Ivoire was forced to reduce the price it guaranteed to farmers by 36% (Bymolt, R., Laven, A., Tyszler, M., 2018). More recently, as in many other regions globally, the COVID-19 pandemic adversely affected Ivorian households and businesses and slowed the economic growth rate to 1.8% in 20203 . In Côte d’Ivoire, about 800,000 farmers make their living directly from cocoa production and are therefore vulnerable to changes in the global cocoa market. Most of these producers are domestic smallholders that are generally poor, without formal education, disorganized, and scattered across the country (Kanga, Moussa, and Sanogo, 2018). Managing money can be particularly challenging for farmers since they receive the majority of their income during the harvest, and this needs to cover their expenses for the rest of the year (Lonie et al., 2018). Smallholder cocoa farmers can only increase the quality of the beans they sell, which would otherwise be a valuable way to increase their incomes and improve their livelihoods. Further, these conditions make it even more challenging for them to cope with climate shocks, declining soil fertility, unproductive cocoa plants, and the spread of cocoa diseases. 2.1.1 Cocoa Production Challenges and Opportunities Good agricultural practices (GAP), including a selection of variety, control of pest disease, and harvesting and post-harvesting techniques, directly impact cocoa quantity, quality, and flavor. Cocoa production in Côte d’Ivoire faces several biotic constraints, including plant diseases, insect pest attacks, and rodent and parasite-related damage. Of special concern are damages caused by the Cocoa Swollen Shoot Virus Disease (CSSVD), an insect-borne pathogen that typically affects a tree's harvest in the first year of production and then kills the tree within three to four years. The harmful impacts of 2 The World Bank, https://www.worldbank.org/en/country/Côtedivoire/overview#1 3 As above. Final Evaluation Report : MOCA Côte d'Ivoire 11 CSSVD can be mitigated with correct treatment, but recent studies from neighboring Ghana show that only about half of the affected farms were treated, and about 73% of the growers who treated their affected farms did so incorrectly (Amon-Armah et al., 2021). Another impactful disease is the Black Pod disease. The disease is initiated by heavy rain that leads to excessive humidity. These conditions are ideal for the development of a fungus that spreads rapidly on the pods and makes it inappropriate for harvest. If left untreated, black rot can infect healthy plants. Of all cocoa diseases, the Black Pod disease, caused by the Phytophthora genus, is the largest impactor on cocoa production, with losses of 30-90% of the crop. On a global scale, it is estimated that Black Pod disease alone leads to losses of 700,000 metric tons (Adeniyi, 2018). Above￾average rain and warming, and humid conditions enhanced by climate change are anticipated to worsen cocoa losses from Black Pod disease in West Africa4 . While current control methods through routine spraying of inorganic fungicides have been found to be effective to some level, they are expensive, hazardous, and environmentally unfriendly. Alternatively, plant disease and cocoa productions methodologies that consider smallholders’ limited access to pesticides and cultural practices to combat Black Pod disease suggest promoting integrated pest management strategies that focus on eliminating sources of primary inoculum, eliminating the use of pesticides, or prescribing precise fungicide application, and combining proper farm management that focuses on reducing canopy shading. Increased light and airflow within the canopy with appropriate spacing, pruning, and weed control, are likely to reduce plant vulnerability to the Phytophthora pathogens 5,6,7 . 2.1.2 Other Challenges and Opportunities Additional challenges to the cocoa sector in Côte d’Ivoire arise from the increasing demand by global consumers, and therefore by many chocolate exporters, to meet higher environmental, ethical, and social standards. It is estimated that 40% of Côte d’Ivoire’s cocoa comes from protected areas, and the country’s extremely high deforestation rates are tightly connected to cocoa production8 . Children in Côte d'Ivoire are often engaged in labor and especially in cocoa production, harvest, and post￾harvesting9 , including in hazardous work of clearing and burning to expand plantation and using sharp tools to break pods. 4 High risk of Black Pod in West African cocoa -Climate42 consultancy, Reuters - Africa, Sept. 24, 2021. 5 World Cocoa Foundation: https://www.worldcocoafoundation.org/wp-content/uploads/files_mf/guest2007.pdf 6 Diversity of Cacao Pathogens and Impact on Yield and Global Production (Adeniyi, 2018). In “Deploying Science for Sustainability of Global Cocoa Economy” - https://www.intechopen.com/chapters/65131. 7 The Impact of Plant Diseases on World Chocolate Production (Bowers et al, 2001). In “Plant Health Progress. 10.1094/PHP-2001-0709-01-RV. 8 3% deforestation rate annually. UN-REDD program, https://www.un-redd.org/post/2019/12/02/integrating-redd-and￾private-sector-cocoa-initiatives-in-c%C3%B4te-divoire-creating-incentives 9 25.6% of children ages 5 to 14 work. US Department of Labor, Bureau of International Labor Affairs, https://www.dol.gov/agencies/ilab/resources/reports/child-labor/Côte-divoire Final Evaluation Report : MOCA Côte d'Ivoire 12 From these challenges, promising opportunities arise, with the capacity to increase farmers’ income from cocoa production and the well-being of rural communities. Recent evidence shows that farmers' membership in a good functioning cooperative, both certified and non-certified, had positive and significant effects on cocoa yields, prices, and living standards (Sellare et al., 2020, and Knößlsdorfer, Sellare, and Qaim, 2021). Therefore, improving cooperative capacity can increase farmers’ income. Applying correct agricultural practices can reduce the negative impact of plant diseases. In addition, the Ivorian government has recently been aiming to transform the cocoa sector by improving higher-quality markets by focusing on quality cocoa rather than increasing productivity. 10 The government adopted its national strategy to improve the farmers’ financial inclusion rate from 41% currently to 60% by 2024. The MOCA project aimed to address these opportunities by increasing the professionalization of producer organizations and cooperative capacity, improving access to inputs, applying good agricultural practices, enhancing quality agricultural services, and increasing access to finance. 2.2 The MOCA Project Cultivating New Frontiers in Agriculture (CNFA) implemented the Maximizing Opportunities for Cocoa Activities (MOCA) project intended to increase the productivity and efficiency of actors in the cocoa value chain in Côte d’Ivoire. The MOCA project was funded by the United States Department of Agriculture (USDA) under the Food for Progress (FFPr) initiative. MOCA applies a holistic value chain development approach supporting farmers and other value actors with quality, production, and efficiency improvement activities, from input provision to market access; and from access to finance to improved cooperative governance. It intended to increase their income through increased trade of and added value to cocoa beans without contributing to an increase in cocoa production. 2.2.1 MOCA Baseline Study CNFA contracted the Royal Tropical Institute (KIT) to conduct the Baseline study for the MOCA project. The Baseline study was required to establish quantitative values for the MOCA performance indicators before starting project activities and to assess project outcomes through the Final Evaluation study. The study also established the baseline situation of two target beneficiary populations: cocoa producers and cooperatives. The Baseline study included a household questionnaire (N=1300), a cooperative questionnaire (N=26), and Focus Group Discussion (FGD) to collect qualitative information from cooperative members on cocoa trade and services. 10 In 2014 the Ivoirian Government started implementing the Quantity- Quality-Growth (2QC) program with the objective to transform the cocoa sector to the benefit of farmers’ income and livelihoods and increase local transformation and value addition to cocoa beans and products. Final Evaluation Report : MOCA Côte d'Ivoire 13 The baseline study informed the MOCA leadership in detail about the actual situation of farmers in the intervention area. These results were used to decide which strategies and activities suggested in the project proposal should be implemented and which should be changed. For example, the number of farmers impacted by plant disease showed a need for increased use of inputs that were not or only partly supplied through the CCC or cooperatives. As a result, MOCA’s identified the need for the establishment of Farm Service Centers. The Baseline also showed that only 1% of producers have a bank loan, and 1% have access to microfinance. This confirmed the need to increase access to financial services dedicated to agricultural production. The Baseline study defined different levels of anticipated project outcomes, outputs activities, and outputs partners as part of the project’s Theory of Change, detailed in the following sections. 2.2.2 Project Outcomes Long-term Outcomes MOCA has two strategic objectives designed to increase cocoa farmers’ income by improving the farmers’ cocoa production and adding value to the producers’ cocoa through increased volume and value of cocoa sales. These strategic objectives aligned with the ambitions of the Côte d’Ivoire government through its Quantité, Qualité, Croissance (2QC; Quantity, Quality, Growth) program, and the African Cocoa Initiative II project implemented by the World Cocoa Foundation (WCF). Working through these objectives, the MOCA project aimed to improve the competitiveness of the cocoa sector in Côte d’Ivoire as part of the Food for Progress goal of the USDA, and especially the aims to “improve agricultural productivity” and “expand trade of agricultural products.” Strategic Objective 1: To increase the productivity and efficiency of actors in the cocoa value chain through strengthening the capacities of cooperatives/producer groups, research institutions, input suppliers, and processors of cocoa. This Strategic Objective aligns with FFPr 1.2: Increased Use of Improved Agricultural Techniques and Technologies. Strategic Objective 2: Improve and expand the trade of cocoa and cocoa products through improving the overall quality of the crop, the processing and post-harvest handling techniques, and strengthening markets. This Strategic Objective aligns with FFPr 2.1: Increased Value Added to Post￾Production Agriculture Products. Medium-term Outcomes MOCA sought to provide added value to 2QC and WCF by implementing a market￾based intervention that uses a ‘market-pull’ strategy instead of a technology-push approach. Within this context, MOCA was designed to work on eight medium-term goals, to contribute to the sustainability of the cocoa sector in Côte d’Ivoire: • Cocoa quality (quality-flavor beans): the quality of the cocoa beans will be improved to a standard that is acceptable to the international quality-flavor cocoa market. Côte Final Evaluation Report : MOCA Côte d'Ivoire 14 d’Ivoire is the biggest producer of ‘ordinary’ cocoa beans but does not supply quality￾flavor beans that obtain a premium price. • Professionalization and diversification of cocoa producers to achieve higher incomes from cocoa activities. • Local transformation: increased engagement of Ivoirian entrepreneurs in cocoa value addition through local and national cocoa transformation industries. • Access to finance: cooperatives become eligible for credit lending due to strengthened capacities in finance and business management. • Access to markets: cooperatives will export cocoa directly to alternative international niche markets (e.g., fine-flavor, certified organic). • Cooperative management capacities: cooperative operations will become more efficient and compliant with cooperative principles. 2.2.3 Project Outputs: Activities and Partners The overall strategy of the MOCA project provided an operational and programmatic overview of activities for its eight programmatic components, defined in the Baseline study: 1. Capacity building: producer groups / cooperatives MOCA worked to build the capacity of selected cooperatives over the life of the project in areas including governance, management practices, human resources, financial management, service delivery, external relations with input and service suppliers and buyers, and sustainability. 2. Capacity building: government institutions MOCA worked to support the Conseil Café-Cacao (Conseil), Centre National de Recherche Agriculture (CNRA), and Agence National d'Appui au Développement Rural ANADER to expand research and propagation of disease-resistant and improved cocoa seeds and seedlings and access to these materials through the project’s grant mechanism and technical assistance. 3. Capacity building: provide business development services MOCA worked to deliver tailored business coaching to processors and supporting businesses in rural and urban areas along the cocoa value chain who are currently or would like to launch businesses in areas such as cocoa grinding and processing, value addition, and pooled transportation. 4. Financial services: facilitate agricultural lending MOCA worked to increase producers’ use of mobile money, insurance, and credit services and pilot new financial services such as crop insurance and innovative delivery channels for cash and in-kind credit. MOCA partnered with banks and microfinance institutions (MFIs) active in the cocoa sector to assess their current financial products and will help them to promote and/or adjust existing activities to best target and impact the cocoa sector. 5. In-kind grants: equipment and inputs Final Evaluation Report : MOCA Côte d'Ivoire 15 MOCA worked to facilitate the research and adoption of improved productivity and post-harvest handling practices through in-kind matching grants to cooperatives, producers, input supply professionals and processors throughout the cocoa value chain in the form of inputs and equipment. 6. Inputs: develop agro-dealers and/or other input suppliers MOCA trained and established a network of spray-service professionals (SSPs)—mostly male youth who are engaged in cocoa productivity—to provide affordable fee-based services, facilitated by cooperatives, for other producers. To support this activity, MOCA launched a Pesticide Evaluation Report and Safe Use Action Plan (PERSUAP) to evaluate health and environmental risks and inform Integrated Pest Management promotion, and trained the participants selected for the program on business management. 7. Training of improved agricultural production techniques MOCA provided a package of support, including trainings and pruning tools for improved agricultural productivity and quality for cocoa and proper and safe application of inputs and business development services and coaching to inform and guide their decision-making during the growth period before plants yield a crop. 8. Market access: facilitate buyer-seller relationships MOCA worked with unorganized farmers and/or weak association/cooperatives who did not have formal relationships with exporters to facilitate stronger linkages with reputable processers and buyers. During the life of project, MOCA worked with over 13,000 farmers receiving training or coaching directly from MOCA or partners (“Primary beneficiaries”), 24 cooperatives, 38 private enterprises, producers’ organizations, trade and business associations, and community-based organizations. In addition to project beneficiaries, MOCA partnered with several key strategic stakeholders. These include: • Governmental organizations, such as Conseil de Café et Cocoa (CCC), a governmental organization responsible for regulating and the cocoa sector in Côte d’Ivoire, Agence National d'Appui au Développement Rural (ANADER), which implemented some of MOCA’s activities, and the Centre National du Recherche Agronomique (CNRA) whose research was supported by MOCA especially for the establishment of the cocoa Quality Flavor Lab. • Banks, Micro-Finance Institutions (MFI), and insurance companies, to establish linkages with MOCA’s cocoa producers, including Advans, Oikocredit, BAOBAB, SAHAM, and SUNU Insurance. • Input and Service Providers, to provide easy access to local input manufacturers for partnership building, including CropLife and Callivoire, OLAM and GIZ that partnered with CNFA to develop Farm Service Centers. Final Evaluation Report : MOCA Côte d'Ivoire 16 • Cocoa Industry actors and exporters, including OLAM, ECOM, Guittard, Tony's Chocolonely, Barry-Callebaut, and the Fine Chocolate Industry Association (FCIA). 2.2.4 MOCA Timeframe and Budget The MOCA project was awarded in September 2017 with a timeframe of 3 years and an operating budget of 10,552,000 USD. The first third of the budget was received only in December 2018, which required project extension through July 30, 2021. Monetary delays created challenges as project activities could not be implemented before approval of the Baseline study and project targets, yet the Baseline study could not be achieved without funding. The project’s operating budget was reduced to 9,736,000 USD in September 2020 and to 9,236,000 USD in January 2021 due to increased prices of commodities in the US and reduced commodity sales prices. This reduction brought the total monetization shortfall to 1.3 USD million and was expected to slow-down activity implementation beginning in January 2021. In March 2021, CNFA submitted a Notice of Funding Opportunities (NOFO) request for supplemental funding to compensate for the unexpected monetization shortfall, with the notion that funding the project below 100% may tamper the project team’s efforts to implement grant activities fully. USDA rejected the NOFO request in May 2021, justified by CNFA’s capacity to adequately address key programmatic objectives and goals not only with decreased funding but also under COVID-19 pandemic restrictions and challenges. In this letter, USDA stated some of CNFA’s accomplishments, including reaching over 8% of its target for financial services and market access activities, respectively, as reasons for not approving the NOFO request. Monetary budget reductions reduced CNFA’s capacity to implement all necessary activities and led to some shortfall in target achievement, as further discussed in Section 4.3.1. The global COVID-19 pandemic led to additional delays and the cancelation of activities because of travel and meeting restrictions. Because of the crisis, the Ivoirian government limited multi-people gatherings, and as a result, all farmer training were suspended. The presidential elections of October 2020 created social unrest and security issues that reduced mobility during September and October 2020. The effective implementation period of project activities runs from January 2019 to the end of June 2021, representing 2.5 years for project implementation. Table 1 summarizes changes in the MOCA timeframe and budget and key activities for each semi-annual period. Final Evaluation Report : MOCA Côte d'Ivoire 17 Table 1: MOCA timeframe, budget, and key activities Semi￾Annual Period Project Budget Operational Status USDA Approval of Indicators/ Workplans Main Activities and Results December 2018- March 2018 MOCA project awarded at $10,552,00 0 for 3- years Project activities postponed due to monetizing delays Year 1 work plan approved • Operational start up • Subaward with SOCODEVI • Coordination with CCC • Initial program scoping April 2018- September 2018 Project activities postponed due to monetizing delays • Network to establish sector partnerships • Prepare baseline study and geo￾mapping October 2018-March 2019 In January 2019 actual project implementation period begins Baseline completed, project targets confirmed • Established PPP with OLAM, Jacobs, and GIZ • Selected 26 cooperatives • RFPs for FSC with Callivoire • Established relationships with 6 U.S.-based chocolatiers April 2019- September 2019 actual project implementation period, COVID￾19 restrictions begin • Engaged with Guittard, CQC and CocoaHouse • Facilitated access to finance • Trained and coached 24 cooperatives • 2 nd round of RFPs for FSC with Callivoire October 2019-March 2020 actual project implementation period, COVID￾19 restrictions continue USDA approves revised project targets • Trained farmers on GAPs with focus on pruning • Additional 4 partnerships with Financial and Micro Credit Organizations • Coop training and coaching continued • International Quality Flavor Cocoa Conference with CCC postponed because of COVID-19 April 2020- September 2020 In September 2020, project budget reduced to $9,736,000 actual project implementation period, COVID￾19 restrictions continue • Trained farmers on GAPs • Distributed 1,500 pruning tools • Increased access to credit and working capital • Coop training and coaching continued • 4 FSC grants provided Final Evaluation Report : MOCA Côte d'Ivoire 18 October 2020-March 2021 In January 2021, project budget reduced to $9,236,000 actual project implementation period, COVID￾19 restrictions continue, USDA approves a no￾cost project extension Year 4 work plan approved • Trained on GAPs • Distributed 7,913 pruning tools • Increased access to credit and working capital • Coop training and coaching continued • Signed 8 matching grants • CAPRESSA cooperative exported the first 25 MT of quality flavor cacao 2.2.5 MOCA Organizational Setup and Governance MOCA has four offices: a headquarter office in Abidjan, and three regional offices in the main city of all three MOCA regions: Abengourou, Daloa, and Soubré. The main positions in the Abidjan office include the Chief of Staff, Marc Steen, a senior director of programs, grants manager, finance director, monitoring and evaluation director, and a communications specialist. To reduce the project’s staff travel time to the implementation locations, the regional offices included value chain development managers, cocoa production specialists, cooperative capacity building facilitators, monitoring and evaluation officers, and supporting staff such as accounting assistants and drivers. The project also employs a gender advisor. CNFA’s headquarters in Washington D.C supports the Côte d’Ivoire office through weekly meetings, periodic visits in Côte d’Ivoire, and collaborations on specialized topics. The project’s management was in continuous and direct contact with USDA’s Foreign Agricultural Service Manager in Ghana, Dan Archibald, and with USDA’s contracting officer representative, Rebecca Coggins, who also visited the project in September 2019. The project has a Performance Monitoring Plan with targets that have been adjusted in January 2020 and a few targets that were revised in September 2020. USDA has approved a Work Plan for the first and fourth years of the project. During the project implementation, semi-annual reports were prepared (a total of seven), describing the main results, activities, and progress made during each semi-annual period. A mid-term review was initially planned for the project at Baseline but was waved by USDA due to the shortening of the project implementation period. 2.2.6 Project and Evaluation Area The MOCA project operated in three cocoa-producing regions: the Indénié-Djuablin region in the East (regional capital: Abengourou), the Haut Sassandra region in the Center-West (major town: Daloa), and the region of Nawa in the South-West (main town: Soubré) (Figure 1). These cocoa-growing regions are blessed with a tropical climate and an average annual rainfall of around 1,300 mm, mostly falling in the summer months. Much of the area is covered with dense forests that have gone through severe ecosystem degradation due Final Evaluation Report : MOCA Côte d'Ivoire 19 to heavy pressure from agricultural activity and logging. Agricultural activities dominate the economy of the region. The main agricultural value chains are cocoa, coffee, rubber, oil palm, and food crops11 . Figure 1 : Physical Characteristics of MOCA’s Geographical Region Three MOCA regions – Daloa, Soubré and Abengourou, are presented on a topographical map Figure 1(A). Côte d’Ivoire average annual precipitation map is presented in Figure 1(B). 2.3 Final Evaluation Purpose The purpose of the MOCA Final Evaluation is to evaluate the project’s performance critically and objectively in terms of the following key indicators: relevance, effectiveness, efficiency, sustainability, and impact. The evaluation also reviewed MOCA’s performance indicators targets and results as defined by USDA. The detailed evaluation matrix is presented in Appendix 1. The Final Evaluation used a mixed-methods approach, employing both quantitative and qualitative methods. Quantitative data was collected from the project’s Primary beneficiaries (farmer participating directly in MOCA interventions), Secondary beneficiaries (farmers that are members of cooperatives MOCA worked with directly), and a Control farmers group (farmers that did not benefit from MOCA’s work and were not members of any of the cooperatives MOCA worked with). The qualitative data was collected from cooperative leadership and farmer organizations. In addition to beneficiaries, qualitative data was gathered from representatives of government agencies, credit facilities providers, and cocoa exporters who provided technical support 11 MOCA Semi-annual Report, April 2018. Final Evaluation Report : MOCA Côte d'Ivoire 20 to beneficiaries during MOCA implementation. In consultation with MOCA staff, these representatives were selected to provide insights on the project performance, based on their availability and ability to provide the required information for the evaluation. Program staff was also interviewed to provide further insights, contexts, and information about the project. The Final Evaluation seeks to find evidence of a cause-effect relationship between the project interventions and results at the farmer level based on a counterfactual evaluation design. Therefore, the comparison of changes before (Baseline) and after (Endline) MOCA’s intervention received much focus in the analysis for the Final Evaluation. The quantitative analysis includes two sets of data: a. Endline Survey – new data collected for the Final Evaluation between June 23 and July 9, 2021 b. Baseline Survey – data collected for the Baseline study between October 7 and October 30, 2018 The sample design, sample strategy, tools, and analysis approach for qualitative and quantitative data are presented below. 2.4 The Khana Group’s Approach to the Final Evaluation The Khana Group (TKG) is a social impact management consulting and research firm working in sub-Saharan Africa focused on using data and evidence to develop solutions to positively impact and transform communities. The firm has offices in Ghana, Nigeria, Liberia, and the United States and has worked on over 200 projects in 20 countries in the past decade. TKG provides strategies that integrate results measurement and impact evaluation techniques proven through field experience, grounded in local knowledge, and informed by state-of-the-art approaches and tools. Final Evaluation Report : MOCA Côte d'Ivoire 21 3 Evaluation Design and Methodology The Final Evaluation had four distinct phases: project inception, data collection, data cleaning and analysis, and reporting and close out TKG adopted a participatory approach and included MOCA/CNFA staff during key points at all four phases of the evaluation to ensure that the project’s information needs are met. The inception phase commenced with an initial kick-off meeting with the MOCA/CNFA team to deliberate on the evaluation approach and timelines and ensure that TKG’s strategy for accomplishing the evaluation is acceptable by MOCA/CNFA. This created a collaborative atmosphere to help ensure a successful evaluation. For example, during the inception phase, MOCA staff elucidated all questions TKG had on project materials in an elaborative and transparent way. TKG suggested a list of stakeholders for KIIs, which the MOCA team provided. During the data collection phase, MOCA staff presented the project to TKG enumerators in the Training Workshop and connected the TKG team to cooperative leadership of the sampled communities. During this phase, weekly meetings were held with key MOCA/TKG staff to address issues in real-time. Last, all reports and deliverables were reviewed by MOCA staff. TKG reviewed the MOCA team's remarks and suggestions, and deliverables were edited accordingly prior to their final acceptance. 3.1 Data Collection 3.1.1 Training and Piloting Following the approval of the Inception Report, TKG recruited 40 local data collectors with prior experience in surveying, interviewing, and facilitating discussions. TKG trained data collectors during a two-day workshop, June 18-19, 2021. On the first day, data collectors were introduced to the MOCA project and the evaluation methodology. MOCA’s Program Director provided a detailed presentation on MOCA’s objectives, activities, and partners and answered questions from the data collectors on project goals and activities. The second day focused on research tools. The data collectors were divided into two groups. The larger group, intended for quantitative data collection, learned surveying methodologies and practiced using the farmers’ survey tool, including the electronic survey data collection tool. Enumerators with more experience and expertise in interviewing and facilitating group discussions were intended for qualitative data collection and compiled the second group of data collectors. This group learned and practiced the KIIs and FGDs tools. Feedback provided by the TKG team ensured that the enumerators acquired a concrete understanding of the best practices to be used with each tool to achieve data collection goals. The following day after the training workshop was devoted to pilot testing of tools in Abengourou. Due to heavy rains, this activity was postponed in one day. After the pilot test, a debriefing session was carried out with the enumerators to provide feedback on the pilot session and finalize the survey tools. The best performing enumerators were selected for deployment – 29 quantitative enumerators and six qualitative specialists. Quantitative data collectors were divided Final Evaluation Report : MOCA Côte d'Ivoire 22 into six teams: five teams with four enumerators and one supervisor, and one team with three enumerators and one supervisor. Three qualitative teams were formed, with two qualitative specialists in each. Two quantitative teams and one qualitative team were deployed in each of the three MOCA regions. 3.1.2 Field Data Collection Upon arrival in the various localities, first, supervisors contacted the cooperative leaders to identify and locate respondents. Next, enumerators entered the communities to collect data. The supervisors performed quality checks on each survey for quantitative data at the end of each data collection day. Completed and validated forms were uploaded to the server using an ADSL connection. Data were checked and cleaned concurrently with data correction, and issues were immediately resolved. For qualitative data, enumerators sought the consent of respondents to record KIIs and FGDs. Once an interview was completed, the recording was uploaded to a Dropbox folder before being transcribed verbatim and translated to English. TKG hired a team to review all transcriptions and compare them with corresponding audio recordings before initiating data analysis. 3.1.3 Data Collection Challenges and Mediation During deployment, the groups of data collectors experienced some difficulties. Due to heavy rains, road conditions were often poor, making travel difficult and reaching locations slower than anticipated. Farmers’ Survey. Enumerators encountered difficulties in locating the original Baseline respondents. Many phone numbers assigned several years ago did not work anymore or were not connected with the same person. Some respondents have left their villages, others left the cooperatives, some were not known even within their communities, and one has unfortunately died. Often, it was difficult to identify respondents based on Baseline names, which did not match the actual name of the respondents. The evaluators worked many hours to identify and synchronize Baseline with Endline respondents. Identifying the original Baseline Direct and Secondary beneficiaries were easier than the original Baseline Control respondents. The MOCA cooperative’s leadership was more in touch with members and could help locate them. Overall, from the original Baseline sample, 49 MOCA Primary respondents (16% of the Primary respondents sampled at Baseline), 319 MOCA Secondary respondents (52% of Secondary respondents sampled at Baseline), and 225 Control respondents (60% of Control respondents sampled at Baseline) could not be located. The reasons for failure to re-sample original baseline respondents included: • Respondents unavailable for interview: on travel, moved outside the community, immigrated to another country, or died. • Respondents unidentifiable: name as spelled in the Baseline database is unknown to the cooperative and community members, and the phone number provided in the Baseline respondent list does not work or was assigned to a different person. Final Evaluation Report : MOCA Côte d'Ivoire 23 To reduce attrition resulting from the inability to locate all the original Baseline survey respondents, a replacement strategy was put in place, prioritizing MOCA’s Primary beneficiaries, when possible, from the same community. This replacement approach was recommended by MOCA staff and accepted by the evaluation team, as discussed in the following section, Sample Framework and Design. Focus Group Discussions. Upon arrival at the villages, enumerators found that not all beneficiaries invited to participate in FGD’s were available. In some cases, cooperative leaders were able to mobilize but FGD capacity was often less than the intended 10 participants. Upon completing the planned two-weeks field work period, TKG decided to extend data collection to achieve higher saturation of Baseline respondents in the Endline survey. Two groups of five enumerators continued to work in the field for two more days, and the other four groups returned to Abidjan and surveyed farmers remotely for another four days. The enumerators used the same questionnaire and electronic tool to interview informants over the phone instead of in the field. During this period, surveyors collected 63 more original Baseline respondents, 24 more Primary beneficiaries, and 39 more Secondary beneficiaries. Field data collection started on June 23rd, 2021, ended on July 4th for four groups, and on July 6th for the remaining two groups. Remote data collection started on July 5th and ended on July 9, 2021. 3.2 Sample Framework and Design Quantitative data was collected from the project’s Primary beneficiaries, Secondary beneficiaries, and a Control farmers group. Qualitative protocols were appended to the quantitative survey to gather qualitative data from cooperatives leadership and farmers' organizations. In addition to quantitative and qualitative data collected from the project’s beneficiaries, key informant interviews were conducted with project partners and stakeholders, as well as with project staff. 3.2.1 Quantitative Data Sample Frame To conduct an impact evaluation, TKG aimed to collect data for the Final Evaluation by surveying the original Baseline sample when possible. Baseline Sample Frame The Baseline approach calculated a required sample size of 1,300 farmers based on a compliance rate of 90% and high heterogeneity in the type of support provided by MOCA to cooperatives and farmers. At the time of the Baseline survey design, the project was already more than a year into the implementation period but still in the preparatory phase due to monetization delays. Although most activities could not be carried out, based on the networking and assessment work conducted during this period, MOCA was able to identify potential cooperatives to work with after Baseline study completion. The Baseline survey was, therefore, able to include two types of respondents: Final Evaluation Report : MOCA Côte d'Ivoire 24 • Treatment - ‘MOCA’. These are farmers that were members of cooperatives MOCA was planning to work with. • Control - ‘Control’. These are farmers that were not members of cooperatives MOCA was planning to work with. Survey participants from the two groups were randomly sampled from cooperative membership lists provided by the MOCA team. The number of sampled households was divided roughly into three between each of the project’s activity regions, regardless of their actual population size. Endline Sample Frame MOCA farmers sampled at Baseline were divided into two types of respondents at Endline (Table 2): • ‘Post-baseline Primary beneficiaries’ - farmers that received direct training or input from MOCA intervention. • ‘Post-baseline Secondary beneficiaries’ - farmers who did not receive training or input from MOCA but were members of a cooperative that received support from MOCA. Not all original Baseline respondents were identified in the field at Endline, as described in Section 3.1.3. Therefore, the replacement strategy plan was initiated, prioritizing Primary beneficiaries, as requested by MOCA. This was done to compensate for the small group of Post-baseline Primary beneficiaries (308; 24% of Baseline respondents; Table 3) and collect more information from Primary MOCA farmers. As a result, the two Baseline respondent groups, MOCA, and Control, were further divided into six groups in the Endline survey (Table 2). Endline survey respondents that were not in the Baseline survey are referred to as ‘Non-baseline’ respondents. Table 2: Survey Respondent Type Categories Baseline Survey Respondent Category Endline Survey Respondent Category Participated in Baseline Survey Did Not Participate in Baseline Affiliation to MOCA: Primary Affiliation to MOCA: Primary Affiliation to MOCA: Control Treatment (MOCA) Post-baseline Primary beneficiaries X X Treatment (MOCA) Post-baseline Secondary Beneficiaries X X Control Post-baseline Control X X Treatment (MOCA) Non-baseline Primary beneficiaries X X Treatment (MOCA) Non-baseline Secondary Beneficiaries X X Control Non-baseline Control X X Final Evaluation Report : MOCA Côte d'Ivoire 25 405 new (“Non-baseline”) Primary beneficiaries were randomly selected from the same communities of the missing Primary beneficiaries out of the list of 13,000 direct beneficiaries provided by MOCA (102 in Abengourou, 159 in Daloa, and 144 in Soubré). 139 new Secondary respondents were randomly selected from the same communities of the missing Secondary beneficiaries that belong to cooperatives supported by MOCA to replace the missing Post-baseline Secondary beneficiaries (25 in Abengourou, 84 in Daloa, and 30 in Soubré). And 59 new Control respondents were randomly selected from the same communities of the missing Post-baseline Control respondents to replace the missing Control respondents (5 in Abengourou, 4 in Daloa, and 50 in Soubré) (Table 3). Table 3: Survey Respondent Population by Categories Endline Survey Respondent Category Baseline Survey Sample Size Endline Survey Sample Size % of Endline to Baseline Post-baseline Primary beneficiaries 308 259 84% Post-baseline Secondary Beneficiaries 616 297 48% Post-baseline Control 376 151 40% Non-baseline Primary beneficiaries _ 405 _ Non-baseline Secondary Beneficiaries _ 139 _ Non-baseline Control _ 59 _ Total 1,300 1,310 The Primary beneficiaries were 24% of the Baseline survey population and 51% of the Endline survey population (Table 4). The number of households sampled in each region followed Baseline guidelines and was nearly equally divided between regions (Table 5). Final Evaluation Report : MOCA Côte d'Ivoire 26 Table 4: Collapsed Survey Respondent Categories Category Baseline Sample Size Endline Sample Size Endline/ Baseline Primary beneficiaries (% of survey) 308 (24%) 664 (51%) 216% Secondary beneficiaries (% of survey) 616 (47%) 436 (33%) 71% Control (% of survey) 376 (29%) 210 (16%) 56% Total 1,300 1,310 101% Table 5: Regional Dispersion of Respondents Abengourou Daloa Soubré Baseline 428 447 425 Endline 415 455 440 Endline/Baseline 97% 102% 104% The Endline survey revisited the same cooperatives sampled in the Baseline survey. The sampled cooperatives in each region and the number of Primary beneficiaries, Secondary beneficiaries, and Control respondents sampled in each cooperative are listed in Table 6, Table 7, Table 8). Table 6: Abengourou Endline Survey Respondent by Cooperative No. ADI* AHONDJO* BOUKABLA CAPA CAPRESSA CEEA* COPPAZA FAHO N’MOH￾BLA SCAASP Primary beneficiaries 198 0 0 8 8 78 0 54 31 6 13 Secondary beneficiaries 117 0 0 12 8 35 0 39 17 6 0 Control 96 24 17 0 0 0 55 0 0 0 0 Total sample size 411 24 17 20 16 113 55 93 48 12 13 *signifies a non-MOCA coooperative Final Evaluation Report : MOCA Côte d'Ivoire 27 Table 7: Daloa Endline Survey Respondent by Cooperative No CA PE DIG CASE DU PLAN TEUR ECO JAD ESP OIR KAPAT CHIVA SAMAC OOP* SCACA LAVA SCO OPS SAH S SCOPR ANHS SOCA DMV SOCA PHS Primary beneficia ries 227 12 13 30 15 24 0 49 18 17 21 28 Seconda ry beneficia ries 14 22 55 18 67 0 14 14 0 0 10 19 Control 0 0 0 0 0 0 3 0 0 0 0 0 Total sample size 452 26 24 85 33 91 3 63 32 17 31 47 *signifies a non-MOCA coooperative Table 8: Soubre Endline Survey Respondent by Cooperative No CA AM CAN OA* CAP CB ECA M ECAS O* SCA ES SCA S* SCOP AB* SOCAG NIPI SOCOOA BAS SOCOOP ACDI Primary beneficia ries 24 0 21 0 18 67 0 72 0 0 24 15 25 Seconda ry beneficia ries 94 5 0 9 31 0 14 0 0 10 16 26 Control 98 0 48 0 0 12 0 7 11 0 0 0 Total sample size 43 2 26 48 27 98 12 86 7 11 34 32 51 *signifies a non-MOCA coooperative 3.2.2 Qualitative Data Sample Frame Focus Group Discussions Nine Focus Group Discussions (FGDs) were conducted with Primary beneficiaries, three FGDs in each of the three MOCA regions. To enhance participation and discussion around collective topics and enable participants to speak more freely, separate groups were held for unique groups: two farmer groups and one cooperative management group were held in Soubré and in Abengourou. In Daloa, where some work was done on gender equity, separate FGDs were held for women, men, and youth. The sample included cooperatives that represented success, such as CAPRESSA and FAHO, and cooperatives that represent challenges, such as Final Evaluation Report : MOCA Côte d'Ivoire 28 SOOCOABASS, that ceased to exist during the MOCA implementing period due to financial issues. The sample plan aimed for 10 participants in each discussion group. As previously discussed, not all invited participants arrived for the FGD sessions. In one case (women farmers FGD), more than 10 participants arrived and asked to join the discussion. In another case (SOOCOABASS cooperative management FGD), the cooperative had only 3 management members. On average, there were 8 participants in each FGD. Table 9 lists the type of FGD and the number of participants conducted in each region. Table 9: Survey Respondent Type Categories Region Cooperative Type of Focus Group No. of Participants Soubré SOOCOABASS Farmers (mixed age and gender) 8 Soubré ECAM Farmers (mixed age and gender) 6 Soubré SOOCOABASS Cooperative Management 3 Daloa KAPATCHIVA Youth Farmers 10 Daloa SCALAVA Women Farmers 12 Daloa CANAAN Men Farmers 6 Abengourou FAHO Farmers (mixed age and gender) 7 Abengourou COOPAZA Farmers (mixed age and gender) 9 Abengourou CAPRESSA Cooperative Management 7 TOTAL 68 Key Informant Interviews (KIIs) Key informant interviews (KIIs) were held with project beneficiaries and project partners, including cooperatives, financial institutes, governmental agencies, and exporters. Additional KIIs were held with CNFA/MOCA/SOCODEVI project staff. Informants were selected based on purposive sampling that represents both local activities, county-level collaborations, and global project management. This sampling process was designed to engage informants from all relevant regions by selecting those most knowledgeable about what the project has achieved and how change has been made. In total, 33 informants were interviewed for the Final Evaluation. The full list of KII informants is included in Appendix 4. Final Evaluation Report : MOCA Côte d'Ivoire 29 3.3 Data Collection Tools Quantitative Survey Tool The final questionnaire included two sets of questions: original Baseline questions and new Final Evaluation questions. To create a manageable questionnaire and acknowledge that less than three years have passed since Baseline, we excluded Baseline questions designed to provide general socio-demographic respondent information. Instead, new questions were introduced to the final questionnaire to capture the respondent’s evaluation of project activities and outcomes. The questionnaire was divided into eight thematic sections: 1. Identification and Location, 2. Farm Characteristics, 3. Production, Income, and Management, 4. Inputs (fertilizers, pesticides, tools), 5. Farming Practices, 6. Harvest and Post-harvest, 7. Other Actors (financial institutes, support with marketing), and 8. Evaluation of MOCA activities. The Quantitative Survey Tool is presented in Appendix 5. Qualitative FGDs Tools Conversation-based FGDs were conducted using classic FGDs methods, enabling participants to bring their insights and opinions and build on each other’s ideas. Discussion guides were designed for each type of group: farmer FGDs, women FGDs, and cooperative leaders FGDs. In this way, the tools were designed to address each group's unique characteristics, involvement with the MOCA project, and anticipated successes and challenges. The Qualitative FGD Tools are presented in Appendix 6. Qualitative KIIs Tools In-depth interviews are ideal for capturing sensitive information and individual attitudes and gaining feedback. Questions were designed to elicit attitudes around performance and identify successes, challenges, and recommendations for future programming. Interviewees were assured of confidentiality, asked for their consent, and were encouraged to provide candid responses in the interest of learning and future project development. Similar to the FGDs, unique semi-structured KII guides were designed for each type of respondent: cooperative management and farmers' organizations, project staff, banks, and financial institutes, government officials, and service and input suppliers. The Qualitative KII Tools are presented in Appendix 7. 3.4 Data Analysis Data analysis was conducted in several stages. First, qualitative and quantitative data collection were continuously reviewed to ensure data saturation and quality assurance during the field collection stage. Final Evaluation Report : MOCA Côte d'Ivoire 30 Next, both types of data were cleaned and prepared for analysis. FGDs and KIIs were transcribed and translated from audio files in French to documented English. Survey data was cleaned by checking for invalid numeric values, identifying outliers for numeric variables, applying uniform variable formats to maintain consistency, identifying ranges, and ensuring that missing data and not-applicable responses were correctly explained. Random data quality checks were also instituted using frequency counts, descriptive statistics, and analysis of missing variables for identifying any unusual patterns. Dedoose, a web application for mixed methods research, was used to code primary qualitative data and compile findings from project reports, external reports, and academic research papers. Codes were created to correspond to evaluation questions, and key findings were drawn from primary and secondary sources in response to those questions. The volume of evidence available to support the findings guided researchers’ decisions on what information to present as key conclusions in this report. Quantitative analysis of the databases was completed using SPSS quantitative data analysis software. For major data points, data was disaggregated by: • Respondent type (Primary beneficiary, Secondary beneficiary, and Control) • MOCA region (Abengourou, Daloa, and Soubre) • Gender The following calculations were conducted for major values of disaggregated data: • Mean (or median, when appropriate) • 95% confidence intervals (high and low) • Standard error and level of statistical precision • Size of non-missing observations and their percentage of total observations • Level of significance, set at 0.05 • Power of t-test (1-ß) The data analysis approach prioritized the integration of information across both primary data collected by TKG and comparison to Baseline data and project monitoring data and reports. Groups and individuals were compared with Baseline results to evaluate changes and trends. Last, a joint analysis of qualitative and quantitative data types was conducted to achieve triangulation of data. In the process of triangulation, two or more independent sources of data or data collection procedures (in this case, multiple qualitative and quantitative sources) employed in the study were used to support each other to ensure the reliability of the findings. 3.5 Ethical Considerations The study ensured that rigorous ethical standards were kept in all instances. Specifically, to ensure that the evaluation presents a clear and true reflection of how MOCA was implemented in Côte d’Ivoire, the following principles were maintained: No harm - Measures were taken when designing and implementing the evaluation tools to avoid causing emotional, physical, or economic harm to participants. Final Evaluation Report : MOCA Côte d'Ivoire 31 Transparency - Participants were clearly informed about the background and objectives of the study. Consent - Participants were informed about topics to be covered, the benefits and risks involved, and how the data will be used. Voluntary participation - Participants were informed that their participation is voluntary and that they have the right to stop the interview at any time or skip a question. Confidentiality - All information gathered was kept strictly confidential, and no names or other personal identifiers were recorded. Privacy - Reasonable precautions were taken to reduce the risk of privacy breaches and safeguard the confidentiality of data shared online. Withdrawal - Participants were informed about their right to withdraw at any point or retract their data after completing the study. Security - A secure communication platform for data collection was used. No sponsor influence - Program sponsors did not influence the study findings. 3.6 Evaluation Limitations The sensitive nature of evaluations and the global economic and health instability due to the COVID- 19 pandemic presented limitations for this evaluation. These limitations were taken into account during the analysis, including: Sample bias. While our sampling methodology aimed to minimize bias and provide a representative sample from the project participants and affiliates, much of the sampling was purposive, and sample sizes were limited due to COVID-19 restrictions. In most cases, FGD participants were selected based on availability and by the cooperative, as it was often difficult to find farmers. The farmer's survey sample underwent some necessary adaptations during field data collection due to the inability to locate some Baseline respondents. Collaboration resistance. The sample for the Final Evaluation included USDA managers, private sector managers, and government officials that were sometimes hard to reach for meetings. Despite recurring efforts, some of these informants remained reticent to share details and insights that may have been valuable to the evaluation. Social desirability bias. While the research staff was trained on and aware of ethics and the importance of reducing bias, they may not have been able to eliminate their influence on FGD and KII participants. Furthermore, there may also be some bias due to the influence of particularly charismatic or opinionated individuals on other participants during FGDs. The facilitators were trained to mitigate this bias, but there may have been some impact, nonetheless. Subjectivity bias. The data collection done by TKG represented respondents’ perspectives measured in a non-standardized way. While rigorous coding methodologies were used, this still left room for subjectivity in both the responses given and the analysis and interpretation of patterns in the data. Final Evaluation Report : MOCA Côte d'Ivoire 32 Baseline survey data quality. Results on the comparison of Baseline and Endline farmers' surveys are reliant on the quality of both datasets. While TKG is responsible for the Endline survey and can attest to its high quality and accuracy, our assumption that the Baseline data was collected using valid methodologies remain an assumption. Time restrictions and travel distances. Data collection is often conducted under tight time restrictions, and although much effort was put into designing an achievable work plan, long data collection days in addition to difficulties of traveling through muddy and often blocked village roads may cause temporary overload on the enumerators that may result in unforeseeable influence on the quality of the data collection. Crisis sensitivity. Shocks and crises can affect how people process and relay information. The COVID-19 effect may cause the mean data collected to be more representative of the respondents’ feelings during the time of crisis rather than during the entirety of the implementation period. Final Evaluation Report : MOCA Côte d'Ivoire 33 4 Findings This section describes the main results from the desk review, data collection, and data analysis conducted as part of the Final Evaluation of the MOCA project in Côte d’Ivoire. The findings are organized such that they directly respond to the evaluation questions as much as possible. Complete tables of all survey indicators and statistical analysis are presented in Appendix 8. 4.1 Relevance 4.1.1 To what extent has the project responded to the actual needs and interests of the target beneficiaries? The MOCA project aimed to tailor activities to the specific needs of beneficiaries the project worked with. Information from the project’s work plans, progress reports, and semi-annual reports describe how MOCA tailored activities to address the unique needs of each type of beneficiary. Interviews with program staff demonstrated how these adjustments were achieved. Some examples include: Farmer best agricultural practices: MOCA’s Baseline results indicated that a high number of farmers were impacted by plant disease, and especially the Black Pod disease12 . MOCA identified the need to enhance practices that effectively treat the Black Pod disease to reduce harvest losses and increase income for cocoa producers. MOCA’s strategy to prevent the disease focused on improving canopy management and removal of infected pods. This strategy is backed by research indicating that such approach is effective in treating the disease, and that turning away from a pesticide-centered practices reduces environmental and health hazards and is more suitable to smallholders’ financial limitation and cultural practices (refer to ‘Section 2.1.1 - Cocoa Production Challenges and Opportunities’ for more details). MOCA therefore decided to focus efforts on improving pruning practices to increase airflow and reduce humidity inside the canopy by adjusting canopy density and removing infected pods. The focus on pruning training combined with the provision of pruning tools was informed by the 2016 annual Cocoa Action report from the World Cocoa Foundation13 indicating that farmers that show willingness to apply canopy management practices often face financial hurdles in adopting the methodologies. MOCA staff listened to various stakeholders that indicated that without the provision of pruning tools, farmers will struggle with removing infected pods and adjusting canopy 12 MOCA’s Baseline study reported that “58% of the interviewed farmers report losses due to Black Pod Disease, the highest percentage mentioned by farmers when discussing production losses.” 13 https://www.worldcocoafoundation.org/wp-content/uploads/2018/07/2016-CocoaActionReport-English_WEB_10- 30.pdf Final Evaluation Report : MOCA Côte d'Ivoire 34 density. These insights, combined with the implication of the shortening of project timeframe, led the MOCA team to focus on pruning. This allowed the project to complete the training and provision of pruning tools within the timeframe of the project. Black Pod persists to be the most prevailing disease impacting cocoa plant, and at Endline, 62% of farmers reported that their cocoa has been impacted by Black Pod, more than any other disease. Cooperative capacity building: Prior to the actual implementation phase, MOCA engaged with cooperatives that expressed interest in collaborating with the project. Following, MOCA conducted a capacity assessment of each cooperative to evaluate its unique needs, current capacities, and interests. Based on these assessments, MOCA selected cooperatives to work with during the implementation phase. Next, project staff developed and implemented tailored capacity development plans for each cooperative. Capacity Building Plans were updated annually for each cooperative. For example, the third and final annual plan focused on finalizing continuous capacity development process. For some cooperatives, activities included improving cooperatives' accounting systems while for others, activities focused on training cooperative leadership on governance and management, or on youth involvement, or client approach and member services. The MOCA team also held one-on-one follow-up meetings with training participants to support them through implementing the newly acquired skills and knowledge and support their individual needs. During the COVID-19 pandemic, MOCA staff adjusted activities to cope with travel and meeting limitations, and converted to a remote session model. The project used these capacity assessments to also monitor and evaluate the progress of each cooperative. Women organizations: MOCA’s gender specialist assessed the unique needs of women entrepreneurs and cooperative leaders. By identifying and then tailoring activities to the unique needs of women in each farmer group, MOCA aimed to provide better training and support to women and help them become successful leaders in the cocoa sector. Out of the eight grantees with which MOCA signed grant agreements, three were women’s groups14 . With the women’s groups, the MOCA team worked to formalize and improve the operation’s efficiency and increase women’s income. These activities were tailored to the specific work focus of each group. Access to Inputs: One of MOCA’s highest priority was to increase farmers’ access to inputs. During the LoP, MOCA established four Farm Service Centres (FCS) in the vicinity of MOCA targets cooperatives to improve the quality of the provision of inputs and input services. 14 One of the three agreement with women organizations was terminated early due to the groups’ failure to develop and submit a revised implementation plan with revised deadlines, regardless of the MOCA team supporting efforts. Final Evaluation Report : MOCA Côte d'Ivoire 35 To leverage the value of the public-private sector investment, MOCA facilitated an additional 900,000 USD investment from GIZ in the establishment of another five FSC in the MOCA intervention areas, and partnered with Olam, GIZ and Jacobs Foundation to establish five Service Provision Units (SPU) and nurseries, and with several exporters and chocolatiers who also provide of inputs to farmers. MOCA procured one truck under its grants facility and facilitated the required financing for a cooperative to purchase its own truck. In addition, MOCA´s grants facility provided cooperatives and other actors operating in the MOCA intervention areas to produce economically and technically viable grants proposals which were intended to meet any unmet needs. The number of grants and volume of the grants fund was limited and not all proposals were awarded. One of MOCA’s strongest strategies was to assure that all cooperatives and farmers participating in the project are matched with a private or public partner capable of providing inputs and training after project termination. These include organizations like Olam, ECCOKIM, and Blommer that can provide GAP training and coaching to farmers, and local input distributor like Callivoire, that can continue to support the FSCs. Since the project´s end, some of these partnerships already have shown their value. For example, Callivoire is continuing to support the FSCs and has launched a promotion campaign for the FSCs after MOCA has ended. Similarly, Project Hope and Fairness has been supporting SOCOPLAN with the reception and installation of the chocolate production equipment after project completion. Professional relationships between finance institutions such as Baobab, Advance and OikoCredit and MOCA-supported cooperatives continued after the end of the close out of the project. KIIs and FGDs verified that MOCA was able to identify the current and actual needs of the target beneficiaries and respond to these needs through training and support. In FGDs, farmers said that MOCA responded to their needs and interests. In all six focus group discussions with farmers and two of the three KIIs (out of a total of 61 farmers interviewed), training for pruning and the distribution of pruning tools was reported as the most important contribution to their needs. This included reports from farmers that were members of KAPATCHIVA, FAHO, ECAM, SOOCOABAS, and CANAAN cooperatives. These farmers said that training on pruning was the most impactful and efficient MOCA activity, teaching them how to trim down trees. The distribution of saws was highly appreciated. Training on harvesting and post-harvesting techniques was also identified as important to farmers. Farmers were proud to cite the learned trainings and list the order of activities they now follow for harvesting, shelling, fermenting, and drying. Nearly all cooperative managers said that MOCA was able to respond to their needs, especially around accountant training and leadership training. This included discussions with cooperative managers in two FGDs and eight KIIs (a total of 18 cooperative managers interviewed). One cooperative manager, for example, said that their cooperative management has taken a big step forward in terms of administration, financial planning, and accounting procedures with the support provided by MOCA. Managers in one of the FGD explained that before MOCA, they needed to hire an Final Evaluation Report : MOCA Côte d'Ivoire 36 external accountant and that today, they are conducting these same procedures internally, therefore MOCA activities where relevant to their needs. In discussions with two women farmers organizations, the organization managers said they have acquired a new understanding of the importance of gender equality in cooperative leadership. They also said they benefited from the project because it allowed them to develop as entrepreneurs and become more independent and confident in their capacities. As with any development project, MOCA was able to address some of the needs beneficiaries had, but not all. Below are some of the unmet needs raised more frequently in FGDs and KIIs. These needs are not directly related to MOCA goals and activities, but were pointed out by beneficiaries as being highly relevant to them: • Inputs. Across the board, all farmers in FGD and KIIs indicated that they still have needs for additional inputs: fertilizers, pesticides, tarps for drying beans, farm tools such as boots, wheelbarrows, bean bags, and marketing and sales inputs such as bags for cocoa beans, balances to weigh the bags, and vehicles to deliver the cocoa to the cooperatives or markets. This does not come as a surprise, and MOCA is not expected to address all the needs of farmers. o In two FGDs, a few farmers said they could not carry out the training they received on pruning because they were “promised tools but have not received any yet”. These situations may have occurred when farmers assumed they will receive saws but did not meet the basic MOCA conditions to receive pruning tool. These conditions require that farmers benefit from the project activity, and that their cocoa plot is at least 2 km away from forests and protected areas. In addition, as explained by one cooperative manager, because of MOCA’s policy to distribute tools directly to approved farmers only and not to their representatives, some farmers may have missed distribution days and were disappointed they did not receive pruning tools. • Continuous support. Both farmers and cooperative leaders said they need the project life time to be longer. Nearly all informants spoke of their need for further support to reinforce the newly acquired information, and their desire to continue the improvement process that they or their cooperatives have started. One cooperative manager said that MOCA has led them through several iterations in which activities and trainings were re-adjusted as their cooperative capacity has evolves, and that their cooperative would benefit from several more iterations. • Gender equality. In FGD with farmers and KIIs with cooperative women leaders, there were a few indications that MOCA did not address the specific needs of women. It is important to note that MOCA did not have a gender objective and worked only to enhance women entrepreneurship and leadership and through grant activities. Yet, this is a reoccurring topic raised in several instances. In the women’s FGD with 12 women participants, most women farmers said they could not maximize what they have learned because of limitations unique to women. Primarily, this refers to the limited participation of women in the cocoa sector, Final Evaluation Report : MOCA Côte d'Ivoire 37 thus any training for gender needs to be focused first and foremost on strengthening their decision-making capacity, and improving their access to land, training, and inputs. In Côte d’Ivoire, women are mainly involved in collecting and transporting the harvested pods from the fields and drying and sorting the beans15. In the women’s FGD, informants said they would’ve benefited from more training and inputs for these specific tasks16 . Further, possibly because women are considered as merely “helping” the men, only a small number of women farmers participated in MOCA’s activities (17% of the individuals who have received short-term agriculture productivity training were women, Table 11). In the women’s FGD, informants expressed their disappointment from not receiving training and tools as the men did17 . Two male cooperative managers said that production would increase if women would’ve received more training on GAP. Two male farmers said they want their wives to receive training to help them become better producers. It is to note that the MOCA project operated a limited Gender Equality Committee with cooperative members (both women and men) from five cooperatives in Daloa and Soubré to support the training for tools and strategies for active participation of women in the cooperative activities. 4.1.2 To what extent does the project integrate or strengthen the national cocoa strategy of Côte d’Ivoire? The Côte d’Ivoire government has been supporting the cultivation of cocoa crops through its Quantity, Quality, Growth (Quantité, Qualité, Croissance, 2QC) program. The strategy aims to support the sustainable cultivation of cocoa and promote the socio￾economic well-being of farmers in the cocoa sector. To implement this strategy, CCC has launched several initiatives that support the productivity of cocoa farms, including training farmers in agricultural best practices, distributing seed, providing plant protection products, and distributing pesticides. As part of the national strategy, Côte d'Ivoire is also interested in increasing income for farmers without increasing cocoa production by improving cocoa quality and flavor. The strategy also works to improve farmers’ financial inclusion rate, which is currently low. MOCA has worked to design project activities to integrate and strengthen these national cocoa strategies. The project management has been in direct and continuous contact with CCC throughout the life of the project to inform the government about project goals and progress and ensure that the government supports the strategic choices of the project. 15 Cocoa Life in Côte-d’Ivoire Gender Assessment Report, 2014, CARE International. 16 Excerpts from the women’s’ FGD: “It is the men that produce cocoa. The women help with the gathering and the harvest”; “They can do trainings on how to gather the cocoa especially for the women. They should teach the women”; “(we need) trainings about gathering, harvesting”; “Like she has said, we mainly need the tarpaulins. You know, the cocoas are on the floor. If there are no tarpaulins, they would not be protected when we dry them”. 17 Excerpts from the women’s’ FGD: “It was the men that they gave machetes, and so on. The women here did not really have trainings. We saw nothing. They gave some tools to the men, and they taught some ways to do some things”; “They did not come to us the women, they went to the men. It was with the men that they worked”; “We are saying that these questions are not for us, but for the men. The people came but they came to our husbands”; “We are trying to think as it was not necessarily the women that were worked with”. Final Evaluation Report : MOCA Côte d'Ivoire 38 MOCA’s strategies, activities, and outcomes have been contributing to the national cocoa strategy of Côte d’Ivoire in several ways, as discussed next. To improve the productivity of the cocoa sector, MOCA worked to train farmers on the prevention of cocoa plant diseases, supported the improvement of input supply, and supported the development of farm business plans. The project also worked to improve farmers’ access to credit and finance by connecting farmers with financial service providers. With cooperatives, the project worked to improve their organizations’ professionalism by training and supporting management capacity building, business plan development, access to finance, and organizing exchange visits to other cooperatives and the Quality Flavor lab in Ghana. Most of the training for improved productivity of the cocoa sector were designed and supported by MOCA, but not directly implemented by project staff. Instead of working directly with farmers, MOCA strengthened the capacities of cooperatives and ANADER to train the farmers themselves. This strategy also aligns with the national policy encouraging NGOs and productivity projects to abstain from directly training producers. MOCA has also worked to improve the quality of cocoa and the quality of cocoa taste. This supported the national cocoa strategy to increase income for farmers without increasing cocoa production. To achieve this strategic goal, MOCA worked with multiple stakeholders of the cocoa sector. With farmers, the project worked to improve production techniques and post-harvest handling that impact cocoa quality and flavor. With cooperatives, the project created and supported linkages with US chocolate makers. MOCA’s work to improve access to credit helped cooperatives support farmers that expressed interest in producing high-quality cocoa and quality flavor cocoa. MOCA collaborated with the USAID-funded African Cocoa Initiative II project implemented by the World Cocoa Foundation to support the newly established CNRA’s Quality Flavor lab. The project used the lab to produce cocoa tasting samples and flavor profiles which allowed cooperatives to observe first-hand how production practices impact the quality of cocoa liquor taste and provided high-quality samples to potential international buyers. MOCA collaborated with the CCC and the African Cocoa Initiative II project to initiate a national quality flavor conference in Côte d’Ivoire in 2020. Unfortunately, the initiative came to a halt due to the COVID-19 restrictions. Governmental representatives, chocolate makers, and financial institution representatives approved that MOCA’s work was well aligned with Côte d’Ivoire’s cocoa strategy. More than a few of these high-level representatives also spoke of the need for better governmental enforcement for high prices on cocoa trade. This was identified as an essential step toward adopting quality cocoa and quality cocoa flavor production and should be supported in future projects. These results show that the project has been well integrated with the national cocoa strategy of Côte d’Ivoire. The extent to which the project has been able to achieve these strategic goals is discussed next. Final Evaluation Report : MOCA Côte d'Ivoire 39 4.2 Impact 4.2.1 Access to inputs and input services Based on the farmers survey results, application rates of insecticides were slightly higher for Primary beneficiaries and Control respondents than for Secondary beneficiaries (Figure 2). These results are lower than those reported at Baseline. For those that do use insecticides, the reported area of cocoa farms on which insecticides were applied was also smaller at Endline than at Baseline, showing a decrease from 4.5 Ha in 2017-2018 to 4.2 Ha in 2020-2021 for MOCA Primary beneficiaries, and from 4.4 Ha at Baseline to 4.0 Ha at Endline for Control farmers (Figure 2). Fungicides were used by only a third of survey respondents at Endline. This, however, represents an increase from Baseline, during which only 23% of respondents reported Objective 1.2: Increased Use of Improved Agricultural Techniques and Technologies Indicator: Number of individuals who have applied new techniques or technologies as result of USDA assistance. On the basis of the survey carried out among 664 farmer beneficiaries, out of the 664 individuals, 631 (95%) applied at least one technique or technology that has been the subject of MOCA training and coaching. Gender: Male: The survey of 664 farmers included 602 male farmers among whom 572 (95%) responded that they applied improved agricultural management practices through MOCA. Gender: Female. The survey of 664 farmers included 62 female farmers among whom 59 (96%) responded that they applied improved agricultural management practices through MOCA. Objective 1.2: Increased Use of Improved Agricultural Techniques and Technologies Indicator: Number of hectares of land under improved techniques or technologies as a result of USDA assistance. The farmers survey asked producers if they applied new techniques and technologies and if yes, on how many hectares. On the basis of the survey carried out among 664 farmer beneficiaries, the average number of hectares per technique is as follows: • Pest Management: 4.17 Hectare. • Soil Related Fertility Conservation: 3.76 Hectare. • Pruning: 3.68 Hectare. Final Evaluation Report : MOCA Côte d'Ivoire 40 they use fungicides. The observed increase was larger for MOCA farmers than for Control farmers. For those that do use fungicides, the reported area of cocoa farms on which fungicides were applied was also larger than at Baseline, but the increase was small – from 4.1 Ha at Baseline to 4.2 Ha at Endline for Primary beneficiaries, and from 4.1 Ha at Baseline to 4.4 Ha at Endline for Secondary beneficiaries (Figure 2). Figure 2: Application of Pesticides (Insecticides and Fungicides) on Cocoa Farms Variable trends were also observed for the application of fertilizers. While a general decrease was observed from Baseline to Endline in the application of chemical fertilizers, an increase was observed in the application of organic fertilizers (Figure 3). For those that do use chemical fertilizers, the area on which it has been applied slightly decreased for MOCA farmers and increased for Control framers. The reported use of organic fertilizers has increased dramatically, from 9% of farmers at Baseline to 26% for Primary beneficiaries and to ~40% for Secondary beneficiaries and Control farmers (Figure 3). For those that do use organic fertilizers, the reported area on which it has been applied has increased from Baseline to Endline, from 2.8 Ha to 3.1 Ha for Primary beneficiaries, from 2.8 Ha to 3.2 Ha for Secondary beneficiaries, and from 3.6 Ha to 3.8 Ha for Control farmers. Final Evaluation Report : MOCA Côte d'Ivoire 41 Figure 3: Application of Fertilizers (Chemical and Organic) on Cocoa Farms Some improvement has been reported on the application of pruning on cocoa farms for MOCA farmers (Figure 4). A 4% increase in the application of pruning was recorded for MOCA farmers from Baseline to Endline, while a 4% decrease was recorder for Control farmers. For those that do practice pruning, the area of the cocoa farm on which pruning was applied was also larger for MOCA farmers (3.7 Ha) than for Control farmers (3.4 Ha) (Figure 3). Figure 4: Application of Pruning on Cocoa Farms Nearly all survey respondents reported that they practice post-harvesting techniques - shelling, fermentation, and drying. This does not come as a surprise for a population of cocoa farmers. Nonetheless, there has been an increase in the number of respondents reporting that they practice post-harvesting techniques to nearly 100% of respondents at Endline for all groups (Figure 5). Final Evaluation Report : MOCA Côte d'Ivoire 42 Figure 5: The practice of post-harvesting techniques Many farmers shared information of how they changed their agricultural practices as a result of MOCA’s training and support. Farmers said that MOCA has enabled them to control pest diseases and improve production: “Before the arrival of MOCA, we weeded with herbicides, but MOCA taught us to do it with machete. I mean, it made our fields look healthy. We were told not to dry the cocoa on the tarpaulin, neither on the tar nor on the terrace because it is made of cement. We were also taught how to store cocoa once it is dry. Also, no Child Labor too. We were told to keep the pesticides safe in secure places to keep them out of the reach of children and out of the bedroom we sleep in. Before treating the farm, ripe pods must be removed, and to clean the field before treating them.” Farmer, FGD In all FGDs, farmers told how MOCA’s training taught them to trim trees correctly and detect diseases in cocoa trees. Many spoke of how they apply the learnings on the use of fertilizers and pesticides. They repeated the knowledge they acquired and listed the changes in the ways they now conduct pest control and post-harvesting techniques on their cocoa farms. As mentioned, training on pruning and the distribution of pruning tools were noted as the most important contribution of MOCA to the farmers’ improved agricultural techniques and technologies. 4.2.2 Access to finance and financial services Access to credit remains one of the key constraints in the cocoa sector, and MOCA worked in different ways to provide access to financial services to farmers. MOCA worked to connect financing institutions with producers through cooperatives and farmer organizations. MOCA facilitated connections between farmers, directly or indirectly via the cooperative, to banks, credit companies, mobile bank companies, and microlending financial institutes, to increased access to credit. MOCA also improved farmers and Final Evaluation Report : MOCA Côte d'Ivoire 43 cooperative access to insurance products. MOCA worked to facilitate access to finance on two levels: At the producers’ level, MOCA organized events in which financial institutes presented their products, offered training, and was able to directly connect with producers. At the cooperative level, MOCA worked with cooperatives to increase input to finance and investments, therefore impacting their members, which are MOCA’s Secondary beneficiaries. Within the complete portfolio of financial services MOCA provided to its beneficiaries, a total of 7,753 project beneficiaries, representing 339% of the project´s revised target of 2,280 individuals, have received at least one financial service (Table 15). Mobile Bank Accounts. Based on survey results, approximately three-fourth of survey respondents currently possess a mobile money account at Endline. For MOCA respondents, this represents an increase since Baseline, while a small decrease in the number of Control farmers reported that they or any other family member has a mobile bank account (Table 10). Traditional Bank Accounts. On average, only 20% of Primary beneficiaries and 16% of Secondary beneficiaries had bank accounts (Table 10). Although numbers are low, this indicates a large improvement since the Baseline. Based on these reports, more MOCA farmers acquired new bank accounts than Control farmers: the number of MOCA farmers that have bank accounts has more than doubled since Baseline, whereas the number of Control farmers that have bank accounts increased by 1.5. These values and trends are similar for microfinance accounts. Table 10: Possession of financial accounts (% of individuals) Baseline: MOCA Baseline: Control Endline: Primary Endline: Secondary Endline: Control Bank Account 8 11 20 16 18 Microfinance account 8 12 19 16 12 Mobile money account 70 72 76 74 71 Loans. The number of respondents that have received a loan over the last 2-3 years has decreased for all respondents, and the number of MOCA farmers that have received a loan was lower than the number of Control farmers (Table 11). A similar situation was reported at Baseline. Out of the 117 survey respondents who received a loan, the average amount of the largest loan received over the last two to three years was 385,000 CFA for Primary beneficiaries, 229,000 CFA for Secondary beneficiaries, and 422,000 CFA for Control farmers. This represents an increase in the amount of loan received since Baseline, for which the average maximum loan amount was 192,000 CFA (Table 12). Final Evaluation Report : MOCA Côte d'Ivoire 44 Table 11: Loans received over last 2-3 years for self or other household member (% of individuals) Baseline: MOCA Baseline: Control Endline: Primary Endline: Secondary Endline: Control Mean 22 28 8 9.4 11 Table 12: Largest amount of loan received (CFA francs) Baseline: MOCA Baseline: Control Endline: Primary Endline: Secondary Endline: Control Mean 191,501 193,121 385,396 228,792 422,173 FGDs confirmed that access to financial services was still a major barrier to farmers. In multiple FGD, farmers said MOCA brought bank representatives to speak with them about savings and loans. This was perceived as a positive event that motivated them, yet those farmers that attended the events with banks said they were unable to put money in the bank because they do not have sufficient funds or have previous debts. In all FGDs and KIIs with farmers (a total of 61 interviewed farmers), not one has disclosed that they personally opened a bank account or received a loan through MOCA activities. Two farmers from two different cooperatives said in general that more people are opening bank accounts and using micro-credit as a result of MOCA’s activities. “I brought the supportive document of the bank training with me, but unfortunately, I have not been able to put that training into practice. I did not get enough money, and my disbursements were way beyond my receipts.” Farmer, FGD In interviews with cooperative managers and farmers’ organizations, managers also confirmed that MOCA helped them connect with financial institutes, including banks, microfinance organizations, and insurance companies. Six of the eight cooperative managers interviewed said they were able to utilize the new connections to improve financial services for their cooperatives. Three managers tied these achievements to the work MOCA did on cooperative capacity building and business development at the organizational level: “…because when you manage well there is no problem. All the companies want to take you to work. For example, at the start of the campaign Zamacom gave us funding, no later than December 15 we have already reimbursed, all the more reason that they have trained us for good governance.” Cooperative manager, KII One cooperative manager said that MOCA’s trainings for cooperative capacity building and business development was critical not only to open accounts and receive loans, but also to be able to return them, because “...if you have the finances and you don't have the skills to carry out the project that I am putting together, it will fail”. Even if the volume of achieved bank accounts was not large, the importance of this activity is in its ability to initiate the change. Opening a bank account is a genuine Final Evaluation Report : MOCA Côte d'Ivoire 45 challenge to farmers. Accessibility to banks in rural areas is low and, in many cases, nonexistent. To open bank accounts, initial funds and a formal ID are required, and many framers do not possess one or both. Further, bank accounts cost money to maintain, and often, farmers are unable or reluctant to spend money on maintaining a bank account, as there are more urgent daily needs such as food, school fees or healthcare. While this issue is outside of MOCA´s mandate and capacity, MOCA worked to organized events in which bank representative arrive to the villages to connect with potential clients. MOCA also looked for other ways to support farmers such as access to finance for farmers through cooperatives. In KIIs with cooperative managers, many noted the pivotal role MOCA took to link cooperatives and farmers with banks: “The operational scope of MOCA has been to link banks and people. This has allowed rural populations to have a wider scope of what they can do. This broadening of thoughts has been an asset because after MOCA, the beneficiaries come to do banking operations. In addition, it allows them to improve their quality of life.” Cooperative manager, KII In interviews, bank and credit organization representatives indicated that MOCA has allowed them to develop good collaboration with the farmers. They said that MOCA has been the only organization that has supported them in creating collaboration with farmers so far. They have observed a shift in the population’s perspective toward banking activity. While these perspectives may have not yet translated into actual behavioral shifts leading farmers to open bank accounts, in KIIs with cooperative leaders and in FGSs with farmers, they say that they have gained more trust in banks and are more assured that banks will respect their confidentiality, and that this shift has been made possible thanks to the work of MOCA with farmers before the arrival of the bank representatives in the communities. MOCA’s defined goals and the need to meet the target goals have helped financial institutes work harder to meet these goals as well, and as a result, they have opened more accounts and distributed more loans to cooperatives, allowing them to increase export and build capacity. On the challenge side, these representatives indicate that some farmers had outdated documents or did not have any documents at all. Consequently, they were not able to open an account. This validates information received from farmers and a few cooperative managers, saying that the administrative burden makes it more difficult for farmers, cooperatives, and farmers’ organizations to open accounts and receive loans. 4.2.3 Access to markets Marketing of the cocoa beans was primarily done through the cooperatives for all survey respondents. Control respondents marketed more of their produce through private trackers (21%), while MOCA farmers rarely did so (2% of farmers). When selling cocoa, most farmers received their payment in cash immediately after the sale (56% of MOCA farmers and 60% of Control farmers, Table 13) or through deferred cash payment (~41% for all survey respondents). Final Evaluation Report : MOCA Côte d'Ivoire 46 Table 13: Primary payment method All Primary Beneficiary Secondary Beneficiary Control Chi-square test (significant) Cash payment on the spot 56.6 55.6 56.4 60.3 .062 Deferred cash payment 40.8 42.4 39.2 38.8 Mobile money transfer .2 .3 0 0 Bank transfer .7 .6 .9 .5 Other 1.7 1.1 3.4 .5 n 1310 665 436 209 Twice as many MOCA farmers as Control farmers reported they have received a bonus on premium cocoa sales (Figure 6). For those farmers that have reported that they received a bonus in 2020-2021 from the sales of premium beans, the average bonus was larger for MOCA farmers than for Control farmers (Figure 7). This bonus still represents a small portion of the farmers’ annual income, 4-5%. Figure 6: Farmers receiving bonus on premium Final Evaluation Report : MOCA Côte d'Ivoire 47 Figure 7: Size of premium Feedback on MOCA’s activities to train and support for improved access to markets was provided mainly by cooperative managers. These managers mentioned a few ways in which MOCA assisted marketing efforts for the benefit of their cooperatives. First, MOCA supported export and trade by connecting them with market players and especially chocolate producers. Second, MOCA arranged field visits with partners and sister cooperatives to enlarge marketing opportunities further. Third, by connecting cooperatives and farmers with financial institutes, funds were more readily available, supporting the export process. Last, MOCA’s work to improve producers’ agricultural best practices and improve production helped increase trade and export. 4.2.4 Inputs Cocoa production is carried out during two distinct seasons in Côte d’Ivoire, ‘la grande traite,’ or the main season, extending from October to March, and ‘la petite traite,’ or small season, extending from April to September. Based on Endline survey values, Objective 1: Increased Agricultural Productivity Indicator: Volume of commodities (metric tons) sold by project beneficiaries. On the basis of the survey carried out among 664 farmer beneficiaries, out of the 664 individuals, 563 answered the question about the volume of commodities sold: The average volume obtained per producer is 2,503 kg for farmers who are direct beneficiaries of training and coaching on good agricultural practices. • Type of producer/firm: Producer – smallholder. On average, smallholders sold 1,919 kg cocoa for money in 2020/2021 for the 419 smallholder respondents that answered this question. • Type of producer/firm: Producer – non-smallholder. On average, non￾smallholder sold 4,407 kg cocoa for money in 2020/2021 for the 134 non-smallholder respondents that answered this question. Final Evaluation Report : MOCA Côte d'Ivoire 48 approximately two-thirds of the total annual yields were produced during the large season. Table 14: Cocoa production and yields Baseline MOCA Baseline Control Endline Primary Endline Secondary Endline Control Total cocoa produced (kg) 1,584 1,384 2,534 2,057 2,401 Cocoa yield 414 360 584 528 558 The average annual cocoa production for Primary beneficiaries at Endline was 2,534 kg. Primary beneficiaries were the highest producers and Secondary beneficiaries were the least productive (Table 14). MOCA farmers produced more than Control farmers at Baseline as well. These production rates show a large increase from Baseline for both MOCA and Control farmers. The increase was larger for Control farmers (a 74% increase) than for Primary beneficiaries (a 60% increase). Cocoa yields (kg of cocoa produced for each Ha used for cocoa cultivation) also showed an increase since Baseline, and Primary beneficiaries where the most efficient producers, with an average yield of 583 kg/Ha (Table 14). MOCA farmers reported on a higher yield than Control farmers at Baseline as well, and the increase in yields since Baseline was larger for Control farmers (a 55% increase) than for Primary beneficiaries (a 41% increase in yield). This analysis, which compares Baseline and Endline survey results for MOCA and Control farmers, indicates that MOCA did not bring a significant improvement in cocoa production and yields. We associate the similarity or even larger improvement in production for Control farmers with the knowledge that Control farmers receive assistance as well: based on survey responds, 74% of MOCA Direct beneficiaries and 64% of Control farmers receive assistance for pest control, 85% of Direct beneficiaries and 66% of Control farmers receive assistance for improved harvest techniques, and 77% of Direct beneficiaries and 64% of Control farmers receive assistance with financial services. Further, if the assistance delivered to Control farmers is through programs and projects that have initiated prior to MOCA, it is reasonable that the increase they produce in cocoa production is larger, as changes in agriculture productivity occur after implementation of several seasons of improved agricultural practices. In FGD’s, farmers have often reported they have observed an increase in cocoa production and have related this increase to the training and support received by MOCA: “We have just received the saws, so we cannot comment on the production. But considering the multiple pruning we have done, we believe that our fields have changed and there will be a good production.” Farmer, FGD Final Evaluation Report : MOCA Côte d'Ivoire 49 “This [training] has developed my production. In putting into practice what I learned from the training, my performance has changed. In the past, I got one bag and half of cacao nibs, but this year I have got 2 bags and half of cocoa nibs because of the good agricultural methods I put into practice. My performance has risen.” Farmer, FGD Cooperative managers have also said that they have observed an improvement in the volume of cocoa production as an impact of MOCA activities. One manager noted that changing farmers' practices is a difficult and slow practice, and therefore they expect the change to be more observant in the forthcoming years. 4.3 Effectiveness 4.3.1 To what extent have the expected results of the project been achieved? USDA’s food assistance projects must establish indicator Baselines and targets to measure performance regularly. Initially, indicator Baselines and targets are established in the project proposal and are then finalized following the approval of the Baseline survey report18. As noted earlier, for MOCA, the approval of the Baseline survey and the establishment of final targets were finalized more than a year into project initiation due to delays in monetizing. MOCA received an amendment to their agreement in September 2020 to amend targets to the shortened life of the project (LoP). As targets of previous years could not be retroactively changed, 2021 targets were lower than 2020 targets for some indicators (Table 15). Based on internal monitoring data, MOCA has achieved the expected results and beyond. Most LoP targets have been largely overachieved. For example, MOCA’s target for Primary beneficiaries was 9,200 individuals, but the project ended working with over 13,000 beneficiaries. The number of loans distributed as a result of the project was nearly three times that of the target LoP, and their total value were more than five times that of the target value. And MOCA was able to leverage nearly 2.5 million USD of the targeted 936,000 USD from new public and private sector investment. Only three performance indicators underachieved LoP targets: • Number of full-time equivalent jobs created with USDA assistance (85% of target). Grants were foreseen as one of the main contributors to this indicator. Due to the budget shortfall, MOCA reduced the number of grants it awarded, resulting in lesser jobs created then foreseen. However, MOCA created 146 new full and part-time positions/jobs that are ongoing for the majority, of which 32% of the beneficiaries are women. These new full and part-time positions are employed by the five established Service Provision Units (128 young individuals) set up through the joint initiative with OLAM, GIZ, and the Jacobs Foundation; cooperative staff (22 individuals), and the Farm Service Centers (16 ful-time 18 Food Assistance Indicators and Definitions, Foreign Agricultural Service – Food Assistance Division United States Department of Agriculture, February 2019. Final Evaluation Report : MOCA Côte d'Ivoire 50 individuals. While this 85% achievement was measured in June 2021, it is likely that the 100% target was reached by the end of September 2021, as cooperatives and FSCs continued to create more positions. • Number of individuals who have applied improved farm management practices (e.g., governance, administration, financial management) as a result of USDA assistance (48% of target). The main activity to achieve the target was the development of Farm Business Plans for farmers to improve their cacao plantation management. ANADER, the national Ivoirian extension agency, was MOCA´s partner in implementing the necessary training and establishing the business plans. Due to the budget shortfall, MOCA only signed one sub-contract with the three regional ANADER offices, while two sub-contracts were initially foreseen. • Number of private enterprises, producers’ organizations, women’s groups, trade and business associations that have applied improved techniques and technologies as result of USDA assistance (95% of LoP target). Due to the budget shortfall, MOCA had to cancel two grants (SOCABS and Inclusive Guarantee) which would have assure 100% achievement of this indicator. Table 15: MOCA FFPr Performance Indicators: Targets and Life of Project (LoP) Values Indicator LoP LoP Target Achievement Value of sales by project beneficiaries (USD) $ 26,812,365 $ 15,430,285 174% Types of producer/firm: Producer - smallholder $ 16,903,188 $ 10,338,291 164% Types of producer/firm: Producer - non-smallholder $ 9,909,177 $ 5,091,994 195% Volume of commodities (metric tons) sold by project beneficiaries (MT) 15,373 13,848 111% Types of producer/firm: Producer - smallholder 8,608 9,278 93% Types of producer/firm: Producer - non-smallholder 6,764 4,570.00 148% Number of full-time equivalent (FTE) jobs created with USDA assistance 44 52 85% Disaggregate (Gender): Male 30 37 81% Disaggregate (Gender): Female 14 15 93% Numbers of individuals who have received short￾term agricultural sector productivity or food security training as a result of USDA assistance 9,043 9,000 100% Gender: Male 7,495 7,200 104% Gender: Female 1,548 1,800 86% Final Evaluation Report : MOCA Côte d'Ivoire 51 Duration: New 3,914 9,000 43% Duration: Continuing 5,129 7,203 71% Type of Individual: Producer 8,858 8,810 101% Type of Individual: People in Firms 185 160 116% Type of Individual: People in Government _ 30 0% Number of individuals who have benefitted directly as a result of USDA assistance 13,146 9,200 143% Gender: Male 11,069 7,360 150% Gender: Female 2,077 1,840 113% Duration: New 3,147 700 450% Duration: Continuing 9,999 8,500 118% Number of individuals who have benefitted indirectly as a result of USDA assistance 78,876 55,200 143% Numbers of individuals who have applied new techniques or technologies as result of USDA assistance 6,183 6,000 103% Gender: Male 4,866 4,800 101% Gender: Female 1,318 1,200 110% Duration: New 4,594 6,000 77% Duration: Continuing 1,590 4,000 40% Type of Technique/Technology: Pest Management 85 1,000 9% Type of Technique/Technology: Soil Related Fertility Conservation 695 _ Type of Technique/Technology: Climate Mitigation or Adaptation 292 500 58% Type of Technique/Technology: Pruning (GAP) 4,849 6,000 81% Number of hectares of land under improved techniques or technologies as a result of USDA assistance (FFPr1) 15,026 12,000 125% Duration: New 8,266 12,000 69% Duration: Continuing 6,760 9,000 75% Type of Technique/Technology: Pest Management 136 4,700 3% Final Evaluation Report : MOCA Côte d'Ivoire 52 Type of Technique/Technology: Soil Related Fertility Conservation 2,780 _ Type of Technique/Technology: Climate Mitigation or Adaptation 1,167 500 233% Type of Technique/Technology: Pruning (GAP) 9,699 12,000 81% Following, we present the results of the farmers’ survey for major indicators at Endline, and present information provided by project beneficiaries and partners in FGDs and KIIs on the extent to which expected results of the project have been achieved. Note that the indicators ‘Access to improved inputs and input services’, ‘Access to finance and financial services’, and ‘Access to markets, and Production’ have been presented and their impact and effectiveness discussed in the above Section 4.2. Final Evaluation Report : MOCA Côte d'Ivoire 53 4.3.1.1 Income from cocoa Sales The average total annual income from cocoa sales for all survey respondents in 2020- 2021 was 1.98 million CFA based on survey respondents' reports. Reported income was largest for Primary beneficiaries – 2.24 million CFA, an income 18% higher than that reported by Control farmers (Table 16). All respondent types increased their income since Baseline. MOCA farmers' income increased by 71% since Baseline, and Control farmers' income increased by 76%. At Baseline, MOCA farmers made 21% more than Control farmers, while at Endline, they made only 18% more. Objective 2.1: Increased Value Added to Post-production Agricultural Products Indicator: Value of sales of cocoa products by project beneficiaries. Survey Question: What is the revenue from cocoa sales for the entire production (in CFA)? On the basis of the survey carried out among 664 farmer beneficiaries, the average revenue from cocoa sales for the entire production was 2,239,565 CFA which corresponds to 4,031 USD for the 517 respondents that answered this question. • Type of producer/firm: Producer – smallholder. On average, smallholder producers obtained 1,721,057 CFA, which corresponds to 3,06 USD for the 369 smallholder respondents that answered this question. • Type of producer/firm: Producer – non-smallholder. On average, non￾smallholder producers obtained 3,903,216 CFA, which corresponds to 7,026 USD for the 120 non-smallholder respondents that answered this question. Survey Question: What is the value of the certification prime received? For this question, 330 respondents answered that they received a premium (1 out of 2 producers sampled). On the basis of the survey carried out among 664 farmer beneficiaries, the average premium received was 76,857 CFA or 137 USD in certification premium. • Type of producer/firm: Producer – smallholder. On average, smallholder producers received 62,610 CFA from sales of premium, which corresponds to 112 USD for the 246 smallholder respondents that answered this question. • Type of producer/firm: Producer – non-smallholder. On average, non￾smallholder producers obtained 127,565 CFA from sales of premium, which corresponds to 227 USD for the 65 non-smallholder respondents that answered this question. Final Evaluation Report : MOCA Côte d'Ivoire 54 Table 16: Cocoa income generated from sales Baesline: MOCA Baseline: Control Endline: Primary Endline: Secondary Endline: Control Income generated from the sale of cocoa (CFA) 1,304,980 1,077,733 2,239,656 1,693,614 1,898,251 Reported income values were similar to those calculated based on average produce sold (kg) times the current reported price received for most cocoa (CFA), confirming the income values reported by survey respondents Table 17. Based on these values, the production income using a 0.0018 exchange rate from CFA to USD was 1.58 USD/kg for Primary beneficiaries, higher than the 1.42 USD/kg for Control farmers. Table 17: Cocoa production income Endline: Primary Endline: Secondary Endline: Control Anova test (significant) Standard Error Sell for money (kg) 2,494 1,892 2,357 0.000 2,151 Current price received for most of cocoa (CFA) 857 877 842 0.0135 151 Calculated income generated from the sale of cocoa (CFA) 2,137,215 1,659,630 1,984,594 0.0001 1,800,000 Production income ($/kg) 1.591 1.482 1.423 In FGDs, nearly all farmers that responded to the question “have you experienced changes in your production of cocoa”, approximately 2-3 respondents in each of the six farmer beneficiaries FGD, have reported they have observed an increase in production and therefore an increase in income since they have started to work with MOCA on improved agricultural practices, and especially pruning. “Good agricultural practices have increased my production and I am making a little more money now.” Farmer, FGD “Yes, it was, for my performance regarding my crop has increased. My production is increasing gradually. But I do not know if that performance is due to the implementation Final Evaluation Report : MOCA Côte d'Ivoire 55 of these techniques, or it is because my land has not been over exploited. I need someone to let me know something about this. This has impacted my income which has doubled.” Farmer, FGD 4.3.1.2 Short-term cocoa sector productivity training as a result of USDA assistance Training was a significant component of MOCA’s activities. As mentioned, much focus was dedicated to pruning techniques, including density adjustment, rehabilitation of cocoa plants, shade adjustment, reducing flower density, and cutting branches infested by Loranthus, and distributing over 9,000 pruning tools. Although to a significantly lower degree (less than 1% of activities), the project also trained farmers on improved pest management, including insecticide application, fungicide application, suppression of moss and epiphytes, sanitary harvesting, and swollen virus control. The project trained on techniques for spreading fertilizers and for composting methods. Other trainings included climate mitigation through shading, multiplication of banana shoots, and soil health and fertility. Trainings also covered improved harvesting and post-harvesting techniques (shelling, fermentation, and drying). In addition to training for improved agriculture production techniques, MOCA worked with farmers to improve farm management techniques and access to financial services. Some training on gender equity was carried out in Daloa. The 664 Primary beneficiary survey respondents were asked if they received training and/or support from MOCA on the following topics: input (provision of pruning saws, fertilizers and pesticides), agricultural best practices for production, post-harvest practices, access to markets, financial services (including participation in events facilitated by MOCA), business development for their farms, and capacity building for their cooperatives. The survey did not limit these questions to training and/or support delivered by MOCA and included other organizations (cooperatives, ANADER, CCC, and any other), as Primary beneficiaries may not have been aware that MOCA provided the support. Final Evaluation Report : MOCA Côte d'Ivoire 56 Figure 8: Number of Primary beneficiaries receiving training and/or support (% out of 664 Primary beneficiaries) Between 88-93% of Primary beneficiaries reported they have received training and/or support on practices related to agricultural practices activities: inputs, agricultural best practices, and post-harvest practices (Figure 8). A smaller number of Primary beneficiaries said they received training and/or support on access to markets, financial services, and business plans - between 55% and 64%. This is related to some challenges related to access to finance mentioned above and further discussed below. Nearly half of Primary beneficiaries found the training and support they received to be satisfactory, and another third of the respondents reported it was very useful (Figure 9). Final Evaluation Report : MOCA Côte d'Ivoire 57 Figure 9: Rating of the training and/or support received by Primary beneficiaries (% out of 664 Primary beneficiaries) Nearly all Primary farmers said they applied the practices they learned. Rates were once again higher for agricultural practices (85-95%) and lower for those related to finance and access to markets (74-77%) (Figure 10). It is important to note that these values are self-reported, and may be different, most likely with higher adoption rates, then “adoption observation” data collection methodologies. Figure 10: Primary beneficiaries applying the training and/or support received (% of 664 Primary beneficiaries) Final Evaluation Report : MOCA Côte d'Ivoire 58 These results were confirmed in nearly all FGD conducted: farmers were highly appreciative of training and support for improved agriculture practices and reported applying them. Farmers were less likely to apply and reported more need for support on financial services, business development, and access to markets. The lower success with the support provided for financial services in comparison to that reported for inputs, agricultural practices, and post-harvesting practices, to our understanding, has two causes. First, farmers still currently have strong needs for inputs, as described in Section 4.1.1. above. Second, applying financial services requires a higher level of organization on the farm and at the community level. Improving capacity and organization on both of these levels are long-term processes that require perception and behavioral changes. These processes may not be effective in short periods such as the implementation period of MOCA. 4.3.1.3 Improved farm management More Primary and Secondary beneficiaries reported they practice bookkeeping on farm activities compared to Control farmers (Table 18). Nonetheless, these rates are still relatively low, and show much work is still needed to be done. This is in line with the project performance indicator of the number of individuals who have applied improved farm management practices, which achieved only 48% of the LoP target. Objective 1.3 Improved Farm Management Indicator: Number of individuals who have applied improved farm management practices (e.g., governance, administration, financial management) as a result of USDA assistance On the basis of the survey carried out among 664 farmer beneficiaries, 272 (41%) responded that they applied at least one improved agricultural management practices through MOCA. • Gender: Male: The survey of 664 farmers included 602 male farmers among whom 252 (42%) responded that they applied improved agricultural management practices through MOCA. • Gender: Female. The survey of 664 farmers included 62 female farmers among whom 20 (32%) responded that they applied improved agricultural management practices through MOCA. Final Evaluation Report : MOCA Côte d'Ivoire 59 Table 18: Practice of Bookkeeping – respondent type Primary Secondary Control Amount spent on inputs 19% 25% 12% Amount spent in the payment of employed workers 17% 22% 11% Number of days household members spent plowing 17% 22% 11% Volume of cocoa produced 45% 53% 37% Volume of cocoa sold, the price and the money received 53% 56% 42% Slightly more than half of the Primary beneficiaries keep records on the volume of cocoa sold, and less than half keep records on volume of production, workdays, and amounts spent on the farm. Some improvement in the bookkeeping of farm practices was observed from Baseline to Endline (Table 19). For example, in the Baseline survey, only 1% of MOCA respondents said they kept records of the number of days household members worked on the farm, while at Endline, 20% of Secondary beneficiaries reported they kept track of workdays. Table 19: Practice of Bookkeeping – number of MOCA farmers that keep records on farm activities (%) Baseline MOCA farmers Endline MOCA farmers (secondary beneficiaries) Amount spent on inputs 27% 25% Amount spent in the payment of employed workers 12% 22% Number of days household members spent plowing 1% 22% Volume of cocoa produced 45% 53% Volume of cocoa sold, the price and the money received 95% 56% Final Evaluation Report : MOCA Côte d'Ivoire 60 Information shared in multiple FGDs with farmers confirmed that these improvements of farm management were related to MOCA’s activities and that the impact on the livelihood of their families was significant: “Lots of good things happened. The project even changed our way of life. Before this training, in a period like this where there is no cocoa to sell, we were going into debts. Now we know how to do a good financial management, how to do savings.” Farmer, FGD 4.3.2 Which activity areas were most critical to the achievement or non￾achievement of project objectives? MOCA’s overall strategy provided an operational and programmatic overview of eight activities. In interviews with project stakeholders, informants found it hard to indicate which activity was more important than others. Through multiple conversations, it became apparent that this was perceived as an advantage of the project, as it created synergies between activities to a level that each activity was able to support the success of the other activities. For example, training for improved agricultural techniques was able to increase the quality of cocoa produced but required work on market access in order to sell the produce. Access to financial services is required for the capacity building of both cooperatives and producers. Nonetheless, the contribution of each separate activity to project achievements is briefly discussed below. Training: improved agricultural production techniques In collaboration with cooperative management and other cocoa sector actors such as ANADER, MOCA worked to provide training on improved agricultural production techniques. MOCA also provided tools to apply the improved practices. As mentioned previously, most farmers indicated that training and tools for pruning were the most important MOCA activity to them. This activity is discussed at length in Section 4.3.3. To what extent did performance in assisted cooperative improve because of MOCA activities? Capacity building: producer groups/cooperative MOCA selected 24 cocoa cooperatives and producer groups based on their partnerships with exporters, chocolate makers and/or finance institutions identified to date and/or depending on cooperatives’ ambitions and vision. With the selected cooperatives, MOCA developed a tailor-made organizational capacity strengthening plan to improve the efficiency of cooperatives in its governance, service provision and economic activities. Capacity building for cooperatives and producers’ groups was highly appreciated by cooperative managers and farmers alike. Together with market access, these two activities helped increase production, yields, and income to farmers. This activity is discussed at length in previous sections. This activity was critical to the achievement of project objectives. Final Evaluation Report : MOCA Côte d'Ivoire 61 Capacity building: provide business development services; In-kind grants; Inputs: develop agrodealers and/or other input suppliers MOCA worked to support Business Development Service (BDS) providers and promote local value addition and women entrepreneurship and leadership. In conversations with some of these farmers groups, positive responses to the goals and strategies of these activities were provided, emphasizing the important role and impact MOCA activities had in improving their capacities and growth. These representatives spoke mostly on MOCA’s importance as a facilitator, connecting them to other sector actors, including ANADER and CNFA. One organization compared their group to another similar producer group that MOCA did not support and said that the other group has “more difficulties connecting to other sector actors”. MOCA managed a competitive fund for in-kind matching grants for inputs and equipment to improve the productivity of cooperatives, producers, input supply professionals and processors. This activity provided seven grant agreements during the life of project. In collaboration with Callivoire, MOCA established four Farm Service Centers (FSC) to facilitate access to input services, agro-chemical inputs, and technical advice on cocoa. With these groups, MOCA worked to improve team, financial, and business management. MOCA also provided other training and support topics tailored to each group. MOCA trained Spray Service Providers for the Service Provision Units (SPUs) that will employ mainly male youth for improved input service provision (spraying, fertilization, harvesting, fermentation), in partnership with OLAM, GIZ and Jacobs Foundation. These activities brought novelty and progress into the cocoa sector, but their establishment a year before project termination or later, reduced the capacity to directly assess their effectiveness and impact on the cocoa sector. Further training and support is required to ensure the sustainability of these activities. These activities were important to the achievement of project objectives but did not have time to materialize and fully develop over the lifetime of the project. Market access: facilitate buyer-seller relationships and Capacity-building: government institutions One of the most interesting activities that also added value to the project was MOCA’s work to improve the trade of quality of cocoa and quality flavor cocoa. In Côte d’Ivoire, producing and exporting quality cocoa and quality flavor of cocoa is still a challenge. MOCA signed a MoU with the Fine Chocolate Industry Association (FCIA) to facilitate access to the fine-flavor cocoa markets. Selected (two) cooperatives were supported in quality improvement and supported them by to facilitating direct exports. To improve cocoa's quality and flavor quality, actions need to be made from the early stages of cocoa production, including the selection of plant varieties and training producers for improved harvesting, post-harvest, and storage practices, which impact the quality and flavor of cocoa. MOCA established a new partnership with Guittard Chocolate Company to export the quality flavor cocoa beans from MOCA project-supported CAPRESSA and CAVA Final Evaluation Report : MOCA Côte d'Ivoire 62 cooperatives. As a result, in June 2021, the first high-quality ‘MOCA’ container was delivered to Guittard. “Yes, I think yes, that the activities of the MOCA project have helped support and solve a particular problem especially in the quality of cocoa, so I can say that the objectives of MOCA and also because at quality level it will solve two problems it will first make it possible to increase the quality of cocoa of Côte d’Ivoire in general and when the quality is good it attracts the chocolatiers too.” Cooperative manager, KII MOCA also facilitated connections between these actors and CNRA’s flavor lab in Bingerville, Côte d’Ivoire. and supported exchange visits to Ghana. MOCA hired a flavor specialist to consult these partners in support of the process. Last, MOCA planned an international cocoa-flavor conference in collaboration with the government of Côte d’Ivoire, which was canceled due to COVID-19. With governmental institutes, MOCA worked to strengthen the capacity of ANADER to provide advisory services in three thematic areas: weather resilience, farm business plans and sustainable cocoa production. MOCA provided financial support to CNRA for research on soil fertility, swollen shoot virus, weather resilience and increasing farmer income through better agricultural practices and diversification Although work around quality cocoa and quality flavor took a small part of MOCA’s activities and resources, it provided proof of concept of the potential of farmers to produce quality cocoa in Côte d’Ivoire, and the willingness of chocolatiers to trade with the country’s cooperatives for these products. This activity was important to the achievement of project objectives and needs to be scaled up. Financial services: facilitate agricultural lending MOCA partnered with several financial institutions to improve farmers’ access to financial services through training conducted by financial institute’s staff and cooperative management. By connected financial institutes with farmers and cooperatives through events facilitated by MOCA, farmers received training and increased access to financial services. Farmers who attended these events found them relevant and informative. As discussed in Section 4.2.2, access to financial services is extremely important as it is still relatively low: the farmer survey indicated on an increase since baseline, and the number of farmers who have a bank account has increased from 8% to 20%, with a similar increase for Control farmers. In FGD’s and KIIs, informants provided some insight on the discrepancy between their motivation to open a bank account, a motivation created by MOCA’s networking activities connecting them with banks, and between their inability to open a bank account, limited by the fact that they simply do not have sufficient cash. In a few KIIs and FGDs, farmers pointed out the challenges they encountered when coming to open a bank account. One farmer said that MOCA ‘brings the American way that is very different than ours’. These results indicate on a large gap in perceptions and Final Evaluation Report : MOCA Côte d'Ivoire 63 capacity of farmers toward opening bank accounts and applying for loans, requiring a higher volume of work that is specifically geared toward this activity. This activity brought novelty and progress into the sector of financial services for cocoa producers. One financial institute representative interviewed said that MOCA was the only project that connected their organization with farmers. MOCA provided proof of concept for an activity that has the potential to grow. This activity was important to achieving project objectives but may be more effective as a focus of a separate project. 4.3.3 To what extent did performance in assisted cooperatives improve because of MOCA activities? MOCA worked closely with 24 cooperatives in the cocoa sector to improve their organizations’ professionalism. This was achieved through training for capacity building of the cooperatives’ management, improvement of service provision to members, training and support on business development, access to finance, and a series of exchange visits between cooperatives including in Gahan, at the quality lab in Bingerville Côte d’Ivoire, and with other cocoa sector actors. In KIIs and FGDs, cooperative managers spoke of how MOCA’s activities were able to improve their performance: “Then at the organizational level me and my team, it gave us some training, it’s a fact that we really can be more organized than before, and the members of the board of directors also know their role and responsibilities so it makes some decisions better since they are more involved in the activities of the cooperative.” Cooperative manager, KII One cooperative manager said that MOCA’s activities helped generate new memberships because producers knew that if they join the cooperative, they will be able to produce more. Another cooperative manager expressed the importance of MOCA’s facilitator role with exporters. Such activities started with introducing the cooperative leadership to exporters and continued toward helping to develop and maintain relationships. This even included assistance with writing emails in English to American exporters. In addition, more than half of cooperative managers interviewed spoke of the important role MOCA took in putting them in touch with banks, microfinance, and insurance companies, which allowed them to develop as a business. One cooperative manager said that MOCA helped the cooperative demonstrate to farmers the importance of committing to their work as producers. MOCA helped these producers understand that only if they commit to the project and its training will they be able to improve their income. Some challenges were also raised. Three of the 18 cooperative managers said that although MOCA has put them in touch with a bank, they have not been able to receive funding. One cooperative leader said that in the future, it would be best if cooperatives Final Evaluation Report : MOCA Côte d'Ivoire 64 would get involved in the early stages of the projects and take part in determining the objectives of the project: “By setting a global goal for the project, it will have more impact. If the initiative comes from cooperatives, it will not have a problem that it will be 100% success.” Cooperative manager, KII At another cooperative, managers in FGD said that MOCA came to train them, and in turn, they trained the farmers. However, the salary they received to perform this task barely covered travel and food expenses, and the remainder had to be paid from their pocket. 4.3.4 To what extent did the improved performance of assisted cooperatives contribute to positive outcomes and impact for farmers? MOCA worked to improve assisted cooperatives’ communication with farmer members and improve the quality, variety, and volume of training they provide to farmers. Many of the topics discussed above, from the cooperative managers’ side, were also reflected in what farmers had to say about the improvement of their cooperatives. A few farmer participants said in FGD’s that before the work MOCA did with their cooperative for capacity building, they did not know whom to reach out to when they encountered difficulties: “Since MOCA came, if you are a member of this cooperative, the cooperative assists you with all your needs.” Farmer, FGD Some farmers said in FGDs that since MOCAs involvement, their cooperative has been more transparent. One farmer said that the MOCA project allowed the growers to gain confidence in the cooperative and that their cooperative has become more efficient. In addition, many of the indicators discussed in this report, such as production, sales, and income, improved good agricultural practices, and access to finance, show an advantage for MOCA beneficiaries in comparison to Control farmers. These results indicate the effectiveness of the project to improved performance of assisted cooperatives. 4.4 Efficiency 4.4.1 To what extent have the various units/levels of management and coordination (grants, project staff, M&E) supported the implementation of the project? With a short project lifetime and ambitious goals, MOCA needed to be extremely efficient. MOCA adopted a “one project – one team” approach, in which CNFA and SOCODEVI staff made up of the project team. MOCA organized and facilitated activities that were conducted by cooperative leadership, ANADER staff, and banks. This, on the one hand, made it difficult for beneficiaries to acknowledge activities with the MOCA ‘brand’, but on the other hand, most likely contributed to the effectiveness of the project’s management structure. Final Evaluation Report : MOCA Côte d'Ivoire 65 In various discussions with project beneficiaries and partners, informants praised the level of professionalism of the MOCA staff. More than a few stakeholders said that they were impressed how information was flowing all along the hierarchical levels of the organization and that staff has always been responsive to their needs. In conversations with project staff, including SOCODEVI, staff shared how they felt committed to the project because they believed in its cause and strategy. Staff said they felt that they were encouraged to work independently and permitted to utilize their skills and knowledge, yet they also had the opportunity to work as a team and be supported by others. Some staff said that one of the advantages of the organization was that each member worked on a specific area, allowing them to achieve the objectives. It is important to note the significant contribution of the project’s management to the project's success. MOCA management practiced strong leadership with an adaptive management style, which was critical to achieving project goals under tight time and budget constraints. Under the management supervision, the MOCA team continuously collected M&E data, reviewed the work plan and achievements, and used this information to adjust and revise of strategies and activities. At least three informants said that project staff and leadership showed willingness to truly listen to farmers and take advice from stakeholders. On the challenging side, more than a few MOCA staff mentioned that the administrative requirements of the project made things harder for them, as well as for cooperatives, organizations, and farmers. 4.4.2 How did the leveraging of funds through Public-Private Partnerships add value to the project? MOCA established public-private partnerships with different value chain actors, including chocolate exporters, chocolatiers, input producers, financial service providers, and industry organizations. Through the life of the project, MOCA established 15 private-public partnerships. One of these examples, improving the trade of quality of cocoa and quality flavor cocoa, is discussed in Section 4.3.2 under market access. 4.5 Sustainability 4.5.1 To what extent did MOCA project management respond to needs and requests for collaboration from cooperatives, farmers, businesses, financing institutions, and the government institutions? Overall, nearly all informants said that MOCA’s management was attentive and responsive to their needs and requests. This was especially reflected in discussions on the activities MOCA implemented to link private and public actors such as banks and chocolate makers to farmers and producers. On the challenge side, many informants said that these collaborations need time to become stronger and more sustainable. Final Evaluation Report : MOCA Côte d'Ivoire 66 “Yes, from my perspective, I think MOCA project established good relationship, but the time was tight to establish more collaboration and be sure that they are strong. However, some of the collaboration between cooperative and financial institution can continue.” Cooperative manager, KII 4.5.2 Are project outcomes likely to be sustained and/or scaled-up after the project closes? Over half of MOCA surveyed farmers (between 58% and 62% of a total of 664 farmers, Figure 11) said that they are very likely to continue to practice the training they have received from MOCA in the future years. Similar to questions on the application of practices and the satisfaction of training and/or support, higher sustainability rates were reported for inputs and agricultural practices, and lower rates for cooperative capacity, business development, and financial services. Figure 11: Sustainability of outcomes (% out of 664 Primary beneficiaries) The likelihood for a high sustainability of project achievement on improved farming practices was also confirmed in interviews with farmers. Nearly all 61 farmers that have participated in FGDs and KIIs said they will continue with the practices they have learned to control plant disease in the coming years, including pruning, and the practices they learned for improved post-harvesting. As for financial services and business development, nearly all farmers said these are less relevant to them at the moment, but that they would like to be able to apply and improve those practices in the coming years. Final Evaluation Report : MOCA Côte d'Ivoire 67 From both quantitative and qualitative data, it becomes apparent that there is a level of dichotomy between activities. Some, such as pruning, application of pesticides and fertilizers, and harvesting and post-harvesting techniques, were easier to learn and adapt. These activities will likely continue to be practiced after project completion and have the potential to scale up as they become a habit in the community. Other activities, such as access to markets, access to financial services, and business development, require more resources in terms of time, attention, and tailored trainings and support. These activities seem to be less likely to be sustained and upscaled in future years. This differenced in the estimated sustainability of project activities may be rooted in the fact that the activity areas were designed with a five-year project in mind, but the project was awarded for a three-year period. Therefore, the project management had to adapt the activity areas and strategies to the shorter timeframe, to assure the results would be achieved and processes with beneficiaries could be finalized within the shorter timeframe. In conversations with cooperative leaders, it becomes evident that without further support, the sustainability and scaling up of project results would likely be highly dependent on the capacity of each cooperative to continue to practice these learnings and further support farmers. For example, for less capable cooperatives, it is likely that even training on GAP will cease, and without the delivery of new pruning tools, members of these cooperative will not continue to practice improved techniques. Other, more advanced cooperatives, that have been able to pick up the training and lead the support provided to members, will continue to grow, and scale up by coaching more and more farmers. Final Evaluation Report : MOCA Côte d'Ivoire 68 5 Summary, Conclusions, and Recommendations 5.1 Summary and Conclusions Cultivating New Frontiers in Agriculture (CNFA) implemented the Maximizing Opportunities for Cocoa Activities (MOCA) project intended to increase the productivity and efficiency of actors in the cocoa value chain in Côte d’Ivoire. MOCA applied a holistic value chain development approach supporting farmers and other value actors with quality, production, and efficiency improvement activities, from input provision to market access, and from access to finance to improved cooperative governance. It intended to increase their income through increased trade of and added value to cocoa beans without contributing to an increase in cocoa production. The Final Evaluation aimed to critically evaluate the project’s performance in terms of project relevance, effectiveness, efficiency, sustainability, and impact. The evaluation used a mixed-methods approach, employing both quantitative and qualitative methods. Quantitative data was collected from a sample size of 1,310 farmer cocoa producers, including farmers that were trained directly by MOCA and their partners (Primary beneficiaries), farmers that belong to cooperatives that MOCA worked with (Secondary beneficiaries), and a Control farmers Group. Qualitative data was collected through nine focus group discussions (FGDs) with MOCA Primary beneficiaries (farmers and cooperative management) and 33 key informant interviews (KIIs) with various stakeholders, including cooperative and farmers’ organization management, MOCA partners such as bank representatives, governmental representatives, exporters, and chocolatiers, and MOCA staff. The evaluation used, when possible, information from the Baseline study to evaluate project impact and change over time. Despite the short project time span compared to most development projects in the agriculture sector, and the further reduction in the project’s implementation period, MOCA achieved and even overpassed by large most of the project performance targets. The evaluation was able to validate these achievements. Based on internal project monitoring, the project achieved 103% of the life of project (LoP) target for the number of individuals who have applied new techniques or technologies as a result of USDA assistant. The farmer survey results validated this achievement, indicating that 95% of MOCA farmers surveyed also applied at least one the training they received from MOCA on improved input, agricultural best practices, and post-harvest techniques. In FGDs and KIIs with project beneficiaries, nearly all farmers confirmed that they apply the new techniques and technologies acquired. This was further validated in FGDs and KIIs with cooperative managers that said that these trainings were practiced by farmers. Based on internal project monitoring, the number of hectares of land under improved techniques or technologies as a result of USDA assistance achieved 125% of the LoP target. When compared to Baseline values, the Endline survey results showed a small Final Evaluation Report : MOCA Côte d'Ivoire 69 increase in the area on which fungicide and organic fertilizer were applied, and a small decrease in the area on which insecticide and chemical fertilizer were applied since Baseline. Trends for MOCA farmers where similar to those for Control farmers. Based on internal project monitoring, the volume of commodities sold by project beneficiaries achieved 111% of the LoP target, and the value of sales of cocoa products by project beneficiaries achieved 174% of the LoP target. On average, MOCA beneficiaries made 69% more than at Baseline, and 17% more than the income of the Control group at Endline. Moreover, MOCA farmers were able to increase their income from bonuses received on premium more than Control farmers: twice as much MOCA farmers than Control farmers received a bonus, and the size of the average bonus received was larger for MOCA farmers than for Control. While income from bonuses account for only a small portion of the farmers annual income (4-5%), it shows potential to scale-up. 60% of respondents said that they will continue to apply the new practices in the coming years. In discussions and interviews, it became apparent that farmers need a longer project time span to retain the knowledge gained. Farmers asked that the support provided by MOCA will continue and said they need ‘refreshers’ and more opportunities to grow and further develop their capacities. The farmers survey showed that the number of farmers that have a bank account has more than doubled since Baseline, and that the increase in the number of individuals that have a bank account increased more for MOCA farmers than for Control Farmers. However, this number is still low, and only 20% of MOCA farmers possess a bank account. In FGDs, more than half of the interviewed farmers said they have attended events with financial institutes arranged by MOCA and received training on how to open bank accounts, yet they did not apply for these financial services. More work is needed to identify bottlenecks and plan strategies to improve finance for producers and cooperatives. MOCA worked to improve the quality of cocoa and quality flavor cocoa production and trade. This activity strongly aligns with Côte d'Ivoire's strategy to add value without increasing cocoa production and high potential to bring change to the sector and increase income for farmers. MOCA’s work on improved production and post-harvesting techniques proved that the country’s producers can produce high-quality cocoa and high-quality flavor cocoa. MOCA’s work to link cooperatives with exporters and chocolatiers that purchased these products provided another proof of concept to the attainability of this activity for future projects. By partnering with sector actors and creating new public-private partnerships, the project was able to leverage USDA’s investment and bring other funders into Côte d'Ivoire's cocoa sector. The value of the new public and private sector investment leveraged by USDA assistance was more than 2.5 times higher than the target. MOCA provided In-kind grants to farmers' producers groups, established Farm Service Centers (FSC), and initiated a network of Spray Service Providers. Because most MoU’s and grants were signed close to project completion, the impact of this activity could not be evaluated at Endline. Similarly, the time that has passed since the opening of the FSC was too short to evaluate their impact or sustainability. Final Evaluation Report : MOCA Côte d'Ivoire 70 These successes can be attributed to the devoted and professional MOCA team, which created effective and efficient training and support activities. Project beneficiaries at all levels and project partners praised the transparency and responsiveness of project staff. The project leadership practiced a result-oriented, adaptive management style that responded to current actual needs and focused on activities that are achievable during the short life span of the project, and activities were tailored to the specific needs of each region, cooperative, and organization MOCA worked with. 5.2 Project Successes MOCA trained and supported farmers on improving agricultural practices to increase cocoa production and cocoa quality through improved harvesting and post-harvesting techniques. The results show that in terms of applying the knowledge, most farmers (~95%) applied the learned techniques and found them useful. This information was verified in multiple discussions and interviews with farmers and cooperatives. In these discussions, training on pruning techniques and the distribution of pruning tools was especially praised by farmers, who attested that these specific activities brought value and helped to increase production and income from cocoa sales. The evaluation results indicated that MOCA's training and support impacted farmers’ production and income despite the short implementation period. MOCA farmers yield increased from 414 kg/Ha at Baseline to 583 kg/Ha at Endline and was higher than the yield for the Control group at Endline (566 kg/Ha). On average, MOCA beneficiaries made 2,206,538 CFA per year from the sale of cocoa, which was 69% more than at Baseline, and 17% more than the Control group income at Endline. MOCA worked closely with cooperatives to build their capacity for improved management plans and business plans. These actives were able to enhance cooperative effectiveness and the level of support they provided to farmers. These impacts were verified both by cooperative managers and cooperative farmer members. With the enhanced capacities, cooperatives were able to reach more farmers and scale up MOCA’s impact. MOCA created linkages in the cocoa sector and increased collaborations between CNRA and quality flavor producers and cooperatives. MOCA also provided In-kind Grants to farmers' producers groups, established Farm Service Centers, and initiated a network of Spray Service Providers. While the outcomes of MOCA’s activities were measurable and notable, it is too early to evaluate the impact of these activities. 5.3 Project Challenges Nearly all MOCA’s beneficiaries currently apply the training on best agricultural practices, yet less surveyed farmers (~60%) said they will continue to use them in the coming years. In discussions and interviews, it became apparent that farmers would like the support provided by MOCA to continue. Farmers said they would benefit from an interactive process with trainers that would allow them to improve and apply lessons learned incrementally. Final Evaluation Report : MOCA Côte d'Ivoire 71 Trainings on best practices for post-harvesting, including fermentation and drying of beans were reported as less successful. Farmers said they need more support in applying the trained techniques in their farms rather than only attending training in farm schools. Farmers, and especially women farmers, said they need more tarps for drying the fermented beans. This is a reason for concern, indicating that more training is required to improve best practices for drying the beans. When beans are dried on mats laying on the ground rather than on elevated beds, this may lead to the development of mold, accumulation of dust from the ground, and an undesirable cocoa flavor, which can lead to the rejection of the final product. In both qualitative and quantitative collected evidence, farmers said they need access to more phytosanitary input and fertilizers. These inputs increase yields and product quality but are expensive and often unattainable to farmers. Although many farmers received free input or received input through cooperative credit, it is often insufficient for their needs. Two cross-cutting issues were raised in various discussions with farmers, cooperative leaders, and partners: gender integration in the cocoa sector and elimination of child labor. Although MOCA was not designed to work on these challenges, some work was conducted indirectly. For example, the project worked only with cooperatives that committed to eliminating child labor. And women’s producers’ groups were encouraged to apply for in-kind grants. 5.4 Recommendations for Future Programming • Develop an exit strategy with concrete plans for cooperatives and partners who provide support to farmers to ensure the continuation of the training and support provided to MOCA farmers and the sustainability of the results of improved agricultural practices. • Planning for future activities should focus on improving post-harvesting techniques and include distribution of inputs, using the successful model MOCA developed by focusing on training and tools for pruning. • Enhance strategies that reduce the gap between farmers' need for fertilizers and pesticides and the availability of these inputs. Such activities can focus on a governmental and/or cooperative strategy to better address farmers' needs. In addition, more training and support can be provided to enhance agroecological methods that reduce the need for pesticides and chemical fertilizers. • Conduct further research to identify gaps and bottlenecks in the application of financial services, including bank accounts, saving accounts, micro-finance, and insurance products. Develop a strategy that will improve farmers’ and cooperatives’ usage of financial services for agriculture production and marketing. • Conduct an ex-post evaluation in three to five years to better assess the impact of value chain linkages created by MOCA, continuing collaborations with governmental agencies, and growth in public-private partnerships, farmers organizations that received In-kind Grants, Farm Service Centers, and the Spray Service Providers network. Final Evaluation Report : MOCA Côte d'Ivoire 72 • Future planning should scale up the successful work for enhancing the production and trade of quality cocoa and quality flavor cocoa. • Projects in the agriculture sector and the cocoa sector in particular need to incorporate programming on topics that represent strong sectoral challenges, such as gender integration, child labor, and deforestation. Work on these topics as an integral part of the project needs to be mandatory and not voluntary. • Projects in the agriculture sector need to have a longer time span, preferably not shorter than five years, to provide more opportunities to utilize resources effectively and provide continuous support to producers and organizations during the growth and development period. Final Evaluation Report : MOCA Côte d'Ivoire 73 6 Appendixes 6.1 Appendix 1: Evaluation Matrix Research Questions Alignment Source/Respondents Relevance To what extent has the project responded to the actual needs and interests of the target beneficiaries? • Desk review: MOCA design documents and work plans, progress reports and semi-annual reports. • FGDs: Primary beneficiary producers. • KIIs: Cooperative managers, MOCA managers, service/input suppliers, banks/MFIs. To what extent does the project integrate or strengthen the national cocoa strategy of Côte d’Ivoire? • Desk Review: MOCA Results Framework & design documents, Côte d’Ivoire national cocoa strategy. • KIIs: MOCA managers, Government of Côte d’Ivoire officials (CCC, ANADER, CNRA), local organizations working in the cocoa sector, USDA managers. Efficiency To what extent have the various units / levels of management and coordination (grants, project staff, M&E) supported the implementation of the project? • KIIs: MOCA Managers & staff, USDA Managers, grantees. How did the leveraging of funds through Public Private Partnerships add value to the project? • KIIs: MOCA managers, local organizations working in the cocoa sector, USDA managers, Olam, GIZ, SOCODEVI, and Jacobs Foundation, Effectiveness To what extent have the expected results of the project been achieved? Considering the eight activities areas described above: which were activity areas were most critical to the achievement or non￾achievement of project objectives? • Desk Review: MOCA Results Framework, performance indicator monitoring data against Baseline data and targets (all indicators with Endline (LOP) results – final data collection to be completed before Final Evaluation). • FGDs with project beneficiary groups (by group): MOCA managers, Government of Côte d’Ivoire officials (CCC, ANADER, CNRA), local organizations working in the cocoa sector, USDA, Cooperative leadership, MOCA’s grantees, and partners Guittard, Blommer, ECOM trading, OikoCredit, Baobab, and Saham Insurance. In general: other MOCA national and international partners. To what extent did performance in assisted cooperatives improve because of MOCA activities? To what extent did the improved performance of assisted cooperatives contribute to positive outcomes and impact for farmers, including access to • Survey: Cocoa Cooperative and Farmer Household Survey. • FGDs with project beneficiary groups (by group): Producers, producer group/cooperative managers and members, and MOCA partners (input suppliers, bank/Microfinance Institutions (MFIs), government officials, exporters). Final Evaluation Report : MOCA Côte d'Ivoire 74 finance and increased value and volume of sales? Impact To what extent did MOCA activities produce increases in cocoa farmer production and income from sales: • Access to improved inputs (fertilizer, pesticides, insecticides etc.) and input services (GAP training and coaching, spraying, harvesting, maintenance, etc.). • Access to finance (loans to individuals and/or coops) and financial services (mobile banking, bank accounts, insurances, etc.). • Access to markets (increased certified bean sales, premiums, Direct exports, etc.). • Production (increased productivity or reduced production, harvest, and post￾harvest losses). • Survey: Cocoa Cooperative and Farmer Household Survey. Sustainability To what extent did MOCA project management respond to needs and requests for collaboration from cooperatives, farmers, businesses, financing institutions, and the government institutions? • Desk review of MOCA’s work plan, progress reports, and other project reports. • Key Informant Interviews: MOCA managers and staff, managers and members of farmer organizations/cooperatives, government agencies, banks/MFIs. Final Evaluation Report : MOCA Côte d'Ivoire 75 6.2 Appendix 2: References 6.2.1 Academic papers and external reports Adeniyi, D., 2019. Diversity of Cacao Pathogens and Impact on Yield and Global Production, Theobroma Cacao - Deploying Science for Sustainability of Global Cocoa Economy, Peter Osobase Aikpokpodion, IntechOpen, DOI: 10.5772/intechopen.81993. Available from: https://www.intechopen.com/chapters/65131 Amon-Armah, F., Domfeh, O., Baah, F., Dzahini-Obiatey. H. K., 2021. Effectiveness of the eradication campaign of cocoa swollen shoot virus disease in Ghana: the extension and implementation problem, International Journal of Pest Management, DOI: 10.1080/09670874.2021.1943049 Bowers, J., Bailey, B., Hebbar, P., Sanogo, S., Lumsden, R, 2001. The Impact of Plant Diseases on World Chocolate Production. Plant Health Progress. 10.1094/PHP-2001- 0709-01-RV. Bymolt, R., Laven, A., Tyszler, M., 2018. Demystifying the cocoa sector in Ghana and Côte d’Ivoire. The Royal Tropical Institute (KIT). Kanga, D., Moussa, R., Sanogo, V., 2018. Cocoa Sector in Côte d'Ivoire: Are Public￾Private Partnerships Helpful? (English). Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/170701580202058469/Cocoa-Sector-in￾Côte-dIvoire-Are-Public-Private-Partnerships-Helpful Knößlsdorfer, I., Sellare, J., Qaim, M., 2021. Effects of Fairtrade on farm household food security and living standards: Insights from Côte d’Ivoire. Global Food Security, Volume 29. https://doi.org/10.1016/j.gfs.2021.100535. Lonie, S., Martinez, M., Oulai, R., Tullis, C., 2018. Opportunities for Digital Financial Services in the Cocoa Value Chain in Côte d’Ivoire: Insights from New Data. World Bank, Washington, DC. © World Bank. https://openknowledge.worldbank.org/handle/10986/30203 License: CC BY 3.0 IGO. Programme National D’investissement Agricole de Deuxieme Generation (2017 – 2025), Rapport Final Novembre 2017 Sellare, J., Meemken, E.M., Koume, C., Qaim, M., 2020. Do Sustainability Standards Benefit Smallholder Farmers Also When Accounting for Cooperative Effects? Evidence from Côte D'Ivoire. American Journal of Agricultural Economics, Vol. 102, Issue 2, pp. 681-695. 6.2.2 MOCA Data sources Semi Annual Reports: • MOCA Semi Annual Report, April 2018 • MOCA Semi Annual Report, October 2018 • MOCA Semi Annual Report, April 2019 • MOCA Semi Annual Report, October 2019 Final Evaluation Report : MOCA Côte d'Ivoire 76 • MOCA Semi Annual Report, April 2020 • MOCA Semi Annual Report, October 2020 • MOCA Semi Annual Report, April 2021 • MOCA Baseline Evaluation Report, September 2019 • MOCA Work Plan Year 2, October 2018 • MOCA Work Plan Year 4, September 2020 Contracts and Legal Documents: • Agreement FCC-681-2017-029-00 - CNFA Côte d'Ivoire, May 2017 • SOCODEVI Subaward, January 2018 • MOCA Contract JRK, February 2020 • MOCA Contract AIC, February 2020 • MOCA Final Evaluation RFP, December 2020 • Agreement between the government of the United States of America and CNFA for the provision of agricultural commodities through the food for progress act amendment III Project Delivery and M&E: • MOCA MEL Plan, July 2017 • MOCA Work Plan Gantt Year 4 MOCA Databases: • Baseline Respondent, clean data Final Evaluation Report : MOCA Côte d'Ivoire 77 6.3 Appendix 3: Project Timeline The tables below provides a detailed work plan, with roles and responsibilities. Project Inception Tasks/Activities Star t Date End Date Responsible Party Key Outputs/Deliverable Kickoff meeting to discuss the overall project goals 6/1 6/1 TKG/ CNFA Meeting agenda Meeting summary Literature/document review 6/1 6/8 TKG Notes/Questions Develop Inception Report/Research Plan 6/1 6/15 TKG Inception report/research plan Develop field team training material 6/1 6/15 TKG Field team training materials Pre-test (internal testing of tools) 6/16 6/15 TKG Pretest data set and notes Modify data collection instrument 6/15 6/15 TKG/ CNFA Modified data collection instruments Approval of training contents 6/15 6/15 CNFA Approved field team training materials Fieldwork Preparation and Data Collection Star t Date End Date Responsible Party Key Outputs/Deliverable Recruit field team 6/1 6/12 TKG Field team recruited Train field team 6/16 6/17 TKG/ CNFA Trained field team Pilot testing of instruments and re-training 6/18 6/18 TKG Pilot testing completed Refine data collection plan and strategy 6/19 6/20 TKG/ CNFA Updated data collection strategy Submit training report 6/20 6/21 TKG Report of training Collect data 6/23 7/4 TKG Data collected electronically Quality Assurance 6/23 7/4 TKG Data quality assured Transcribe and translate KIIs 6/25 7/21 TKG KIIs transcribed Data Cleaning and Analysis Star t Date End Date Responsible Party Key Outputs/Deliverable Conduct Data Cleaning 6/22 7/10 TKG Data cleaned Analyze data 7/10 7/20 TKG Data analyzed Final Evaluation Report : MOCA Côte d'Ivoire 78 Report and Closure Star t Date End Date Responsible Party Key Outputs/Deliverable Submit rough draft evaluation report 7/26 7/26 TKG Draft Evaluation Report to CNFA Receive feedback from CNFA 7/26 7/28 CNFA Feedback received Submit Draft Evaluation Report 8/15 8/18 TKG Finalized Report Submitted Final Draft Evaluation Report 8/23 8/25 TKG/CNFA/USD A Feedback received Receive feedback from CNFA and USDA 9/20 9/10 TKG/ CNFA/USDA Feedback received Project closure TBA TBA TKG Final report, raw files, project closure notes, etc. 6.4 Appendix 4: List of KIIs Last Name First Name Position Organization Function Location 1 Kone Zatcha Customer advisor CORIS Bank Bank/MFI Daloa 2 Yves Komaclo Investments West Africa OikoCredit Bank/MFI Abidjan 3 Ousseni Sawadog o President CAPA Cooperative Zaranou 4 Eric Kouame Director CAPRESSA Cooperative Abengourou 5 Kanga Mea Farmer CAPRESSA Cooperative Zaranou 6 Isiaka Coulibaly Director CAVA Cooperative Vavoua 7 Isiaka Coulibaly Farmer CAVA Cooperative Daloa 8 Boureliman Djibo Director Coopaza Cooperative Zaranou 9 Adama Meite Director ECAM Cooperative Meaguy 10 Nicodem Ahoutou Farmer ECAM Cooperative Soubre 11 Helene Koume Director ECAMOM Cooperative Meaguy 12 Assale Herve Tiasman Director FAHO Cooperative Abengourou 13 Holo Kambou Farmer FAHO Cooperative PRAKRO 14 Bathelemy Kouakou Director Kapatchiva Cooperative Bonon 15 Doumbouy a Abdoul Karim Director SCALAVA Cooperative Vavoua 16 Eugeinie Laco President Groupement des femmes D'ECOOJAD Farmer Organization Ziguedia Final Evaluation Report : MOCA Côte d'Ivoire 79 17 Odette Anaki President Groupement des femmes D'ECOOJAD Farmer Organization Vavoua 18 Salif Kone Farmer SOCOOABAS Farmer Organization Konedougo u 19 Abdoulaye Diabate President SOCOOABAS Farmer Organization Konedougo u 20 Loius Ban Koffi Head of Lab Department CNRA Government Abidjan 21 Alphonse Kouame Value chain development manager CNFA Program staff Soubre 22 Emily Hendrick M&E Director CNFA Program staff Washington DC 23 Hugues Tape Program Director CNFA Program staff Abengourou 24 Marc Steen Chief of Party CNFA Program staff Abidjan 25 Mariame Toure Value chain development manager CNFA Program staff Daloa 26 Maxime Yao Cocoa Production Specialist CMFA Program staff Soubre 27 N'Bondann a Ganon M&E Director CNFA Program staff Abidjan 28 Olivia Gbodo Cocoa Production Specialist CNFA Program staff Abengourou 29 Zakaria Traore M&E Officers CNFA Program staff Daloa 30 Deli Gue Cooperative Capacity Building Facilitator SOCPDEVI Program staff Soubre 31 Christian Dompou President I'UPS Niapidou Service Spray Provider Sassandra 32 Bill Guyton Executive Director Fine Chocolate Industry Association Service /input suppliers Washington DC 33 John Kehoe Director of Sustainability Guittard Service/inpu t suppliers San Francisco Final Evaluation Report : MOCA Côte d'Ivoire 80 6.5 Appendix 5: Farmer Household Questionnaire (Quantitative Tool) Final Evaluation Report : MOCA Côte d'Ivoire 81 Final Evaluation Report : MOCA Côte d'Ivoire 82 Final Evaluation Report : MOCA Côte d'Ivoire 83 Final Evaluation Report : MOCA Côte d'Ivoire 84 Final Evaluation Report : MOCA Côte d'Ivoire 85 Final Evaluation Report : MOCA Côte d'Ivoire 86 Final Evaluation Report : MOCA Côte d'Ivoire 87 Final Evaluation Report : MOCA Côte d'Ivoire 88 Final Evaluation Report : MOCA Côte d'Ivoire 89 Final Evaluation Report : MOCA Côte d'Ivoire 90 Final Evaluation Report : MOCA Côte d'Ivoire 91 Final Evaluation Report : MOCA Côte d'Ivoire 92 Final Evaluation Report : MOCA Côte d'Ivoire 93 Final Evaluation Report : MOCA Côte d'Ivoire 94 6.6 Appendix 6: Semi-structured FGD’s Questions (Qualitative Tool) 6.6.1 Farmers Focus Group Discussion Tool Introduction Good morning/afternoon, my name is ____from The Khana Group. I am part of the team conducting the Final Evaluation study for the Agribusiness Investment for Maximizing Opportunities in Cocoa Activities project (MOCA) on behalf of CNFA. The purpose of the study is to collect information on the project’s impact. Please be informed that your participation in this discussion is voluntary and all information collected will be treated with the utmost confidentiality and the information you provide will be aggregated with information from other respondents and used to draw general conclusions on the MOCA performance. The discussion will take the shortest time possible, and you should not hesitate to say you do not understand a question or do not know the answer. Please note that there are no right or wrong answers. To begin, let’s go around the room and introduce ourselves. Please say your name, where you are from, and if you have any formal position. Open Discussion Facilitators will ask FGD participants to discuss these questions at length. Facilitators will continue to prob participants if answers are too short or too few with questions such as – what else? In what way? Can you be more specific? RELEVANCE 1. What specific needs do you, your household, or your cooperative have regarding cocoa production, harvesting and post-harvesting, marketing, and selling? IMPACT In this section, we would like you to say in what way and to what extent has MOCA activities impacted your cocoa production, cocoa sales, and income from sales. 2. Can you describe some of MOCAs activities? [at this point, let participants name activities they most strongly remember]. 3. Training and services for improved agricultural practices (pruning, fertilization, tree density adjustment, shading adjustment, maintaining, cleaning, and weeding): • Have you or your cooperative received training and/or technical assistance on agricultural practices from MOCA or organizations that partner with MOCA? • How did the training change the way you grow cocoa? Please be specific, provide examples. 4. Improved access to inputs and pest management (fertilizer, pesticides, insecticides, fungicide): • Did you receive any inputs from MOCA and /or MOCA partners? Final Evaluation Report : MOCA Côte d'Ivoire 95 • Are you using these inputs, and if yes, in what way? • If yes, has the use of improved inputs impacted your production, sales, and income? • If no, why not? 5. Harvesting and post-harvesting (harvesting, shelling, fermenting, drying): • Did you receive any training or support for harvesting and post-harvesting from MOCA and /or MOCA partners? • Are you applying these practices, and if yes, in what way? • If yes, have these practices impacted your production, sales, and income? • If no, why not? 6. Financial planning and farm management (agro-ecosystem analysis, the role and responsibilities of the member, year-round management of farm for food and income, agriculture entrepreneurship, marketing) • Have you or your cooperative received training and/or assistance on financial planning and farm management from MOCA or organizations that partner with MOCA? • How did the support change the way you grow and sell cocoa? Please be specific, provide examples. 7. Access to agricultural credit (micro-lending, loans to individuals, loans to cooperatives, mobile banking, opening bank accounts, insurance, etc.) to finance the purchase of cocoa beans and invest in activities: • Have you or your cooperative improved your access to finance with the aid of MOCA? • If yes, how where they used? • Did they help improve production, sales, and income? • If no, why not? 8. Cooperative capacity building: • Did your cooperative leaders and staff receive training on good governance, human resources, accounting, etc. from MOCA or partners? • Has that impacted your production, sales, and income? EFFECTIVENESS 9. Considering the activities areas described above: which activity areas were most important to you, and why? Which activities were less or not important, and why? Consider both challenges and advantages of these activities brought to you. 10.Which of the activities were most effective in increasing cocoa productivity, sales, and income from sales, and why? Which activities were not effective? 11.Has the project been able to effectively respond to your actual needs and interest? In what way? [Remind informants of issues they mentioned in question 1]. 12.Has the project been able to address the unique needs of women, youth, and other groups? If yes, in what way? Please give example of things that were improved, and some that were not. Final Evaluation Report : MOCA Côte d'Ivoire 96 13.Was the implementation period (January 2019 to end of June 2021) sufficient to effectively improve your cocoa production and income? EFFECIENCEY 14.How was your experience working with MOCA project trainers, managers, and partners? Do you feel they worked well to support the project, or not, and in what way? 15.Can you suggest any recommendations for improvement? SUSTAINABILITY 16.Do you plan to continue applying the learnings from training and the changes you’ve made because of the project’s support in future cocoa growing seasons? If yes -which, and if not - why not? 17.What kinds of support might help you continue with new farming and marketing practices in the future? 18.Do you think that your cooperative will continue to apply the training learned, especially regarding good governance, finance, and business development? LESSONS LEARNED 19.What changes could have been made in the project design or implementation to improve the impact of MOCAs activities? Final Evaluation Report : MOCA Côte d'Ivoire 97 6.6.2 Cooperative Staff and Managers Focus Group Discussion Tool Introduction Good morning/afternoon, my name is ____from The Khana Group. I am part of the team conducting the Final Evaluation study for the Agribusiness Investment for Maximizing Opportunities in Cocoa Activities project (MOCA) on behalf of CNFA. The purpose of the study is to collect information on the project’s impact. Please be informed that your participation in this discussion is voluntary and all information collected will be treated with utmost confidentiality and the information you provide will be aggregated with information from other respondents and used to draw general conclusions on the MOCA performance. The discussion will take the shortest time possible, and you should not hesitate to say you do not understand a question, or do not know the answer. Please note that there are no right or wrong answers. To begin, let’s go around the room and introduce ourselves. Please say your name, where you are from, and if you have any formal position. Open Discussion Facilitators will ask FGD participants to discuss these questions at length. Facilitators will continue to prob participants if answers are too short or too few with questions such as – what else? In what way? Can you be more specific? IMPACT In this section, we would like you to say in what way and to what extent has MOCA activities impacted the cooperatives’ cocoa production, cocoa sales, and income from sales. 1. Cooperative capacity building: • Did your cooperative receive training on good governance, human resources, accounting, etc., from MOCA or partners? • Has that impacted the capacity of your cooperative? • More specifically, has that impacted members’ production, sales, and income from cocoa? • What were some of the most successful capacity building activities? • What were some less successful capacity building activities? • Has these training been able to support the equal involvement of women staff and leadership in your cooperative? 2. Access to credit (loans to cooperatives, in-kind matching grants): • How has your cooperative been impacted by MOCA’s support to improved access to credit, and in what way? • How has your cooperative used these loans and grants? • Have your members benefited from these loans and grants, and how? 3. Access to markets: • Have any of MOCA activities help your cooperative increased bean sales, and how? Final Evaluation Report : MOCA Côte d'Ivoire 98 • Which beans specifically – certified, premiums, Direct exports, etc.? • Can you mention some more successful activities that increased your access to markets and some that were less successful? 4. Cooperative business development services: • Has your cooperative received training and support for the development of a business plan? • If yes, in what way has that impacted your production, sales, and income? • External relations with input and service suppliers: • Has your cooperative received support from MOCA to develop and increase your relationships with input and service suppliers and buyers? • If yes, in what way has that impacted your production, sales, and income? • Facilitating Market Relationships: • Has your cooperative received support from MOCA to develop and increase your relationships with the chocolate industry – association and private exporters? • If yes, in what way has that impacted your production, sales, and income? EFFECTIVENESS 5. Considering the activities areas described above: which activity areas were most important to you, and why? Which activities were less or not important, and why? Consider both challenges and advantages of these activities brought to you. 6. Has the project been able to effectively respond to your actual needs and interest? In what way? Are there other needs that MOCA did not work around? 7. Was the implementation period (January 2019 to the end of June 2021) sufficient to effectively improve your cooperatives’ operations? EFFECIENCEY 8. How was your experience working with MOCA project trainers, managers, and partners? Do you feel they worked well to support the project, or not, and in what way? 9. Can you suggest any recommendations for improvement? SUSTAINABILITY 10.What kinds of support might help your cooperative to continue to grow in the future? 11.Was MOCA able to establish strong collaborations between cooperatives, farmers, businesses, financing institutions and government institutions? 12.Do you think that the established partnerships will continue to take place and grow in the next year or so? What about the longer term, let’s say in five years from now? 13.Do you think that your cooperative will continue to apply the training learned, especially regarding good governance, finance, and business development? LESSONS LEARNED Final Evaluation Report : MOCA Côte d'Ivoire 99 14.What changes could have been made in the project design or implementation to improve the impact of MOCAs activities? Final Evaluation Report : MOCA Côte d'Ivoire 100 6.7 Appendix 7: Semi-structured KII’s Questions (Qualitative Tool) The following component of the tool relates to all types of KIIs: Good morning/afternoon, my name is ____from The Khana Group. I am part of the team conducting the Final Evaluation study for the Agribusiness Investment for Maximizing Opportunities in Cocoa Activities project (MOCA) on behalf of CNFA. The purpose of the study is to collect information on the project’s impact. Please be informed that your participation in this discussion is voluntary, and all information collected will be treated with the utmost confidentiality and the information you provide will be aggregated with information from other respondents and used to draw general conclusions on the MOCA performance. The discussion will take the shortest time possible, and you should not hesitate to say you do not understand a question or do not know the answer. Please note that there are no right or wrong answers. 6.7.1 MOCA and SOCODEVI staff Introduction Please state your position at the MOCA project and for what period and describe some of your main activities and responsibilities. IMPACT 1. Based on your experience, what were some of the biggest impacts MOCA had in Côte d’Ivoire, thinking especially about the two main strategic objectives: increase productivity and efficiency of actors in the cocoa value chain, and improve and expand the trade of cocoa through improved productivity, handling, and market linkages? 2. What key changes, positive or negative, resulted because of the MOCA project activities toward these goals? Thinking of farmers, cooperatives, and the cocoa value chain. EFFECTIVENESS 3. What were some of the most effective activities conducted by MOCA and its partners? Think of MOCA’s eight main programmatic components. What do you think led these activities to be especially successful? 4. What were some of the challenging or less successful activities conducted by the project? In what way were they less contributing, and why? 5. How well were project activities able to synergistically support each other? 6. Whom do you consider the most important partners and/or stakeholders supporting MOCAs activities toward achieving project goals? Are there important partners the project has not engaged with? EFFICIENCY Final Evaluation Report : MOCA Côte d'Ivoire 101 7. What were some of the challenges and benefits of engaging multiple partners in MOCAs activities? 8. Based on your experience, why do you think some farmers adopt advanced farming practices such as pruning, pest management, business development, etc., because of MOCA’s activities, while others did not? How should activities be planned or implemented differently to enhance farmers' adoption of advanced farming practices? 9. Based on your experience, why do you think some cooperatives took advantage of MOCA’s training on improved management and finance while other cooperatives did not? How should activities be planned or implemented differently to enhance cooperative adoption of advanced management practices and financial solutions? 10.Now I want to ask you about MOCA as an organization. How efficient do you feel were the mechanisms and organizational setup of the various units and management levels in supporting the implementation of the project? Relate to: Program governance, including Steering Committee, Technical Committee, and Regional Technical Committees, Program management, Program staff, Monitoring and Evaluation, Grantmaking, Coordination with partners, private sector, and government, Contracting and legal assistance. 11.Has MOCA been able to address your needs and requirements for the services you needed to provide? This includes staff capacity strengthening and any physical inputs. 12.How well were you able to meet your initial work plan and responsibilities? RELEVANCE 13.Do you feel that the training and support you provided to cooperatives and other organizations through the MOCA project were relevant to the needs of the cocoa value chain of the country? 14.Were these activities planned in a way that efficiently addressed those needs? 15.Is there anything you would have changed in the design of MOCAs activities to better address the needs of the value chain? SUSTAINABILITY 16.Thinking about the positive changes created by the MOCA project, what changes do you think will continue in the future and which ones might reverse course? 17.Was MOCA project management able to establish strong collaborations between cooperatives, farmers, businesses, financing institutions and the government institutions, such that these collaborations will continue after MOCA ends? Did MOCA respond to requests for collaboration from cooperatives? 18.Imagining an ideal situation, what kind of support would be necessary to continue the positive changes in the future? 6.7.2 Cooperative Management and Farmers Organizations Final Evaluation Report : MOCA Côte d'Ivoire 102 Introduction Please state your name, cooperative, and position at the organization, and for what period. Please describe some of your main activities and responsibilities. How did you and your organization become involved in the MOCA project? What were the primary avenues through which you and your organization received support from the project? HIGH-LEVEL IMPACT 1. Based on your experience, what were some of the biggest impacts MOCA had on your cooperative? In other words, in what way has your cooperative changed, both positively and negatively, because of the MOCA project activities? SPECIFIC IMPACT 2. Please described if, and in what way, has MOCA activities impacted: • The number of members in your cooperative? The revenues of your members and their financial situation? • The governance of your cooperative? i.e., the way you manage your cooperative operations, including financial and human resources? • The way you engage with exporters, banks, input suppliers, and other businesses? • Are women more involved in production or cooperative management? RELEVANCE 3. What are the main needs your cooperative and farmers have right now? Thinking in general, not necessarily related to MOCA. 4. Have MOCA’s activities addressed your actual needs? Has MOCA been able to support these needs? EFFECTIVENESS 5. What were some of the most effective activities conducted by MOCA and its partners? Why do you think these activities were more successful than others? 6. What were some of the weaker or more challenging activities conducted by the project, and in what way? Why were they less successful? 7. How well were project activities able to support each other? This means, for example, if your cooperative received training on business development, did this also help farmers in any way? And if your farmers received pruning tools, has this impacted your cooperative business development? Please provide examples. 8. Did MOCA respond to any requests your cooperative may have had for collaboration with banks, insurance companies, microfinance, government, input suppliers, exporters, etc.? 9. Did MOCAs activities help you and your cooperative in improving: • Linkages with exporters Final Evaluation Report : MOCA Côte d'Ivoire 103 • Access to funding • Linkages to banks, insurance companies, and microfinance • Business development • Marketing cocoa EFFICIENCY 10. Now I want to ask you about MOCA as an organization. How efficient were the mechanisms and organizational setup of the various units and management levels in supporting the implementation of the project? Think of: Program staff, Grantmaking, Contracting, Coordination with partners 11. Based on your experience, why do you think some farmers adopt advanced farming practices such as pruning, pest management, business development, etc., because of MOCA’s activities, while others did not? How should activities be planned or implemented differently to enhance farmers' adoption of advanced farming practices? 12. Based on your experience, why do you think some cooperatives took advantage of MOCA’s training on improved management and finance while other cooperatives did not? How should activities be planned or implemented differently to enhance cooperative adoption of advanced management practices and financial solutions? SUSTAINABILITY 13. Thinking about the positive changes created by the MOCA project, what changes do you think your cooperative will continue to practice in the future and which ones might reverse course? 14. Was the MOCA project able to establish strong collaborations between your cooperative and farmers, and businesses, financing institutions, and the government institutions in a way that you can continue these connections after MOCA ends? 15. What could be done more to increase the productivity and efficiency of your farmers and cooperative? What could be done more to improve and expand the trade of cocoa and cocoa products? 16. Imagining an ideal situation, what kind of support would be necessary to continue the positive changes in the future? 6.7.3 Financial Institutes, Governmental Representatives, and Exporters Introduction Please state your name, organization, and position, and for what period. Please briefly describe some of your main activities and responsibilities. How did you and your organization become involved in the MOCA project? What were the primary avenues through which you and your organization supported the MOCA project and beneficiaries? IMPACT 1. Based on your experience, what were some of the biggest impacts MOCA had in Côte d’Ivoire, thinking especially about the two main strategic objectives: increase productivity and efficiency of actors in the cocoa value chain, and improve and expand the trade of cocoa through improved productivity, handling, and market linkages? Final Evaluation Report : MOCA Côte d'Ivoire 104 2. What key changes, positive or negative, resulted because of the MOCA project activities toward these goals? 3. How has partnering with MOCA changed inputs and/or outputs related to cocoa production, marketing, and selling in Côte d’Ivoire? 4. How has partnering with MOCA changed your engagement with producers and other stakeholders from the cocoa value change? RELEVANCE 5. What are some of the largest challenges and strongest needs of the cocoa sector in Côte d’Ivoire right now, thinking of the different levels, from government to farmers? 6. Have MOCA’s activities focused on these actual needs? 7. Have MOCA activities been able to support these needs? EFFECTIVENESS Prompt: Remind the informant of the eight project activities: 1. Producer groups and cooperatives capacity building 2. Government institutions capacity building 3. Business development services capacity building 4. Facilitate agricultural lending and financial services 5. In-kind grants for equipment and input 6. Develop agrodealers and other input suppliers 7. Training for improved agricultural production techniques 8. Facilitate buyer-seller relationships for market access Ask them to respond to the next set of questions mostly regarding activities they are involved with. 8. What were some of the most effective activities conducted by MOCA and its partners? 9. Thinking about these successes, what were some of the key strategies that contributed to these successes? 10.What were some of the weaker or more challenging activities supported by the project, and in what way? 11.How well were project activities able to synergistically support each other? 12.Who do you consider the most important partners and/or stakeholders supporting MOCAs activities toward achieving project goals? Are there important partners the project has not engaged with? 13.Were MOCA activities able to improve the performance of assisted cooperatives, and in what way? 14.To what extent did improved performance of assisted cooperatives where able to improve conditions for farmers? In addition to other outcomes, think of how access to finance and value and volume of sales have changed. Final Evaluation Report : MOCA Côte d'Ivoire 105 15.In which way have project activities able to improve: • Linkages along the cocoa value chain? • Private – public collaborations? • Business development of cooperatives and companies? • Marking of improved cocoa varieties? • Funding for participants of the cocoa value chain? EFFICIENCY 16.What were some of the challenges and benefits of engaging multiple partners in MOCAs activities? 17.How efficient were the mechanisms of the various units and management levels in supporting the implementation of the project? • Program management • Program staff • Monitoring and Evaluation • Grantmaking • Coordination with partners 18.How did the leveraging of funds through Public-Private Partnerships add value to the project? SUSTAINABILITY 19.Thinking about the positive changes created by the MOCA project, what changes do you think will continue in the future and which ones might reverse course? 20.Was MOCA project management able to establish strong collaborations between cooperatives, farmers, businesses, financing institutions and the government institutions in a way that these connections will continue to be active after MOCA ends? 21.What could be done more to increase the productivity and efficiency of actors in the value chain? What could be done more to improve and expand the trade of cocoa and cocoa products? 22.Imagining an ideal situation, what kind of support would be necessary to continue the positive changes in the future? In addition to the above questions, the following questions are specific questions for ANADAR: 23.How well were you able to meet your initial work plans, including around: • Improving farmers weather resilience • Establishing farm business plans • Sustainable cocoa production 24.Has MOCA been able to address your needs and requirements for capacity building, and support your services? This includes staff capacity strengthening and any physical inputs. Final Evaluation Report : MOCA Côte d'Ivoire 106 25.How was the experience working with MOCA in terms of signing service provision contracts with exports and cooperatives? 26.Do you feel that the training you provided to cooperatives and other organizations through the MOCA project was-? • Relevant to the needs of the cocoa value chain of the country? • Planned in a way that efficiently addressed those needs? • Anything you would have changed? Final Evaluation Report : MOCA Côte d'Ivoire 107 6.8 Appendix 8: Statistical Tables of Farmer Survey Results Final Evaluation Report : MOCA Côte d'Ivoire 108 Final Evaluation Report : MOCA Côte d'Ivoire 109 Final Evaluation Report : MOCA Côte d'Ivoire 110 Final Evaluation Report : MOCA Côte d'Ivoire 111 Final Evaluation Report : MOCA Côte d'Ivoire 112 Final Evaluation Report : MOCA Côte d'Ivoire 113 Final Evaluation Report : MOCA Côte d'Ivoire 114 Final Evaluation Report : MOCA Côte d'Ivoire 115 Final Evaluation Report : MOCA Côte d'Ivoire 116 Final Evaluation Report : MOCA Côte d'Ivoire 117 Final Evaluation Report : MOCA Côte d'Ivoire 118 Final Evaluation Report : MOCA Côte d'Ivoire 119 Final Evaluation Report : MOCA Côte d'Ivoire 120 Final Evaluation Report : MOCA Côte d'Ivoire 121 Final Evaluation Report : MOCA Côte d'Ivoire 122 Final Evaluation Report : MOCA Côte d'Ivoire 123 Final Evaluation Report : MOCA Côte d'Ivoire 124 Final Evaluation Report : MOCA Côte d'Ivoire 125 Final Evaluation Report : MOCA Côte d'Ivoire 126 Final Evaluation Report : MOCA Côte d'Ivoire 127 6.9 Appendix 9: MOCA Final Evaluation Scope of Work The final evaluation, conducted by the Contractor, will focus on the impact of MOCA activities on beneficiaries in terms of increasing productivity and expanding trade of cocoa products. The rationale for conducting the final evaluation is to assess different aspects of the project including the overall project design, effectiveness of implementation, the quality and structure of the project management, best practices learned from project implementation and their replicability on future projects both in Cote d’Ivoire and elsewhere. The final evaluation will also assess the impact of the MOCA project on the intended beneficiaries, with a focus on direct beneficiary cocoa producers, in terms of success in increasing productivity and expanding trade of cocoa products. The final evaluation follows the baseline evaluation completed by the KIT Royal Tropical Institute in September 2019, which established baseline information including baseline values for the performance indicators for cocoa farmers in the target intervention areas. The baseline evaluation report is available on the Development Experience Clearinghouse.1 Specific Issues or Questions to be Addressed in the Evaluation: Following the Monitoring and Evaluation Policy guidance for USDA programs, MOCA’s final evaluation will assess: Relevance – The extent to which the project interventions met the needs of the project beneficiaries and is aligned with the country’s agriculture and/or development investment strategy and with USDA and US Government’s development goals, objectives, and strategies. Relevance should also address the extent to which the project was designed taking into account the economic, cultural and political context and existing relevant program activities. Effectiveness – The extent to which the project has achieved its objectives. Effectiveness should also assess the extent to which the interventions contributed to the expected results or objectives. Efficiency – The extent to which the project resources (inputs) have led to the achieved results. An assessment of efficiency should also consider whether the same results could have been achieved with fewer resources or whether alternative approaches could have been adopted to achieve the same results. An Impact Assessment of the medium and long-term effects, both intended and unintended, of a project intervention. Effects can be both direct or indirect and positive or negative. To the extent possible, the evaluation should assess the extent to which the effects are due to the project intervention and no other factors. Sustainability – Assessment of the likelihood that the benefits of the project will endure over time after the completion of the project. Sustainability should also assess the Final Evaluation Report : MOCA Côte d'Ivoire 128 extent to which the project has planned for the continuation of project activities, developed local ownership for the project, and developed sustainable partnerships. Impact - To what extent did MOCA activities increase cocoa farmer access to improved services and inputs, loans, and financial services; increased productivity or value; and income (sales) Impact should also assess to what extent can these impacts be attributed to MOCA interventions as opposed to other factors. The evaluation will also consider the effect of COVID-19 and 2020 electoral process in Cote d’Ivoire, assessing the extent to which these factors impacted the project’s relevance, effectiveness, efficiency, sustainability and impact. Deliverables The Contractor shall deliver to CNFA the following deliverables, in accordance with the schedule set forth in Section below. All deliverables shall be submitted electronically, in both PDF files and MS Word or MS Excel files. All deliverables should be submitted in English unless otherwise specified. As appropriate, all photographs or other graphics/figures in the reports will also be submitted as separate editable files. Finally, the Contractor should also submit the final clean dataset used to inform the analysis. Deliverable No. 1: Protocol of the evaluation (including work plan, data collection plan including the Cocoa Cooperative & Farmer Survey design/sampling plan, and data treatment and analysis plan) Before commencing the quantitative survey, the Contractor will be required to submit a protocol of the study that includes: Workplan of all activities - For the Cocoa Farmer Survey, a sampling plan, including sample design, sampling frame, sample size calculation, level of statistical precision and power, and respondent selection procedures for both the direct beneficiary and comparison group. Training and field manuals for supervisors and enumerators. Plans for supervisor, enumerator, and data entry training; questionnaire translation, back-translation, pretesting, and finalization; and questionnaire piloting. Data treatment and analysis plan, including estimation procedures (sample weighting and other adjustments) and indicator tabulation and subgroup analysis. Deliverable No. 2: Finalized data collection instrument(s), in French The Contractor will develop the questionnaire using a participatory approach to engage stakeholders in designing the questions and ensure that the tool is appropriately contextualized for MOCA intervention areas and populations. The Contractor will hold a workshop with stakeholders including managers of Final Evaluation Report : MOCA Côte d'Ivoire 129 cooperatives and staff of partner structures (CCC, ANADER, OLAM, etc) for design and validation of the data collection instruments, including the survey questionnaire as well as the focus group formats and questions. The Contractor will also ensure that the data collection tools are gender-sensitive. Questions should be inclusive and gender￾sensitive data collection techniques must be used to ensure that women are able to participate in the survey; such techniques may include women-only focus groups, and the use of female enumerators to interview women to ensure female respondents are fully included in the survey and results. The Contractor will be required to submit the survey questionnaires and other data collection instructions (such as for key informant interviews and focus groups) for validation with MOCA before starting of the survey. The data collection instruments must take into account all the information necessary for the successful completion of the activity. After validation of questionnaire, contractor must make data entry tool in French. Deliverable No. 3: Training Report & training manual(s) for field staff Before data collection begins, training of data collectors and supervisors will be held. Contractor will submit a training report on this activity. This report will consider information about training activities, a training manual(s) for field staff in French, level of knowledge of data collector, monitoring documents for data collection activities, expectations, and recommendations of Contractor to CNFA for the successful completion of the activity. Deliverable No. 4: Database The contractor will deliver to CNFA the database of the survey with all raw data files in SAS, SPSS, or STATA format (sampling frames, raw data sets, transformed data sets and syntax, edit rules, code book/data dictionary, sampling weights). This file must take into account calculation tools of indicators and metadata. Deliverable No. 5: Draft Evaluation Report The Contractor will submit a draft evaluation report, with preliminary information. At a minimum, the Draft Evaluation Report will include the Executive Summary, Introduction and Purpose, Evaluation Design and Methodology, and Findings and Recommendations. The following information must also be included: Tables of indicator estimates along with their confidence intervals and standard errors from the Cocoa Cooperative & Farmer Survey. Interpretation of the results: Sample size and number of observations Final Evaluation Report : MOCA Côte d'Ivoire 130 Coefficients and standard errors for the dependent and independent variables R2 - coefficient of determination Root Mean Squared Error Statistical significance of equation (F-test) and estimates (t-test) or corresponding p￾values Statistical significance of equation (F-test) and estimates (t-test) or corresponding p￾values Deliverable No. 6: Final Evaluation Report The Contractor will submit the final evaluation report at the end of the Contract. The final Report must incorporate CNFA’s feedback and address CNFA’s comments on the first draft. It must take into account all information about the evaluation methodology, results, and recommendations. At a minimum, the Final Report will contain the following sections: Executive Summary, Introduction and Purpose, Evaluation Design and Methodology, and Findings and Recommendations. Final Evaluation Report : MOCA Côte d'Ivoire 131 6.10 Appendix 10: Results Framework