This publication was produced at the request of the United States Department of Agriculture. It was prepared independently by Nicaraguan Foundation for Economic and Social Development (FUNDIDES) Foreign Agricultural Service, United States Department of Agriculture A collage of four photographs showing cacao and cattle production in Nicaragua Nicaragua Food for Progress Project FINAL EVALUATION November 2019 DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. PROGRESA Caribe Final Evaluation Report Program: Food for Progress Agreement Number: FCC-524-2014-056-00 Funding Year: Fiscal Year 2014 Project Duration: 2014-2019 Implemented by: Catholic Relief Services Evaluation Authored by: The Nicaraguan Foundation for Economic and Social Development (FUNIDES) Alvaro López-Espinoza, Camilo Pacheco Magaly Sáenz Laureano Arcia Jafet Baca Gabriela Orozco PROGRESA Caribe Final Evaluation Report Report prepared by the Nicaraguan Foundation for Economic and Social Development (FUNIDES)1 at the request of Catholic Relief Services (CRS) Managua, November 15, 2019 1 This document has been prepared by the Services Unit of FUNIDES (hereinafter Evaluation Team). The work team has been led by Alvaro López-Espinoza, under the supervision of Camilo Pacheco. The research team included Magaly Sáenz and Laureano Arcia. Jafet Baca and Gabriela Orozco provided research assistance. Logistics assistance during the field work phase was provided by CRS MEAL’s team. Project Title: Nicaragua FFPr PROGRESA Caribe Donor United States Department of Agriculture Implementing Organization: Catholic Relief Services Agreement Number: FCC-524-2014-056-00 Start Date: October 1, 2014 End Date: May 31, 2020 Reporting Period: 2014 - 2019 Contact: Jorge Brenes, Chief of Party Tel: (505) 2278 3808 E-mail: Jorge.brenes@crs.org Table of Contents List of tables............................................................................................................. 5 List of figures ........................................................................................................... 6 List of graphs ........................................................................................................... 6 Acronyms and abbreviations.................................................................................. 7 Executive summary ................................................................................................. 8 1. Introduction ................................................................................................... 11 2. Description of the program .......................................................................... 13 2.1. Strategic objectives....................................................................................................... 13 2.2. Program activities ......................................................................................................... 13 2.3. Rationale for the evaluation .......................................................................................... 16 3. Objectives of the evaluation ......................................................................... 18 3.1. Development of the hypothesis and theory of change .................................................. 18 3.2. Evaluation questions..................................................................................................... 18 3.3. Main intermediate and outcome indicators................................................................... 20 4. Evaluation design .......................................................................................... 21 4.1. Quantitative design at farmers level.............................................................................. 22 4.1.1. Eligibility criteria, targeting and selection of beneficiaries................................. 22 4.1.2. Impact evaluation design ...................................................................................... 24 4.1.3. Empirical strategy................................................................................................. 27 4.2. Participatory approach to evaluation............................................................................. 30 4.2.1. Learning Cycle self-assessment tool: ADA........................................................... 30 4.2.2. Evaluation of the stakeholders in the value chain: LINK tool.............................. 30 4.2.3. Survey on the satisfaction and perception of beneficiaries................................... 31 5. Impact of the Program on the Beneficiaries and Key Stakeholders........ 31 5.1. Impact of the program on farmers................................................................................. 31 5.1.1. Introduction........................................................................................................... 31 5.1.2. Characteristics of the producers........................................................................... 32 5.1.3. Producer practices ............................................................................................... 32 5.1.3.1. Cattle................................................................................................................. 32 5.1.3.2. Cacao ................................................................................................................ 35 5.1.4. Infrastructure and equipment................................................................................ 39 5.1.5. Productive yields................................................................................................... 40 5.1.6. Association between productive practices and productive yields......................... 41 5.1.6.1. Cattle producers................................................................................................ 42 5.1.6.2. Cacao producers............................................................................................... 44 5.1.7. Sustainability of practice application ................................................................... 46 5.1.8. Commercialization................................................................................................ 47 5.1.9. Farmer’s perceptions on financial education....................................................... 51 5.1.10. Wellbeing of producer families......................................................................... 52 5.1.10.1. Sociodemographic profile of the producer families.......................................... 52 5.1.10.2. Socioeconomic wellbeing.................................................................................. 53 5.2. Producer organizations performance ............................................................................ 57 5.2.1. Management of producer organizations (ADA Tool) ........................................... 57 5.2.2. Business relationships with buyers (LINK Tool) .................................................. 60 6. Conclusions and lessons learned.................................................................. 62 6.1. The relevance of the program ....................................................................................... 63 6.2. The efficiency of the program....................................................................................... 63 6.3. Effectiveness and impact of the program...................................................................... 64 6.4. Good agricultural and commercialization practices...................................................... 65 6.5. Sustainability................................................................................................................. 66 6.6. Key recommendations for future interventions ............................................................ 66 References .............................................................................................................. 68 Annexes................................................................................................................... 70 Annex I. Changes in the activities of PROGRESA-Caribe ..................................................... 70 Annex II. Summary of the results framework.......................................................................... 74 Annex III. Matrix of intermediate and Outcomes indicators ................................................... 76 Annex IV. Detailed explanation of the evaluation design........................................................ 78 Quantitative design at farmers level.................................................................................... 78 Participatory approach to evaluation ................................................................................. 84 ADA Learning Cycle self-assessment tool........................................................................... 85 Annex V. Detailed explanation of the empirical strategy at farmers level .............................. 89 Sampling strategy ................................................................................................................ 89 Data collection .................................................................................................................... 91 Assessment of the data’s quality and comparability over time ........................................... 92 Data collected...................................................................................................................... 93 Attrition................................................................................................................................ 96 Revalidation of the evaluation design ............................................................................... 102 Econometric method.......................................................................................................... 103 Annex VI. Index of agricultural practices.............................................................................. 108 Annex VII. Production value for the program ....................................................................... 110 Annex VIII. Statistical tests ................................................................................................... 111 Final Evaluation Report PROGRESA-Caribbean Page 5 of 112 List of tables Table 1. Distribution of the beneficiaries of PROGRESA-Caribe ......................................................... 23 Table 2. Beneficiaries by treatment assignment ..................................................................................... 27 Table 3. Average yields per producer..................................................................................................... 40 Table 4. Application of practices for the management of livestock (percentage on producers)............. 43 Table 5. Differences in productive yields of producers.......................................................................... 44 Table 6. Application of practices for the cacao production (percentage of producers).......................... 45 Table 7. Differences in productive yields of producers.......................................................................... 46 Table 8. Commercialization outcomes.................................................................................................... 48 Table 9. Revenues ($) per ha .................................................................................................................. 49 Table 10. Productive values (US$ millions)........................................................................................... 50 Table 11. Evolution of cattle income during the program (US$ millions) ............................................. 51 Table 12. Financial knowledge assessment (percentage of producers) .................................................. 52 Table 13. Household size ........................................................................................................................ 53 Table 14. Demographic characteristics of the families........................................................................... 53 Table 15. Socioeconomic outcomes........................................................................................................ 54 Table 16. PPI (%) transition matrix ........................................................................................................ 55 Table 17. Producers organizations selected for ADA tool...................................................................... 58 Table 18. Score of ADA tool by producer organization ......................................................................... 59 Table 19. Score of ADA tool by area...................................................................................................... 60 Table 20. Producer organizations selected for LINK tool ...................................................................... 60 Table 21. Score of LINK tool by producer organization ........................................................................ 61 Table 22. Score of components of LINK tool......................................................................................... 62 Final Evaluation Report PROGRESA-Caribbean Page 6 of 112 List of figures Figure 1. Intervention areas of PROGRESA Caribbean......................................................................... 11 Figure 2. Comparative timeline of treatment allocation and data collection .......................................... 12 Figure 3. Treatment selection process..................................................................................................... 26 List of graphs Graph 1. Sanitary practices..................................................................................................................... 33 Graph 2. Livestock management ............................................................................................................ 34 Graph 3. Farm management practices..................................................................................................... 34 Graph 4. Crop genetics............................................................................................................................ 35 Graph 5. Pest management activities...................................................................................................... 36 Graph 6. Disease management................................................................................................................ 37 Graph 7. Fertilization practices............................................................................................................... 38 Graph 8. Farm management practices..................................................................................................... 38 Graph 9. Livestock - Infrastructure and equipment ................................................................................ 39 Graph 10. Cacao - Infrastructure and equipment.................................................................................... 40 Graph 11. Share of the producers trading commodities.......................................................................... 48 Graph 12. Land Transactions.................................................................................................................. 56 Graph 13. Socioeconomic effects ........................................................................................................... 57 Final Evaluation Report PROGRESA-Caribbean Page 7 of 112 Acronyms and abbreviations ADA Learning Cycle Self-Assessment Tool ASIHERCA Asociación de Iniciativas de Hermanamiento de El Castillo ATET Average treatment effect on the treated BCIE Banco Centroamericano de Integración Económica CDSO Crude degummed soybean oil CRS Catholic Relief Services CENAGRO National Agricultural and Livestock Census COMPROMUB Cooperativa multisectorial de productores orgánicos de Muelle de los Bueyes COODEPROSA Cooperativa Multisectorial de Desarrollo Productivo del San Juan COOMUNSOL Cooperativa Multisectorial Nuevo Sol de Yaoya R.L. COOPROCAFUC Cooperativa de Productores de Cacao Familias Unidas de El Castillo COOSEMUCRIM Cooperativa de Servicios Múltiples de Cacao en la Reserva de Indio Maíz COOSEMUP Cooperativa de Servicios Múltiples “Unión de Paiwas” COOSEMUVIS Cooperativa de Servicios Múltiples de Villa Siquia, R.L. COOMULBAN Cooperativa Multisectoral Banacruz, R.L. COOPELACTME Cooperativa de Producción de Lácteos Mejía R.L. COOSAGRO Cooperativa de Servicios Agropecuarios San Carlos R.L. EMNV Encuesta Nacional de Hogares sobre Medición de Nivel de Vida FIDA Fondo Internacional de Desarrollo Agrícola FUNIDES Fundación Nicaragüense para el Desarrollo Económico y Social HDDS Household Dietary Diversity Score ICT Information and communication technologies IDB Inter-American Development Bank IPM Integrated pest management INIDE Instituto Nacional de Información de Desarrollo LWR Lutheran World Relief MAHFP Months of adequate household food provisioning MEAL Monitoring, evaluation, accountability and learning MEFCCA Ministerio de Economía Familiar, Comunitaria, Cooperativa y Asociativa NICADAPTA Programa de apoyo a la adaptación al cambio climático mediante la producción de café y cacao de pequeños productores en zonas agroclimáticas aptas OCSA Soil and water conservation practices PPI Progress out of Poverty Index PROCACAO Programa Mejoramiento de las Capacidades Organizativas y Productivas de los Productores y Productoras de Cacao en el Triángulo Minero PROGRESA Program for rural enterprise management, health and the environment RACCN Región Autónoma de la Costa Caribe Norte RACCS RCT Región Autónoma de la Costa Caribe Sur Randomized Controlled Trial TNS TechnoServe UCA Ahmed Campos Unión de Cooperativas Ahmed Campos USAID United States Agency for International Development USDA United States Department of Agriculture Final Evaluation Report PROGRESA-Caribbean Page 8 of 112 Executive summary The Program for Rural Enterprise, Health and Environment – Caribbean Zone (PROGRESA Caribbean) was a five-year (2014–2019) value chain strengthening program for the cacao and livestock (dual purpose) sectors in the Caribbean Coast Region of Nicaragua: North Caribbean Coast Autonomous Region (RACCN, by its acronym in Spanish), South Caribbean Coast Autonomous Region (RACCS, by its acronym in Spanish), Jinotega, Río San Juan and Chontales. The Food for Progress program, financed by the United States Department of Agriculture, had a total budget of US$10,254,800 and was implemented by Catholic Relief Services (CRS) in consortium with TechnoServe (TNS) and Lutheran World Relief (LWR). The design of PROGRESA Caribbean considered the economic, political, and cultural context of the intervention areas. A high percent of the population of these areas live in poverty, and historically, these areas have received fewer resources for socio-productive activities. The program targeted 5,108 small cacao and/or livestock farmers from 14 municipalities of the Caribbean Coast, Río San Juan and Chontales. The activities were focused on existing farmers, instead of promoting new plantations. The program also worked on strengthening 23 producer organizations and associations, and to promote 11 commercial relationships under inclusive business models. The program aimed to increase the income of the farmers families and to reduce poverty in the intervention areas. Project activities were aimed at fulfilling two strategic objectives: • Increase agricultural and livestock productivity. • Expand trade of cacao and cattle products. The final evaluation of PROGRESA Caribbean followed a mixed approach; in other words, it incorporated both quantitative and qualitative analysis. Both methods were complementary to each other. The quantitative approach followed a quasi-experimental methodology that measured the impact of the program on the welfare of the families and the performance of the producers. The qualitative approach assessed the impact on organizations of both producers (cooperatives) and private companies as well as beneficiaries' satisfaction of the services provided by the program. The implementation of the program was approximately 36 months. The program was designed in two phases for comparison purposes; with an early treatment group and a late treatment group. The intervention strategy proposed by the Evaluation Team made it possible to randomly identify a temporary counterfactual group and to randomize the duration of the program for each of the beneficiaries at farmer level. The first phase of the intervention started in October 2015 and the second phase started in May 2017, both ending in September 2019. This strategy is called randomized order of phase-in, which included a randomized selection of beneficiaries that received the treatment in the first (early treatment) and second (late treatment) phases. Final Evaluation Report PROGRESA-Caribbean Page 9 of 112 The Evaluation Team concluded that the overall performance of the program was positive in terms of meeting its objectives. The cacao producers increased their productivity (due to the intervention) and total output, and cattle producers increased their volume of beef and milk. These results were not only due to higher yields but also because producers expanded areas under production (1.72 ha/production in baseline to 1.96 ha/production in end-line). The commercialization rates of milk and cattle increased by 24.8 percent and 36.7 percent respectively at the end-line. In other words, more producers were selling their products rather than consuming them, which indicates an essential shift in their production management. Cattle production value, on the other hand, went down between baseline and end-line driven by both the drop-in quantity sold/produced and the drop-in prices. Given that the productive yields of cattle increased (kg/hectare), the Evaluation Team concluded that the reduction in quantity traded can be attributed mainly as a price response, where producers withheld the commodity waiting for better prices. Cacao production value, on the other hand, went up driven by an increase in quantity. The Nicaraguan economy suffered a contraction of 3.8 percent due to the socio-political crisis that broke out in April 2018. Agricultural production suffered a deceleration going from an annual growth of 6.3 in 2017 to 3.3 in 2018. Unrest and road blockades around the country, caused by the sociopolitical crisis, affected the internal and external markets, forcing processing plants to limit buying cattle and raw milk from producers for more than two months. This situation had a negative effect on prices leading to a 5.4 percent contraction in the cattle sector in 2018 compared to 2017. The evaluation team analyzed the evolution of cattle income to assess if the decline in income was consistent with the timing of the crisis. The results of the data collected by the program MEAL team, showed that cattle income had been increasing prior to the economic crisis, which was consistent with the improvement in the families' welfare indicators. Despite the crisis in 2018, all welfare characteristics of the farmers and their families improved during the program. Poverty incidence changed from 60.1 percent at baseline to 52.9 percent at end￾line for the early treatment group, and from 66.6 percent at baseline to 55.1 percent at end-line for the late treatment. In other words, poverty incidence was reduced by at least 7 points in 4 years, which is more than the average poverty reduction rate according to the last available official information2 . The poverty reduction was driven primarily by 3 factors. First, there was an increase of 13 points in the number of households that had sanitary facilities, reducing the number of households that did not have any type of sanitary services. Second, the number of households sending all school age children to school went from 33 percent to 51 percent. Lastly, at end-line only 18 percent of the households had 7 or more members, a reduction of 9 points compared to baseline. 2 The National Institute for Development Information (INIDE, by its acronym in Spanish) estimated that poverty incidence in Nicaragua changed from 29.6 percent in 2014 to 24.9 percent in 2016; this represents a reduction of 4.7 base points in the mentioned period. Final Evaluation Report PROGRESA-Caribbean Page 10 of 112 During the life of the program, families increased their monetary resources, compared to baseline, which could explain the improvement in indicators like nutrition, education, and housing. Although migration may have had some impact on household size, there could be other reasons for reduction in average household size (natural deaths, marriages, etc.,), and therefore, would require further follow up to determine direct causality between participation in the program and the incidence of poverty. It is important to mention that while the poverty incidence of the country increased during 2018 and 2019 as a result of the economic crisis (20% in 2017 to 30% in 2019), the poverty incidence of the families in the program reduced. Additionally, the number of families that underwent food shortages for some month during the year changed from 34 percent at baseline to 14 percent at end-line for the early treatment group, and from 33 percent at baseline to 17 percent at end-line for the late treatment group. This means that for both groups, the number of families that underwent food shortages during some month of the year was reduced by half. Final Evaluation Report PROGRESA-Caribbean Page 11 of 112 1. Introduction The Program for Rural Enterprise, Health and Environment – Caribbean Zone (PROGRESA Caribbean) was a five-year (2014–2019) value chain strengthening program for the cacao and livestock (dual purpose) sectors in the Caribbean Coast Region of Nicaragua: North Caribbean Coast Autonomous Region (RACCN, by its acronym in Spanish), South Caribbean Coast Autonomous Region (RACCS, by its acronym in Spanish), Jinotega, Río San Juan and Chontales. Figure 1. Intervention areas of PROGRESA Caribbean The Food for Progress program, financed by the United States Department of Agriculture, had a total budget of US$10,254,800 and was implemented by Catholic Relief Services (CRS) in consortium with TechnoServe (TNS) and Lutheran World Relief (LWR). The program was designed in two phases for comparison purposes: with an early treatment group and a late treatment group. The intervention strategy proposed by the Evaluation Team made it possible to randomly identify a temporary counterfactual group and to randomize the duration of the program for each of the beneficiaries at farmer level. The first phase of the intervention started in October 2015 and the second phase started in May 2017, both ending in September 2019. This strategy is called randomized order of phase-in, which included a randomized selection of beneficiaries that received the treatment in the first (early treatment) and second (late treatment) phases. Final Evaluation Report PROGRESA-Caribbean Page 12 of 112 Figure 2. Comparative timeline of treatment allocation and data collection The program was built upon previous experiences of CRS and its partners in the Northern and Caribbean Coastal Regions of Nicaragua. It also worked with producers that participated in previous programs. The idea was to build a critical mass of technical knowledge and commercial capacity that would transform the cacao and livestock value chains in Nicaragua and allow producers to capture greater shares of income and profit as their value chains become better developed and their participation more active and organized. Although the initial target population was 4,250 farmers, the final number of beneficiaries was 5,108 small cacao and/or livestock farmers from 14 municipalities of the Caribbean Coast, Río San Juan and Chontales, exceeding the target by 20.2 percent. The activities were focused on existing farmers, instead of promoting new plantations. The program also worked on strengthening 15 producer organizations and associations, and to promote 11 commercial relationships under inclusive business models. The consortium partners collected quantitative and qualitative data at farmer and organization level for the baseline3 and mid-term. However, as suggested by the United States Department of Agriculture (USDA) in the mid-term evaluation, the final evaluation data was collected by the Evaluation Team. The collected data allows one to evaluate the evolution of intermediate and outcome indicators and to estimate the final impact of the program. The aim was to identify lessons learned and recommend strategic actions to improve future related interventions. This report presents the results of the final evaluation of the PROGRESA Caribbean program in Nicaragua. The document is comprised of eight sections including; 1) the Introduction; 2) Background 3 The Evaluation Team randomly supervised 10% of the baseline data collection at farmers level. Baseline survey Midterm survey End-line survey Participants in early treatment effectively enter the program Participants in the second stage enter the program Randomization in treatment allocation 0 1 18 19 Final Evaluation Report PROGRESA-Caribbean Page 13 of 112 description; 3) Objectives of the evaluation; 4) Quantitative and qualitative evaluation design; 5) Sampling strategy and the data collected; 6) Validation of the evaluation design; 7) Impact of the program on the beneficiaries and key stakeholders, and finally; 8) Lessons learned and recommendations. 2. Description of the program PROGRESA Caribbean worked to strengthen the value chain of cacao and (dual purpose) livestock in the Caribbean Region of Nicaragua. The program benefited 5,108 smallholder producers from 14 municipalities: Waslala, Siuna, Bonanza and Rosita in RACCN; El Ayote, La Cruz de Río Grande, Muelle de los Bueyes, Bocana de Paiwas, El Rama, and Nueva Guinea in RACCS; San Carlos, El Castillo in Rio San Juan; Jinotega; and Santo Domingo in Chontales. Furthermore, the program worked to strengthen 15 producer organizations and associations and promote 11 commercial relationships under inclusive business models. The duration of the program activities was approximately 36 months. 2.1. Strategic objectives The purpose of the program was to fulfill two strategic objectives: 1. Increase agricultural and livestock productivity. 2. Expand trade of cacao and cattle products. Improvement in productivity was accomplished by providing necessary inputs, facilitating genetic material for cacao (grafting) and cattle (artificial insemination), individual credit to the producers and through cooperatives, as well as technical assistance and training. These interventions enabled producers to improve their production techniques and enhance the management of their farms. The expansion in trade of cacao, dairy products, and cattle was achieved through training producers and cooperatives in business management; facilitating construction and improvement of the post-harvest processing infrastructure; increasing access to markets; promoting the participation in national tasting contests; and establishing lasting public-private and private-private alliances, including improved services in the value chain, and leveraging resources from these relationships. 2.2. Program activities The Evaluation Team reviewed the donor agreement and the results of the final evaluation and according to our observation the program operated in line with the scope of its objectives: 1. Increase agricultural productivity: increase productivity of the cacao and dual-purpose livestock value chains. 2. Expand trade of agricultural products: expand trade of cacao, dairy products, and cattle. Final Evaluation Report PROGRESA-Caribbean Page 14 of 112 The program modified some activities during its implementation, which were approved under a previous amendment, allowing the consortium partners to focus on their combined strengths in addressing the needs of the producers and their organizations and respond to the realities on the ground4 . Activities to increase agricultural productivity in the targeted value chains were focused on training to improve agricultural production techniques and farm management, as well as fomenting farm￾level financial and non-financial services that producers needed to implement best practices. The program implemented ten types of interventions: 1. Technical assistance: group or individual extension techniques that favor communication between the extensionist and the farmer's family, through the exchange of ideas, data, and information, to provide comprehensive solutions to the existing problems or to enhance the production system. 2. Trainings: theoretical and practical sessions that aim to transmit knowledge through a curriculum targeted to respond to a specific problem. 3. Field schools: group teaching from project technicians to farmers in the same geographical area that brings together concepts and methods. The process is based on the experience and development of communities, with the support of new practices and technologies to improve production systems. 4. Workshops: group work that aims at improving a certain task, studying, reflecting and working to achieve an improved product (a document, a technology, an instrument, a project, a proposal). 5. Informative talks: is a method that is informative and dynamic, where the facilitator transmits or communicates verbally a given topic, defining, analyzing, and explaining it, with audiovisual and written means, to promote interest and interaction between participants. 6. Provision of goods and services: access to goods and services that support productive activity and the value chain in general. These goods and/or services can be targeted to groups or individuals, who sometimes make contributions that cover a percentage of the value of the goods based on subsidy policies according to the purchasing capacity of the program beneficiaries. 7. Dissemination through ICT, specifically through mass media. 8. Participation in fairs, is a space where buyers and suppliers sign letters of intent, access updated technical information, contacts with financial and nonfinancial service providers are established, etc. 4 See Annex I. Changes in the activities of PROGRESA-Caribbean Final Evaluation Report PROGRESA-Caribbean Page 15 of 112 9. Field days: is a method of communication with groups, which tend to highlight one or several agricultural practices, carried out in a local setting, in order to raise interest and the desire to adopt improved practices, resulting in the transmission of technology and knowledge. 10. Exchange tours: visit of a group of farmers to another farmer or group of experienced producers to learn about a successful experience about a particular practice or crop. An exchange visit consists of organizing a meeting between a group of visiting producers with another host with the objective of exchanging opinions and views on a specific crop or topic. Additionally, the program facilitated the creation of community promoters to guarantee the continuity of the technical assistance in the form of local leaders. The promoters were trained in using a learning by doing methodology and developing a set of soft skills. A total of 57 producers completed the promoter training and 40 of them completed additional training and were certified by a local university. Moreover, the program supported 30 young adult children of the producers to receive technical training relevant to their family’s productive activity (cacao and/or cattle). Lastly, the program assisted groups of young local entrepreneurs to strengthen their business skills and business plans, which focused on the creation of providing services for the value chains. Final Evaluation Report PROGRESA-Caribbean Page 16 of 112 2.3. Rationale for the evaluation At the global level, there is a growing tendency to formulate evidence-based development strategies, which has been accompanied by a change from an input-based approach to a results-based approach. This has made it possible to expand specific development projects to other contexts and to a larger scale, improve budgetary planning, better direct development policies, and improve the credibility of reporting on interventions (Gertler et al., 2011). Monitoring and evaluation are vitally important to this, as they also help to improve efficiency and effectiveness in the intervention results throughout its different phases. There were many reasons for the evaluation of PROGRESA Caribbean, both for the program itself as well as for the context of agricultural and livestock programs that have been implemented in Nicaragua. Regarding the former, the project supported farmers in the Caribbean Coast regions, Río San Juan and Chontales, because these territories (which are mainly rural) present a high incidence of Box 1. Monetization During PROGRESA-Caribe, Catholic Relief Services (CRS), the leader of the consortium, monetized 11,497.7 MT of crude degummed soybean oil (CDSO). Since the country is not self-sufficient in oil for human consumption, CRS identified in 2013 that CDSO was a viable commodity to be monetized in Nicaragua. At that time, three companies, with relatively small capacity each, were identified as possible buyers. These companies together imported CDSO for decades from Pasternak Baum; a U.S. company based in New York that was willing to buy the product from CRS and sell to the traditional buyers in the country, assuming the cost of insurance for shipping. Using a commercialization channel already established in Nicaragua, it allowed companies involved (seller & buyers) to work with CRS and plan sales during their normal calendar and at the same time maintain a steady supply of the commodity for the country. For these reasons, PROGRESA-Caribe sold CDSO throughout the life of the project to ensure enough funds to support the program. During the last sale (early April 2017) a small impasse occurred due to an inappropriate handling of the product during the loading process at the US Port. Therefore, 51 metric tons of CDSO (1.46% of the total amount) had to be destroyed in Nicaragua for having been deposited in vessel tanks that did not pass a certificate of cleanliness during the loading process. That represented a total loss of US$40,000. Despite this impasse, the evaluators consider CDSO being a viable option for monetization in Nicaragua given that it is a needed commodity and was able to generate the necessary proceeds for the program. Final Evaluation Report PROGRESA-Caribbean Page 17 of 112 poverty5 and historically, these areas have received fewer resources for socio-productive projects. In addition, these territories are areas of priority to USDA in Nicaragua, where the consortium partners have extensive experience. The PROGRESA Caribbean program was unique in its dual focus on cacao and cattle production systems. This combination may lead to more dynamic opportunities for generating income for the farmers, as the farmers diversify their sources of income, ensuring a relatively stable flow of earnings during the year. Nevertheless, there is no previous evidence for Nicaragua that documents the socio￾economic impact of a program of this type, much less in the regions where the intervention was undertaken. The evaluation of this program is of special interest, taking the above into account and given that the cacao and dual-purpose cattle value chains are important to the geographic regions selected, as the Fourth National Agricultural and Livestock Census (CENAGRO) in 2011 found that 61 percent of cacao farms and 24.1 percent of cattle farms in the country are located in this region6 . The programs’ success, as well as the lessons learned from it, may lead to future related interventions in the region or adapted to other contexts (for example, municipalities). In terms of accountability, the evaluation informs USDA about whether its resources were well invested (upward accountability). This may have an impact on its decision to finance similar interventions in the future or, in fact, may provide input to attract new donors. Even more important, the evaluation makes it possible to inform the beneficiaries and other key stakeholders (cooperatives and enterprises) about the intermediate and final results of the program (downward accountability). This will also enable the key stakeholders to have a better understanding of the perspectives of the project and of the contribution of each to the achievement of the objectives (Bonbright, 2012). The context of agricultural and livestock programs in Nicaragua, provides the justification for the evaluation because most of these programs have not been adequately evaluated in the country. The existing evaluations generally tend to analyze the results of the beneficiaries’ ex-post compared to the ex-ante outcomes or to compare the beneficiaries to different populations. These methods do not make it possible to identify the effect of the interventions relative to the outcomes for the beneficiaries if the intervention had not occurred. The sparse number of formal evaluations of agricultural and livestock programs in Nicaragua have been carried out on projects financed by the United States Agency for International Development (USAID), the World Bank and the Inter-American Development Bank (IDB). They include the Rural 5 According to the 2014 Living Standards Measurement Study (LSMS) published by INIDE the incidence of poverty (using a poverty line of $1.8 dollar a day) in the Caribbean Coast is 39 percent of the population and in the rural areas is 50.1 percent of the population. 6 The estimate of 24.1 percent does not include the Río San Juan and Chontales province, because the program only worked with farmers in the cacao value chain in Rio San Juan and with cattle in Chontales. For the case of Jinotega, the program worked with producer organizations. Final Evaluation Report PROGRESA-Caribbean Page 18 of 112 Business Development Project of the Millennium Challenge Account (USAID), the Agro-food Production Support Program of the Nicaraguan Government (IDB) and Phase II of the Agricultural Technology Project of the Nicaraguan Government (World Bank). The first two evaluation designs were experimental, while the third was based on a quasi-experimental methodology (matching estimator and difference in differences estimator). Due to the nature of the Progresa Caribbean program (support for productivity and commercialization) and its geographic scale, this evaluation is a valuable opportunity to contribute to improving the culture of evaluation of agricultural and livestock programs in the country. 3. Objectives of the evaluation The final evaluation addresses whether the program had achieved its objectives, as indicated in the results framework7 . The report responds to the key evaluation questions and verifies the compliance and the quality of the services provided by the program. It also presents the main results and recommendations to the consortium partners, USDA, and other key actors, which later can be used for future interventions. 3.1. Development of the hypothesis and theory of change The theory of change of PROGRESA-Caribbean was built from the results framework, detailed in the program´s implementation plan, and from the indicators established in the partnership agreement between CRS and USDA. The theory of change was modeled in a results chain, where the causal logic was defined from the beginning of the project to the end, taking into account the following elements: activities (the “how”), products (the “why”), results (the “if”), and final results (the “then”). Based on the results of the value chains, the Final Evaluation sought to confirm the following hypothesis: If the program activities as a whole improved agricultural and livestock productivity and if the trade of cacao, dairy and beef products expanded, then this will increase the income of the farmers’ families and will contribute to poverty reduction in the intervention areas. The hypothesis tested must identify, to the extent possible, the effect of the intervention as a whole in the outcomes specified: family income and poverty levels. 3.2. Evaluation questions The impact evaluation provides responses to key questions about the relevance, effectiveness, efficiency, impact and sustainability of the program, as well as specific questions about learning in the value chains that are important to USDA, the local partners, the donor community and other key stakeholders. For the final evaluation, the questions to be answered included: 7 See Annex II for a summary of the results framework by strategic objective. Final Evaluation Report PROGRESA-Caribbean Page 19 of 112 1. Key evaluation questions Relevance: ▪ To what extent did the project meet the needs of the project beneficiaries and take into account the economic, cultural and political context? Effectiveness: ▪ To what extent has the project achieved its stated objectives? ▪ Did the project increase farmers’ productivity in cacao and dairy/beef products? ▪ Did the project enhance the quality and safety of the food produced by participating farmers? ▪ Did the project increase participating farmers’ sale of products? ▪ Did the project increase participating farmers’ family income? ▪ Did the project reduce the incidence of poverty in the intervention areas? Efficiency: ▪ Was the monetization process efficient and effective? ▪ Was the monitoring system designed efficiently to meet the needs and requirements of the project? ▪ Was the coordination with project partners and other programs and activities carried out by the main stakeholders, effective and efficient? Impact: ▪ Evaluation of the intentional or unintentional final effects of the project and, to the extent possible, evaluation of the degree to which the effects are attributable to the project and not to other factors. Sustainability of outcomes and impacts: ▪ To what extent are the project’s activities foreseen as sustainable? ▪ Will project activities continue in the absence of support from both USDA and consortium partners? If so, how? If not, why not? ▪ Will the effects of the project on producers, producer organizations, partner organization sustain overtime after the project ends? ▪ To what extent are the outcomes and impacts of the program resilient to risk over time? 2. Key value chain learning questions Final Evaluation Report PROGRESA-Caribbean Page 20 of 112 ▪ What is the return on investment (technology, infrastructure, etc.) that the participating families (and their organizations) achieved as a result of their participation in the value chains? ▪ What is the relationship between infrastructure investment and changes in sales and income at both the household and cooperative level? ▪ How do the different enterprise models established for value chain development, affect impacts at the producer and cooperative levels? ▪ In what form do the enterprise models generate incentives for private sector businesses to invest in productive infrastructure and provision of services (technical and financial) for smallholder producers? ▪ What is the household-level impact of affiliation with cooperatives that are connected with value chains? 3.3. Main intermediate and outcome indicators The indicators to be evaluated included those (intermediate) results that were achieved when the population perceived the benefits of the program and the final outcomes of the intervention, corresponded to the strategic objectives. In addition, the implementation indicators agreed upon with USDA were evaluated. These have all been monitored since the program’s baseline study ensuring constant evaluation of the value chains and better identification of the causes for the program results at final evaluation. The following indicators were analyzed: Outcome indicators - Progress out of poverty index. - Family income. - Household dietary diversity score. - Months of adequate supply of food in the household (MAHFP). - Return on investment for the farmer and processing activities of the cooperatives. - Number of individuals that received financial services as a result of USDA assistance. Intermediate indicators - Volume of animal origin product selected per animal per unit of time (liters of milk per cow per day). - Volume of selected crops harvested per hectare (yield in metric tons of cacao). - Average weight of cattle per hectare. - Value of production – cacao. Final Evaluation Report PROGRESA-Caribbean Page 21 of 112 - Value of production – cattle. - Gross margin per hectare – cacao. - Gross margin per hectare – cattle. - Number of hectares managed under sustainable agricultural practices. Annex III presents the matrix of intermediate and outcome indicators that were evaluated, including a description of the indicator, calculation formula (operationalization), measuring frequency and means of verification. 4. Evaluation design8 PROGRESA Caribbean final evaluation followed a mixed-methods approach; in other words, it incorporated both quantitative and qualitative analysis. Both methods were complementary to each other. Bamberger (2012) states that this approach allows for: 1. Triangulation of the results of the evaluation, which strengthens the validity and credibility of the results. • Using information from one method to develop the instrument for another. • Complementarity in the results. • Diversity in the value dimensions of the evaluation; and generates new knowledge about the evaluation results. • Generating new knowledge into evaluation findings. Furthermore, the evaluation followed an Evaluative Thinking approach as suggested in Archibald, Sharrock, Buckley and Cook (2016), which implied close interaction with the Monitoring, Evaluation, Accountability and Learning (MEAL) and the Knowledge Management Division of CRS. The Evaluation Team was able to conduct several sessions to exchange information about the program with CRS and its partners. This made it possible to more deeply understand the assumptions of the program set forth in the theory of change and in the mechanisms for the implementation of activities, to respond to the evaluation questions, and to recommend future actions. This section is divided in two parts. The first part discusses the methodology used to measure the impact of the program on the welfare of the families and the performance of the producers (quantitative approach). The second part addresses the participatory approach to the evaluation (qualitative approach) that assessed the impact on organizations of both producers (cooperatives) and private companies as well as beneficiaries' satisfaction on the services provided by this program. The qualitative methods were 8 Annex IV presents a detailed explanation on the design. Final Evaluation Report PROGRESA-Caribbean Page 22 of 112 complementary to the (quantitative) experimental evaluation of the impact of the program on the beneficiary’s outcomes. 4.1. Quantitative design at farmers level 4.1.1. Eligibility criteria, targeting and selection of beneficiaries The program targeted the RACCN, the RACCS, Río San Juan, Jinotega and Chontales, because as mentioned in the previous section, these territories (which are mainly rural) present a high incidence of poverty and historically, these areas have received fewer resources for socio-productive projects. The targeted municipalities were selected after consultation processes with local organizations and implementing partners, local leaders and other key stakeholders. The project benefited 5,108 smallholder producers from 14 municipalities: Waslala, Siuna, Bonanza and Rosita in RACCN; El Ayote, La Cruz de Río Grande, Muelle de los Bueyes, Bocana de Paiwas, El Rama, and Nueva Guinea in RACCS; San Carlos, El Castillo in Rio San Juan; and Santo Domingo in Chontales. The selection of beneficiaries was based on the following criteria: a) cacao growers, dual purpose cattle ranchers, and producers that worked on both value chains, whose production was fundamental to their livelihood strategy; b) producers that had between 5 and 100 head of cattle in production (and less than 20 hectares allocated to livestock production) and/or with less than 5 hectares for agriculture production; c) families that were willing to actively participate in the program; and d) identification of women farmers who worked in these chains in order to achieve 20 percent participation by women in the program. According to the program beneficiary selection criteria, beneficiaries could be supported in one or two value chains, if they fulfilled the selection criteria. The program also targeted farmers who had participated in prior interventions in the area with the consortium partners and who met the selection criteria (repeated beneficiaries). In practice, the selection of the communities was based on the geographic location of the producers that met the eligibility criteria and who accepted being part of the program. The consortium partners initially targeted more than 4,000 farmers to participate in PROGRESA Caribbean. At baseline, a total of 4,041 producers were registered, and by the mid-term evaluation the total number of beneficiaries had increased to 4,134; this represents 93 more farmers compared to the baseline. According to CRS monitoring data through May 20199 , the number of active producers in the program had increased to 5,164, which is equivalent to 1,123 more farmers compared to the baseline. The distribution of beneficiaries by value chain and treatment phase was the following: 9 This information was prior to the final evaluation. Final Evaluation Report PROGRESA-Caribbean Page 23 of 112 Table 1. Distribution of the beneficiaries of PROGRESA-Caribe 31.0 percent of the beneficiary producers were supported in the dual-purpose cattle chain, 48.9 percent in the cacao chain, and 20.1 percent in both chains. Geographically, most of the selected cacao producers were in Waslala and El Castillo, while most of the livestock farmers were in the RACCS (mainly in the municipalities of Nueva Guinea and Bocana de Paiwas). The selection of the universe of beneficiaries of PROGRESA Caribbean generated two types of potential selection bias, due to: 1) administrative rule10 (based on observable characteristics) and 2) self￾selection; this is explained in more detail in the Annex IV. Although the identification of beneficiaries did not follow a random selection as in traditional experimental designs, the proposed strategy allowed for randomly identifying a temporary comparison group and also to randomize the duration of time in the program. The assumption is that both treatment 10 Selection bias by administrative rule implies that the program targeted beneficiaries based on observable characteristics, such as gender, location, socio-economic status, among others. Table 1. Distribution of the beneficiaries of PROGRESA-Caribe Both Cacao Cattle Total Both Cacao Cattle Total Both Cacao Cattle Total Both CacaoCattle Total Both Cacao Cattle Total CRS 321 295 149 765 194 179 42 415 29 84 28 141 – – – – 544 558 210 1312 LWR – 795 – 795 – 428 – 428 – – – – – 78 – 78 – 1301 – 1301 TNS 190 254 757 1201 128 25 300 453 21 16 97 134 – – – – 339 295 1154 1788 Total 511 1344 906 2761 322 632 342 1296 50 100 125 275 0 78 0 78 883 2154 1364 4401 Source: Based on CRS data. Total Partner Early treatment Late treatment New beneficiaries Member of cooperatives Final Evaluation Report PROGRESA-Caribbean Page 24 of 112 and comparison groups are exposed to the same environmental conditions given the context and targeting in the Caribbean Coast and Río San Juan11, which are locations where the farmers tend to face the same environmental conditions. 4.1.2. Impact evaluation design The key aspect of all impact evaluations is to identify what would be the outcome of the beneficiaries in the absence of the program. Obviously, it is not possible to observe the counterfactual outcome because individuals can only have one treatment status (being treated or not). In the case of PROGRESA Caribbean, the selection of beneficiaries did not follow an experimental design because it was very complicated to have a “pure” counterfactual outcome in agricultural and livestock programs, as only certain producers can be selected to participate for ethical reasons, program objectives and budgetary issues (Winters et al., 2010). Since the activities did not start at the same time with all producers, the Evaluation Team proposed an intervention strategy that made it possible to randomly identify a temporary counterfactual group and to randomize the duration of the program for each group of beneficiaries. This strategy is called randomized order of phase-in (Duflo, Glennerster and Kremer, 2007) and was applied to the case of Nicaragua in the Social Protection Network Program (see Maluccio and Flores, 2004) and in the Rural Business Development program of the Millennium Challenge Account (see Carter, Toledo and Tjernström, 2012). The consortium partners implemented a two-phase intervention. The first one started in October 2015 and the second one in May 2017, both ending in September 2019. The beneficiary selection that received the treatment in the first (early treatment) and second (late treatment) phases was randomized. Nevertheless, one of the main problems in agricultural programs are the spillover effects (or contamination), i.e., late treatment could be indirectly exposed to the program (Winter et al., 2010). For example, if a producer is selected to be early treatment while another from the same community corresponds to late treatment, the latter can learn from the producer that is receiving technical assistance and/or any other types of services from the early treatment group. This would generate a contaminated comparison group and could dramatically underestimate the program effects. Recently, the experimental approach in several agricultural programs has been designed to capture spillover effects for eligible and non-eligible units (double randomization). However, PROGRESA Caribbean was not designed for this purpose. Therefore, in order to diminish the contamination risk in the program, it was decided to randomize the treatment allocation at the level of 65 groups (clusters) of adjacent communities (cluster randomization). These clusters (adjacent communities) were created in coordination with the consortium partners based on geographical location criteria. Within each group, 11 At the time the Evaluation Team designed the evaluation, Jinotega and Chontales were not part of the targeted municipalities. Final Evaluation Report PROGRESA-Caribbean Page 25 of 112 the communities were adjacent and not among the targeted groups, since the geographical distances are very far. At this level the communities within each group had the same probability of being selected so the risk of contamination was marginal. The consortium partners stressed during the evaluation design that in order to meet the goal of reaching a certain number of beneficiaries per year they had to initiate activities with more than 50% of beneficiaries, to take into account dropouts. It was therefore decided a (cluster) random selection of 65 percent for early treatment and 35 percent for late treatment, which generated an early and late assignment of 42 and 23 clusters, respectively. At beneficiary level, this implied 2,676 beneficiaries for early treatment (63%) and 1,571 for late treatment (37%). Since all clusters (adjacent communities) had a similar opportunity to be part of the early treatment, we had a treatment group (early treatment) and a temporary comparison group (late treatment) with similar characteristics, as reported in the baseline. For the final evaluation, the first cohort becomes the treatment group and the second one corresponds to the comparison group, since the first group benefited longer from the same interventions. A successful experiment requires a clear identification strategy. In this regard, the Evaluation Team explained the experimental design and its challenges to the consortium and implementing partners, so that they could inform all beneficiaries and producer organizations about the randomized treatment allocation process. The diagram below presents the treatment selection process12 . 12 It is worth noting that information regarding the excluded producers was not available because they did not comply with the selection criteria or declined to participate. Final Evaluation Report PROGRESA-Caribbean Page 26 of 112 The random cluster13 assignment generated the following beneficiary distribution by treatment status, value chain, region, sex and repeating14 farmers: 13 As explained earlier, the clusters selected are groups of adjacent communities. 14 Farmers who met the selection criteria and had participated in prior interventions in the area and supported by the consortium partners. Treatment selection process Source: Baseline Report of PROGRESA Caribe (2014). Figure 3. Treatment selection process Final Evaluation Report PROGRESA-Caribbean Page 27 of 112 It is important to mention that at the beginning of the program the early treatment group was supposed to be exposed to the intervention twice the time of the late treatment. However, the Evaluation Team found that farmers of both treatment groups participated on average the same amount of time on activities. Despite this, the Evaluation Team considered that those farmers who started early were able to devote more time to putting into practice the learning and had greater follow up by community trainers and extension technicians. Randomized Control Trials (RCTs) are very challenging to implement, given potential spillover and attrition, especially in large scale programs. As explained later in this document, the Evaluation Team found a bias in the evaluation sample during the final round, identifying attrition as correlated with treatment status. This made the RCT strategy no longer viable. Therefore, it was decided to apply a quasi-experimental approach. Specifically, the combination of two techniques was used to isolate the bias: difference-in-differences and matching estimators. This is addressed in the next section. 4.1.3. Empirical strategy15 The information for the final evaluation at farmers level comes from a panel data that compiles the characteristics of producers and their families for two evaluation rounds: baseline and final. As explained in the Annex V, mid-term data was not considered for the final evaluation due to comparison difficulties. The collected data allows one to evaluate the evolution of intermediate and outcome indicators and to estimate the final impact of the program at farmers level. The aim was to identify lessons learned and recommend strategic actions to improve future related interventions. 15 Annex V presents a detailed explanation on this strategy. Table 2. Beneficiaries by treatment assignment F M F M F M F M F M F M F M F M Both chains 25 194 15 87 28 139 12 Cacao 32 171 11 81 38 242 75 296 17 127 65 175 3 5 Livestock 9 99 5 27 108 571 3 16 Total 66 464 31 195 38 242 0 0 75 296 17 127 201 885 6 33 Both chains 21 127 3 43 84 476 4 Cacao 17 94 7 61 36 161 43 124 21 43 10 Livestock 4 35 3 13 140 1 Total 42 256 10 107 36 161 0 0 43 124 21 43 97 626 0 5 Note: F = female, M = male. Source: Based on PROGRESA Caribe monitoring records. Value chain Late treatment Early treatment RACCN RACCS Río San Juan RACS New Repeaters New Repeaters New Repeaters New Repeaters Table 2. Beneficiaries by treatment assignment Final Evaluation Report PROGRESA-Caribbean Page 28 of 112 The sampling strategy was designed to allow the collection of the necessary data to answer key evaluation questions and to analyze the effects of the intervention on intermediate and final outcomes. The results from the baseline suggested a representative sample of 980 observations (490 for each treatment status). Assuming a rate of non-response (attrition) of slightly more than 20 percent the total planned sample was 1,210 producers (605 for each treatment status). The list of program participants was used as the framework for the sample design. The selection of the sample was based on simple random sampling. Also, a random sample of 14 percent of replacements was estimated, which was distributed proportionally in the geographic areas of the program. The data collection for the final evaluation was exclusively the responsibility of the Evaluation Team. Logistics assistance during the field work phase was provided by CRS and its partners. The consortium partners collected data at farmers level for the baseline and mid-term with technical assistance from the Evaluation Team16. However, it was recommended by USDA that the Evaluation Team be responsible for the data collection at the end line (final evaluation). The final data sample was lower than the planned sample in the baseline, because of attrition during the program. It was comprised of 537 complete interviews that were obtained (77% of the planned sample), of which 313 (58%) corresponded to early treatment and 224 (42%) to late treatment. The program had dropouts from the evaluation sample for both early and late treatment, during the life of implementation. The Evaluation Team found that this affected the statistical power required to identify statistical differences between groups. The Evaluation Team asked the CRS MEAL team to document the reasons for attrition in the evaluation sample. The reasons for attrition were varied, but the main reasons included: 1. Loss of interest while waiting to join the late treatment. 2. Initiation of work on the government’s ProCacao and NICADAPTA projects while waiting to join late treatment group. 3. Migration to other areas of the country or abroad. 4. Change of economic activity17 . 5. Distance and insecurity. 6. Natural Deaths. The program started with 23 cooperatives and ended with 15, with 8 dropping out. According to the CRS MEAL team some of the reasons for attrition amongst the cooperatives were associated with 16 The Evaluation Team provided comments on the instrument and randomly supervised 10% of the field work. 17 These farmers decided to not focus on cacao or livestock and instead focused on growing other agricultural products or work outside the farm. Final Evaluation Report PROGRESA-Caribbean Page 29 of 112 failing to maintain or secure their registration with the government, opportunity to join government or other NGO funded programs, and the economic situation in the country. The Evaluation Team ran several statistical analyses and concluded that the observed attrition correlated with treatment allocation, at least in the case of the cattle value chain. In the late treatment "the best ones remained", that is, those farmers with the best productive performances. This bias in the evaluation sample for the final round made the RCT strategy no longer viable. Therefore, it was decided to apply the combination of two techniques in order to isolate the bias: difference-in-differences (which isolates time invariant characteristics that might be correlated with treatment status) with matching estimators (which mitigates the bias by observable characteristics). Firstly, given that the data had been collected for two periods (baseline and final line), the Evaluation Team was able to exploit the panel data structure by using the difference-in-differences methodology. The first difference corresponds to the difference in the after-and-before outcomes for the treatment group (early treatment), which controls for time-invariant characteristics. The second difference relates to the difference in the after-and-before outcomes for the comparison group (late treatment). Subtracting the first difference from the second, isolated the effect of time-varying factors, under the assumption that both treatment and comparison groups were exposed to the same environmental conditions (parallel trend assumption); as mentioned earlier. This method provided a more precise estimate of the impact and allowed the treatment and comparison group to differ in characteristics, provided that the parallel assumption trend was accomplished and that the unobservable characteristics of the beneficiaries were time invariant. As noted earlier, the balance in observable characteristics between the early and late treatment groups was affected by attrition (nonresponse) of the evaluation sample. The above-mentioned technique was combined with matching estimators, which is one of the most popular techniques in the impact evaluation literature when the experimental design is affected by attrition; it is considered as a quasi-experimental method. This method evaluates the effect of an intervention by comparing outcomes between similar groups. The comparison group is determined to be a suitable match by measures of observed characteristics. Difference-in-difference combined with matching estimators compare the change (over time) in outcomes for treatment to the change in outcomes for the comparison group member (Ravallion & Jalan, 1999). In the context of this evaluation, the idea behind matching estimators was to statistically identify (based on observable characteristics) two “identical” individuals in the data, with the exception that one was treated earlier and the other individual was treated later, so that any difference in the result between the two may be attributed to the program. Matching requires the assumption that the selection in the program is only based on observable characteristics, which also affect the expected results (conditional independence assumption). If this assumption is fulfilled, conditional on a set of covariates, the allocation of treatment is random. Final Evaluation Report PROGRESA-Caribbean Page 30 of 112 Therefore, it is assumed that two individuals with the same characteristics, one treated and the other not, have the same counterfactual (Caliendo and Kopeing, 2008). The key in this method lies on the careful identification of the “clone” for the treated individual. However, as mentioned before, there might be time invariant characteristics that are correlated with treatment status, so this technique is combined with the difference-in-differences estimation in order to avoid this potential bias. The information related to the econometric method applied for this evaluation is explained in Annex V. 4.2. Participatory approach to evaluation The consortium partners implemented the program using different participatory methodologies with the producers, cooperative members and staff, territorial leaders and private sector representatives. The methodologies relied on assigning scores based on opinions and/or perceptions from the beneficiaries and stakeholders (producer organizations and enterprises). The qualitative methods applied were complementary to the (quantitative) experimental evaluation of the impact of the program on the beneficiary’s outcomes. 4.2.1. Learning Cycle self-assessment tool: ADA The final performance evaluation of the producer organizations was based on the application of the Learning Cycle self-assessment tool18 (ADA, by its acronym in Spanish). This tool enabled the producer organizations to conduct a quick analysis of their business management and organizational process. Gottret, Junkin and Ugarte (2011) state that this methodology aims at a participatory self￾assessment process by the leadership bodies, members, and the management, administrative and technical teams of the cooperatives.19 The tool was applied to five areas of assessment: 1) strategic orientation; 2) business management; 3) technical services; 4) structure and functionality; and 5) governance in partnership processes. ADA made it possible to compare the results of the self-assessments with the results obtained by other rural associative enterprises and among themselves over time. This tool was applied by the consortium partners during the baseline and mid-term and by the Evaluation Team for the final evaluation. The ADA tool was applied to 15 cooperatives at end-line. 4.2.2. Evaluation of the stakeholders in the value chain: LINK tool The stakeholders in the value chain are linked by commercial relationships in which each act as a buyer and seller for its neighboring links. The LINK methodology developed by Lundy, Becx, Zamierowsli, Amrein, Hurtado, Mosquera & Rodríguez (2012) was used to evaluate the degree of connection (inclusion) in the value chain. This methodology was mainly aimed at those stakeholders that 18 Facilitated self-assessment of the management of rural associative enterprises (RAE), (In Spanish ADA), which was developed by the Learning Alliance in Nicaragua. 19 More information about the ADA tool is available in Annex IV. Final Evaluation Report PROGRESA-Caribbean Page 31 of 112 play the role of facilitators for the processes between sellers and buyers, which was the role played by CRS and its partners.20 For the final evaluation, as in the baseline and the mid-term, the Evaluation Team and the consortium partners agreed to apply Tool #3, business model principles for inclusive businesses, to determine whether each business that links rural producers with buyers is truly inclusive. This tool assessed six principles: 1) collaboration among stakeholders; 2) effective market linkage; 3) transparent and consistent governance; 4) access to services; 5) inclusive innovation; and 6) measurement of results. The Evaluation Team applied the tool in each round. The objective was to obtain different perceptions and opinions and a better understanding of the context of the value chains at the end of the project. At the end-line, the tool was applied to 11 cooperatives and their main commercial partner. 4.2.3. Survey on the satisfaction and perception of beneficiaries The success of the program greatly depended upon the efficient and high-quality provision of technical support, business and financial services to the producer families, and the producer organizations. In this regard, the beneficiaries' perception on the quality of the services provided during the program was addressed through a survey on satisfaction and perception of the program. This instrument was developed by the Evaluation Team and constituted a module in the final evaluation. It covered the most and least-liked aspects of the program and rated each of the major activities developed. However, the satisfaction module uncovered some degree of heterogeneity across territories, mostly regarding terminology and the methodology of the knowledge transmission. CRS explained that the intervention approach was diverse due to different partner approaches, terminologies used, and the territorial extension of the program (more than 200 communities in 14 municipalities). This tool was complementary to the (quantitative) experimental evaluation of the impact of the program on the beneficiary’s outcomes and helped the Evaluation Team get inputs for the conclusions and lessons learned. 5. Impact of the Program on the Beneficiaries and Key Stakeholders 5.1. Impact of the program on farmers 5.1.1. Introduction The design of PROGRESA Caribbean considered the economic, political, and cultural context of the intervention areas. The implementation of the program was approximately 36 months. As activities did not start with all producers at the same time, the Evaluation Team proposed an intervention strategy that made it possible to randomly identify a temporary counterfactual group and to randomize the duration of the program for each of the beneficiaries at farmer level. The consortium partners 20 More information about the LINK tool is available in Annex IV. Final Evaluation Report PROGRESA-Caribbean Page 32 of 112 implemented a two-phase intervention. The first one started in October 2015 and the second one in May 2017, both ending in September 2019. The selection of beneficiaries that received treatment in the first (early treatment) and second (late treatment) phases was randomized This section discusses the comparison of the before (baseline) - after (end-line), in order to analyze the evolution of key indicators at the end of the program and assess the impact of the program on intermediate and outcome indicators, specifically yields and poverty incidence21. The idea was to obtain input on the services provided by the program to later infer the association between the program and the producer’s outcomes. 5.1.2. Characteristics of the producers The Evaluation Team was able to obtain information for 537 producers, of whom 322 (60%) corresponded to early treatment and 215 to late treatment (40%). Women made up 14.0% of the sample; this proportion was similar for both treatment groups. In terms of geographical distribution, 46.0% of the surveyed producers resided in the RACCS, 41.3% in the RACCN, and 12.6% in Río San Juan. There was a relatively greater participation of RACCS producers in the early treatment group, whereas in late treatment the largest proportion of producers was located in the RACCN. By value chain, 41.0% of the producers surveyed worked only with cacao, followed by those who worked with livestock (33.9%) and those who served both value chains (25.1%). 5.1.3. Producer practices22 23 The application of agricultural practices reflects the knowledge, skills, and abilities that the farm families acquired either on their own or by the efforts of the program. The application of these practices reflects the human capital of the producers. The sections below present the results for each value chain. 5.1.3.1. Cattle Deworming and vaccination were already among the practices with a high application at baseline, for both the early and the late treatment. At end-line, the application of those practices increased. Deworming, at end-line, was applied by almost every producer and the vaccination rate increased by 9 21 The effect on commercialization was not addressed because it was highly influenced by the market, which cannot be controlled by the consortium partners. Nevertheless, some aspects of commercialization that the program influenced are analyzed. 22 The analysis of producer practices over time was based on simple regressions of a given practice as a function of a binary variable that takes the value of 1 if the observation corresponds to the final round. Hence, the discussion of changes (increases, reductions) in the proportion of producers applying the different practices was based on "statistically significant" changes. 23 As mentioned in previous sections, the mid-term survey applied an evaluation period of 6 months. The consortium partners indicated that many of the practices evaluated could have had a downward bias due to the periodicity, as some practices were carried out only once a year. Considering this issue, the Evaluation Team decided to evaluate the practices comparing the baseline and the end-line. Final Evaluation Report PROGRESA-Caribbean Page 33 of 112 points among the early treatment group and 14 points among the late treatment group. Testing for diseases, specifically mastitis and tuberculosis were the practices with lower application, but at end-line, both application rates showed an improvement for both early and late treatment groups. Except for artificial insemination, producers improved significantly by adopting good cattle management practices. The share of producers milking in a clean environment was twice as high at end￾line compared to baseline. At the same time, more than half of the producers adopted cattle identification. Also, by end-line, the incidence rate of pasture rotation was almost 100 percent, while 3 out of 4 producers were dehorning their cattle. Graph 1. Sanitary practices Final Evaluation Report PROGRESA-Caribbean Page 34 of 112 The program also promoted improved farm management practices, which consisted of teaching producers how to keep track of different transactions like expenditures, sales, and investment. At baseline, the incidence of those practices was significantly low for both treatment groups. However, at end-line, the incidence rates of such practices increased similarly for the early and late treatment, as shown in the graph below. Graph 2. Livestock management Graph 3. Farm management practices Final Evaluation Report PROGRESA-Caribbean Page 35 of 112 5.1.3.2. Cacao For cacao production, the program focused on promoting the following practices: 1. Crop genetics. 2. Pest management. 3. Disease management. 4. Soil-related fertilization and conservation. In terms of crop genetics, the elimination of poorly developed buds was the most common practice among early and late treatment producers (97.4% and 98.3%, respectively at end-line). However, the practice that showed the most progress was rehabilitative pruning, around 70 percent of producers adopted this practice by end-line. Pest management activities were focused on promoting maintenance pruning and integrated pest management (IPM). Consistent with the crop genetics practices, by end-line, maintenance pruning was adopted by almost all the producers. Additionally, compared to baseline, the adoption of IPM at end-line was twice as high for those in early treatment and more than four times higher for those in late treatment. Graph 4. Crop genetics Final Evaluation Report PROGRESA-Caribbean Page 36 of 112 For disease management, the intervention promoted a Bordeaux mixture24 and disease control. In the early treatment group, this application was adopted by 1 in 5 producers, 9 times higher than at baseline. In the late treatment group, the application was adopted by almost 1 in 3 producers, 15 times higher than at baseline. On the other hand, disease control went from being applied by less than half of producers to being applied by more than 90 percent in both treatment groups. 24 Bordeux mixture is a combination of copper sulfate, lime, and water— an effective fungicide and bactericide that has been used for decades to control diseases of fruit and nut trees, vine fruits, and ornamental plants. These natural minerals, when mixed in the correct order, provide long-lasting protection to plants against diseases. Graph 5. Pest management activities Final Evaluation Report PROGRESA-Caribbean Page 37 of 112 The fertilization and soil conservation practices promoted during the program included: 1. Organic fertilization. 2. Soil and water conservation practices (OCSA, by its Spanish acronym). 3. Incorporation of green manure. 4. Ground cover management. 5. Green and foliar manures. At the baseline, the application of these practices was low. Ground cover management was the practice with the higher incidence and by end-line, the late treatment showed a significant increase going from less than 1 in 5 producers to more than 1 in 4 producers using this practice. For the early treatment group, there was not a significant increase. More importantly, the incidence of organic fertilization and green and foliar manures increased for both groups. Graph 6. Disease management Final Evaluation Report PROGRESA-Caribbean Page 38 of 112 Graph 7. Fertilization practices As mentioned earlier, the program promoted not only productive practices but farm management practices. The application rate at baseline for these types of practices was low, but by end-line this rate showed a modest increase. Notably, sales tracking was adopted by more than half of the producers by end-line. Graph 8. Farm management practices Final Evaluation Report PROGRESA-Caribbean Page 39 of 112 5.1.4. Infrastructure and equipment25 Infrastructure, tools and equipment represented the physical assets owned by the producer families. The program had subsidized the equipment transfer component in order to help producers with adoption of practices, reach more producers, and build sustainable relationships with agricultural suppliers. Regarding the livestock value chain, there were significant changes in the share of producers owning corrals, water tanks, water troughs and milking parlors. The changes were similar between the early and late treatment group. Graph 9. Livestock - Infrastructure and equipment The infrastructure of the producers treated in the cacao value chain saw a significant change in the proportion of producers owning the fundamental equipment necessary including fumigation pumps, chainsaws, toolkits and agricultural tools. Notably, the number of producers owning a toolkit almost doubled between baseline and end-line. 25 The infrastructure and equipment data base are broad. In this section, the analysis focuses on indicators selected by the Evaluation Team. Final Evaluation Report PROGRESA-Caribbean Page 40 of 112 5.1.5. Productive yields As a main goal, the program aimed to increase agricultural productivity. Activities to increase agricultural productivity were focused on training to improve agricultural production techniques and farm management, as well as promoting farm-level financial and non-financial services that producers needed to implement best practices. The table below shows the evolution of average yields per producer26 . In the livestock value chain, milk yields measured in liter produced by cow per day remained constant for the early treatment group, but a small decrease (in statistical terms) was observed in the late treatment group. On the other hand, the standard cattle weight by farm size significantly increased for 26 The indicators are calculated for producers who presented performance data for both the baseline and the end-line. Table 3. Average yields per producer Baseline Endline Baseline Endline Dry cacao yields (MT/Ha) 0.3 0.5 0.2 0.3 Milk yields (lts per cow per day) 2.4 2.4 2.8 2.4 Average cattle weight (kg/Ha) 389.7 493.9 398.2 508.8 Early Treatment Late Treatment Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Indicator Table 3. Average yields per producer Graph 10. Cacao - Infrastructure and equipment Final Evaluation Report PROGRESA-Caribbean Page 41 of 112 both treatment groups. Cacao yields on the other hand increased for both early and late treatment. In both cases, the increase was at least 30 percent. The Evaluation Team applied a difference-in-difference estimator combined with matching to approximate a causal effect because the internal validity27 of the experimental design was affected by the attrition rate reported earlier. The results suggest that the program increased cacao yields by 13 percent, but there was no statistically significant effect on cattle weights and milk production. 5.1.6. Association between productive practices and productive yields As previously mentioned, activities to increase agricultural productivity in the target value chains were focused on training to improve agricultural production techniques and agricultural management, as well as promoting financial and non-financial services at the farm level. The program helped farmers achieve an increase in productive yields through activities that increased the availability of improved inputs and increased farmers' knowledge about improved techniques, technologies, and farm management. In this framework, the increased productivity of cacao and livestock was due to the adoption of better practices and improved knowledge of technologies and techniques among producers. The program offered technical assistance using cascade training and demonstration plots. Following the recommendation made by USDA at the mid-term evaluation, an additional survey was carried out by the Evaluation Team at end-line to take a more in-depth look at the practices adopted by the farmers, focusing not only on the adoption but also on the intensity and quality of the practice. The following sections present the findings using this data. It is important to note that given the nature of this new data, only the end-line data was used to link productive practices to productive yields, but the data only helps to understand associations rather than attribute a causal effect. Using the information collected, two indexes28 were constructed to represent the application of the practices and their quality. These indexes were constructed following an axiomatic approach, establishing scores that were sensitive to the quality of the practices applied on the surveyed farms29 . Both indexes have a scale from 0 to 100, where 0 represents the non-performance of any of the practices evaluated, and 100 represents that all practices were carried out according to the recommendations given by the program. To evaluate the integral performance of the surveyed producers, it was suggested that the threshold to consider that the producers have applied the practices correctly is 60 points in both indexes. 27 In previous sections it was mentioned that the main objective of the experimental design was to eliminate differences between groups through random assignment of producers based on early and late treatment. 28 The methodology used to construct the indexes is presented in Annex VI, specifying the practices, activities, and scores defined. 29Following the recommendation made by USDA, in November 2018 research to take a more in-depth look at the practices adopted by the farmers was done resulting in the creation of these indexes. Final Evaluation Report PROGRESA-Caribbean Page 42 of 112 5.1.6.1. Cattle producers Cattle production is highly influenced by controllable factors like nutrition and disease prevention. Almost all cattle producers reported improved pastures, however, only 3 percent of them had the recommended 0.282 hectares of improved pastures per head of cattle. On the other hand, a relatively high portion of the producers provided supplemental nutrition with minerals to their cattle and most of them did it in the recommended frequency. Additionally, more than 95 percent of producers had water available to cattle all year round. However, less than 30 percent of the producers reported having water inside the corrals. The application of vaccination and deworming was high, more than 85 percent for vaccination and almost full compliance for deworming. However, vaccination quality indicators were not as high. For vaccination, none of the producers vaccinated twice against anthrax, and only about 30 percent transported the vaccines in the recommended container. Deworming was done mostly at the recommended frequency and was done along with the recommended bathing. Pasture management was the practice with the lowest compliance. Cattle producers implemented cleaning and fertilizing, but none of them had enough pastures, obstructing the compliance of rest days. The table below shows the indicators of intensity and quality of each practice. Final Evaluation Report PROGRESA-Caribbean Page 43 of 112 For cattle producers, the observed compliance on practice adoption resulted in an average general index of 55 points. This implies that on average cattle producers scored just above half of the possible points. The were no observed differences (in statistical terms) between early and late treatment groups. Cattle producers by far had better compliance in deworming practices. On average the producers scored 85.4 percent of the possible points. As discussed earlier, almost all the producers dewormed their cattle, but more importantly, the compliance on the recommended frequency was above 90 percent. Pasture management on average surpassed the compliance threshold. On average, cattle producers scored 61 percent of the possible points, whereas, both nutrition and vaccination practices did not reach the average compliance threshold. In both cases, producers scored less the 45 percent of the total possible points. For vaccination, producers struggled with the frequency of the anthrax vaccination and with the Early Treatment Late Treatment Improved grass 100.0% 99.3% Concentration of animals 3.5% 2.8% Mineral salts 79.2% 78.2% Mineral salts frequency 73.8% 65.5% Mineral salts ratio 26.2% 25.4% Salt quantity 51.0% 46.5% Water availability 97.5% 97.2% Water in the pens 25.7% 21.1% Number of pastures 13.4% 14.8% Days of grazing 31.7% 27.5% Days off 58.9% 59.2% Pasture cleaning 98.5% 99.3% Fertilizer 98.5% 99.3% Vaccines 85.6% 85.9% Anthrax vaccine twice in a year 0.0% 0.0% Vaccine transfer 30.7% 26.1% Needles reused 74.8% 74.6% Deworming 99.5% 99.3% Deworming frequency 95.5% 94.3% Washings are carried out 76.2% 73.9% Washing frequency 94.2% 98.1% Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Table 4. Application of practices for the management of livestock (percentage of producers) Nutrition Pasture management Vaccination Deworming Table 4. Application of practices for the management of livestock (percentage on producers) Final Evaluation Report PROGRESA-Caribbean Page 44 of 112 usage of proper containers to carry out vaccinations. In the case of nutrition, some struggled with providing each head of cattle with enough land of improved pastures. The differences between average yields for producers above and below the compliance threshold for each of the main practices show that those producers who surpassed the compliance threshold had on average (in statistical terms) higher milk yields than those who did not; the difference seems to be driven by nutrition practices. The results of cattle weight yields suggest that the relationship between productive practices and productive yields was insignificant. This could be due to exogenous factors associated with yields, such as the decision of farmers to increase meat production in a low-price environment. 5.1.6.2. Cacao producers Out of the four main practices, pruning had a higher application. For both treatment groups, the application was higher than 95 percent with “removing sucker” being the most common pruning technique. More than half of those in the early treatment group, and more than 60 percent in the late treatment group followed the recommended frequency for this technique. Handsaws were used by a significant share of the producers as pruning tools. Weed removal at the base of the trees was adopted by more than half of the producers, but only half of them adopted the practice as recommended. Descoping and trimming the top of the trees were activities with low incidence, and even for those who did it, only a small share followed the recommended frequency. The table below shows the indicators of intensity and quality of each practice. Table 5. Differences in productive yields of producers Producers above the threshold Producers below the threshold Diff Prob Ha: Diff≠0 Producers above the threshold Producers below the threshold Diff Prob Ha: Diff≠0 Nutrition 2.9 2.4 0.5 0.019 499.0 499.5 -0.5 0.995 Pasture handling 2.5 2.4 0.1 0.548 484.6 555.2 -70.5 0.326 Vaccination 2.3 2.5 -0.2 0.359 501.3 498.6 2.6 0.968 Deworming 2.4 2.5 -0.1 0.569 508.8 470.6 38.3 0.572 Cattle index 2.7 2.4 0.3 0.055 460.8 517.3 -56.5 0.369 Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Milk yields Cattle weight yields Practice Table 5. Differences in productive yields of producers Final Evaluation Report PROGRESA-Caribbean Page 45 of 112 The observed compliance and adoption rates resulted in an average general index of 50 points. Meaning that on average cacao producers scored half of the possible points. There were no observed differences between both treatment groups. Pruning was the practice with the better score, where on average producers scored 64.1 percent of the possible points assigned to pruning. Fertilization on the other hand was the practice with the lowest score, as producers on average scored 33.6 percent of the possible points. The low score was because it did not include any quality control over the practice, it just accounted for the application. Additionally, only a few producers reported compliance with the fertilization recommendation. Early Treatment Late Treatment Pruning 96.3% 96.5% Blunthing 82.0% 82.4% Blunthing frrequency 72.1% 66.7% Removing sucker 96.8% 95.3% Removing sucker freq. 45.2% 37.0% Removal of dead branches 92.6% 94.1% Removal of dead branches freq. 81.6% 74.5% Removal of crossing branches 90.8% 89.4% Removal of crossing branches freq. 74.6% 67.1% Machete not used 21.7% 26.5% Weed removal at the base of the trees 61.8% 54.1% Frecuencia caseo 52.2% 56.5% Cacao amendments 45.6% 46.5% Ash and lime amendment 46.0% 44.4% Amendment quantity 100.0% 100.0% Fertilizer Fertilizers used 31.3% 36.5% Shade handling 90.3% 87.6% Shade well-distributed 97.0% 96.6% Thinning 75.6% 73.5% Trimming the top of the trees 48.8% 48.2% Trimming the top of the trees twice at year 28.3% 23.2% Trimming the lateral branches 47.0% 40.6% Trimming the lateral branches freq. 20.6% 24.6% Shade setting 47.0% 40.0% Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Pruning Amendments Shade handling Table 6. Application of practices for the cacao production (percentage of producers) Table 6. Application of practices for the cacao production (percentage of producers) Final Evaluation Report PROGRESA-Caribbean Page 46 of 112 It was established that the producers who obtained 60 points or more in the practice index had the frequency and minimum quality requirements recommended by the program, of which30.23 percent of the producers met these parameters. Analyzing the difference between average yields for producers above and below compliance threshold for each of the main practices showed an insignificant relationship (in statistical terms) with productive yields. This could be because the index did not measure other important aspects in cocoa production such as soil type and genetics. 5.1.7. Sustainability of practice application As discussed previously, one of the program goals was to increase agricultural productivity. The long-term effects of the program depend mostly on the continuity of the application of the promoted practices. The producers for both the cacao and cattle value chains showed great willingness to continue applying the practices learned during the program. 99 percent of the cattle producers and 93.2 percent of the cacao producers said they wanted to continue applying the new knowledge and lessons learned, but many expressed some concerns about their ability to continue with the good practices adopted. One of the concerns expressed was the lack of access to technical assistance. For the producers, having the technician come to their farm was very valuable. Technicians not only taught the “know-how of the good practices”, but served almost as consultants, advising on specific issues. For the cattle value chain, technicians helped in veterinary diagnostics and treatment, a service that is scarce and expensive in the area. To ensure the continuity and expansion of the capacities developed, the program trained and certified community promoters. Community promoters are local leaders who voluntarily participated in the program as the main link with the producers in their communities. As the community promoters were not paid, they were incentivized in other ways. They received more than 120 hours of training, participated in farmer exchanges to share experiences, and were provided inputs such as pruning tools, grass cutters, and tools necessary for the establishment of demonstration plots on their farm. A cohort of promoters were also certified as agricultural promoters by the Universidad de las Regiones Autónomas Table 7. Differences in productive yields of producers Producers above the threshold Producers below the threshold Diff Prob Ha: Diff≠0 Pruning 0.381 0.401 -0.020 0.747 Amendments 0.458 0.364 0.094 0.168 Fertilization 0.457 0.357 0.100 0.119 Shade handling 0.402 0.381 0.021 0.734 Cacao index 0.456 0.363 0.094 0.159 Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Table 7. Differences in productive yields of producers Final Evaluation Report PROGRESA-Caribbean Page 47 of 112 de la Costa Caribe Nicaragüense (URACCAN). This was a major achievement for many of the promoters as it now makes them more credible extension agents in their communities. The objective of this component of the program was to create local capacity to reduce the gaps in technical assistance and transfer of agricultural technologies once the program ended. Local promoters were thus trained to become a support system for the implementation of good practices. Promoters were not only trained in technical knowledge, they were also trained in communication strategies, leadership, and different soft skills. The Evaluation Team believes that the combination of a knowledge centered program with active participation of the producers, the cost sharing strategy, and the community-promoter support systems, created the foundation towards sustainability. For future interventions, the Evaluation Team recommends that this type of local support system play a bigger role, assuring a good relationship between the promoter and the community. 5.1.8. Commercialization The program also aimed to expand the commercialization of agricultural products. The program focused on increasing market access for smallholder producers through improved post-harvest handing and processing, training on certifications and standards, and facilitating private and/or public partnerships for buying and selling, co-investment in infrastructure, and the exchange of market information. Complementary, foundational activities included increasing the capacity of cooperatives and trade associations to serve their smallholder members. Compared to the baseline, 22 percent more cattle producers were trading cattle by the end-line. The increase was more notable for milk trading, where there was an increase of 28 percent of the share of producers trading milk. For cacao, there was no statistically significant differences over time. This can be attributed to the differences in methodologies when collecting the data, as contrary to cattle production, the decision to work on cacao production was not one motivated by auto-consumption. That does not mean that there was no cacao auto-consumption, it just means that it was not commonly the exclusive production use. Producers may not have reported selling in a month because the production that month was not large enough or because they transformed cacao (dry cacao) to sell later. Final Evaluation Report PROGRESA-Caribbean Page 48 of 112 The table below presents the main commercialization outcomes. For comparability purposes, the estimations are done using those producers that traded the commodity in both baseline and end-line. Table 8. Commercialization outcomes Baseline Endline Baseline Endline Total Sales $ 4,854.0 4,206.2 3,529.5 4,068.4 Quantity (lts) 16,970.9 15,927.4 14,629.8 16,877.0 Price ($/lts) 0.3 0.3 0.2 0.2 Total Sales $ 8,632.6 3,746.6 7,497.0 4,124.3 Quantity (kg) 5,190.5 3,288.8 4,865.9 4,211.2 Price ($/kg) 1.9 1.3 1.9 1.4 Total Sales $ 814.1 879.6 815.1 824.7 Quantity (MT) 0.3 0.5 0.3 0.5 Price ($/MT) 2481.0 1837.9 2448.2 1854.8 Note: MT = metric tons; lts = liters. Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Early Treatment Late Treatment Milk Cattle Cacao Table 8. Commercialization outcomes Graph 11. Share of the producers trading commodities Final Evaluation Report PROGRESA-Caribbean Page 49 of 112 The cacao revenue grew (in statistical terms) modestly for both groups of producers30. The growth was slightly higher for the early treatment producers. In both cases, the modest revenue increase was driven by an increase in quantity sold, given that prices went down. In other words, the increase in quantity sold compensated the decrease in prices. For producers in the cattle value chain, the price of milk remained constant31. Therefore, the observed change in revenue was driven by quantity sold. Those producers in the early treatment decreased the quantity sold, resulting in a decrease in revenue. On the other hand, producers in the late treatment increased quantity sold and revenue. It’s important to note that the statistics discussed in this section correspond only to raw milk trade. That said, the majority of producers (99.1%) transform milk into cheese for sale or consumption. Lastly, revenue for cattle trading decreased for both groups of producers. Despite the increase in prices, in both cases, the revenue decrease was driven by a decrease in quantity sold. Comparing revenue among all producers, notably, cacao producers got more revenue per hectare. However, on average, cacao producers had less than two hectares, while cattle producers had more than 40. As a result, cattle producers had higher revenues. The cacao revenue increase was only significant for the early treatment, which as stated, increased because of the quantity sold. The production value for the program is extrapolated using the average total revenue by value chain and the target number of producers treated by the program. 32. The production value follows the same trends as the one discussed in commercialization. Cattle production value went down between baseline and end-line driven by both the drop-in quantity sold/produced and the drop-in prices. Given that the production yields of cattle increased (kg/hectare), the Evaluation Team concluded that a plausible explanation for the reduction in quantity traded could be attributed mainly as a price response, where producers withheld the commodity waiting for better prices. Some associations of agricultural producers, 30 The quantity sold is included. Sales are multiplied by the unit price to obtain the value of sales, which corresponds to sales revenue. 31 Prices are given in dollars, which account for part of the inflation. 32 The program-level indicator uses the sum of the income of all the producers under study. This indicator is extrapolated to the universe of early treatment. Table with calculations is presented in Annex VII. Table 9. Revenues ($) per ha Baseline Endline Baseline Endline Cattle (all products) 207.9 183.7 212.3 183.5 Cacao 642.2 789.8 557.0 574.2 Early Treatment Late Treatment Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Table 9. Revenues ($) per ha Final Evaluation Report PROGRESA-Caribbean Page 50 of 112 outside of the program, also reported this behavior as a response to the price reduction in 2019. Cacao production value, on the other hand, went up driven by an increase in quantity. The Nicaraguan economy suffered a contraction of 3.8 percent in 2018. As a result, Agricultural production experienced a deceleration going from an annual growth of 6.3 in 2017 to 3.3 in 2018. 33 The road blockades around the country, caused by the sociopolitical crisis, affected the internal and external markets, forcing processing plants to limit buying cattle and raw milk from producers for more than two months. This situation had a negative effect on prices leading to a 5.4 percent contraction in the cattle sector in 2018 compared to 2017. The below estimations observe two points in time (baseline vs end-line), not disclosing the true evolution of income during the program. As mentioned, the livestock sector was affected by the economic crisis in the country. The prices of cattle products decreased during the crisis, as revealed during the end-line evaluation. The evaluation 33 BCN (2019). Informe anual 2018. Managua: Banco Central de Nicaragua. FUNIDES (2019). Informe de Coyuntura 2018. Managua: FUNIDES Table 10. Productive values (US$ millions) Baseline End-line Variation Rate 36.40% Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Early Treatment $63.84 $54.88 -14.00% $31.47 $27.11 -13.90% $32.37 $27.78 -14.20% Total Late Treatment $57.18 $45.81 -19.90% Cacao Early Treatment $3.12 $5.05 61.90% Late Treatment $3.53 Total Total $4.02 13.90% $6.65 $9.07 $28.45 $22.06 -22.50% Late Treatment $28.83 $23.75 -17.60% Early Treatment Cattle Project Table 10. Productive values (US$ millions) Final Evaluation Report PROGRESA-Caribbean Page 51 of 112 team also analyzed the evolution of cattle income to assess if the decrease in income was consistent with the timing of the crisis. The results of the data collected by the program MEAL Team34 showed that cattle income had been increasing prior to the economic crisis, which was consistent with the improvement in the families' welfare indicators. 5.1.9. Farmer’s perceptions on financial education This section provides relevant information on the degree of farmer’s understanding on financial management. As the results were based on farmer’s perception, these do not provide relevant information on the outcomes of the program regarding farmer’s income, cost and profits. The following estimation presented corresponds only to the end-line results35 . Contrary to the results in previous sections, the percentage of producers who reported having positive profits from their economic activity in the 2018-2019 cycle was high for both early and late treatment. In both groups, more than 90 percent of the producers considered the economic cycle was profitable. The perception on the investment of profits revealed that for both early and late treatment, the most common destinations of profits were towards health, education and home improvements. Investment components related to farm management, such as the purchase of animals, land or equipment, were low. On the other hand, when analyzing how many of the early treatment producers surveyed considered keeping some type of register of the activities on their farm, it was found that only 2 in 5 kept any type of record/registry. 34 The estimation uses a balanced panel of producers in all four periods of the monitoring data. 35 The module is constructed based on 10 questions, developed by the consortium partners, to which a rating was assigned according to the indicated answers. It was applied for the first time during the mid-term evaluation. Median Income Average income April - September 2016 1.00 1.00 October 2016 - March 2017 1.72 1.41 April 2017 - September 2017 2.59 1.44 October 2017 - March 2018 2.47 1.52 Source: Based on monitoring data of PROGRESA Caribe. Table 11. Evolution of cattle income during the program (US$ millions (Base period: April-Sept 2016 = 1)) Table 11. Evolution of cattle income during the program (US$ millions) Final Evaluation Report PROGRESA-Caribbean Page 52 of 112 With reference to savings, half of the producers from the early treatment group reported to have savings, while 42 percent of the late treatment group had savings. Regarding access to credit, just 17 percent of the early treatment producers reported having access to some type of credit in the last year. For the late treatment group, only 11 percent of producers reported having access to credit. Finally, the general score of the financial knowledge assessment derived from the metrics of the test shows that there was no statistical difference in the financial knowledge between producers in the early and late treatment groups. Additionally, the average score was just above 75 percent, meaning that on average, most producers are financially proficient. 5.1.10. Wellbeing of producer families The PROGRESA Caribbean theory of change stated that if the intervention improves agricultural productivity and the trade of commodities, farmer’s income is supposed to increase, and thus contribute to poverty alleviation. Taking this into account, this section discusses the evolution of the final outcomes of the program, poverty, and food security. 5.1.10.1. Sociodemographic profile of the producer families On average, at end-line households had around 5 members. Those households in the early treatment group experienced an increase (in statistical terms) in the number of members compared to baseline. However, the number of large households, defined as households with seven members or more, decreased for both, early and late treatment. The share of large households went from 19.8 percent to Early Treatment Late Treatment Reported Profits 92.5% 89.8% Investment: Liquidate debt 24.2% 26.5% Investment: Home repairs/improvments 28.6% 28.8% Investment: Health 68.3% 64.7% Investment: Farm equipment 8.4% 7.9% Investment: Animals 18.9% 21.4% Investment: Land 9.9% 10.7% Investment: Education 38.5% 35.3% Accounting system 42.9% 43.3% Budget 26.1% 16.7% Savings 50.6% 42.8% Credit 17.4% 11.2% Financial score 75.8% 75.4% Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Table 12. Financial knowledge assessment (percentage of producers) Table 12. Financial knowledge assessment (percentage of producers) Final Evaluation Report PROGRESA-Caribbean Page 53 of 112 15.6 percent for producers from early treatment, and from 26.2 percent to 21.4 percent for producers from the late treatment. As expected, the age of the head of household increased for both groups. However, an important number of households had young heads of households; 1 in 4 households for the early treatment group and 1 in 5 for the late treatment group. At the same time, the average number of years of schooling also increased for both early and late treatment, 1.1 and 0.8 years, respectively. 5.1.10.2. Socioeconomic wellbeing The measurement of poverty was based upon the 2015 version of the Progress Out of Poverty Index (PPI)36 . Poverty incidence changed from 60.1 percent at the baseline to 52.9 percent at the end￾line for the case of the early treatment, and from 66.6 percent at the baseline to 55.1 percent at the end￾line for the case of the late treatment. In other words, poverty incidence was reduced by at least 7 points 36 The PPI methodology for Nicaragua was updated by Mark Schreiner in 2013 using the 2009 EMNV (standard-of-living survey) as a reference for estimating the probability of being poor. Table 13. Household size Baseline Endline Baseline Endline Number of members in the household 4.8 4.9 5.2 5.2 Large Household 19.8% 26.2% 15.6% 21.4% Note: A large household is one with 7 or more members. Early Treatment Late Treatment Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Table 14. Demographic characteristics of the families Baseline Endline Baseline Endline Household head age 45.3 49.2 47.4 51.5 Young household head 36.4% 24.9% 32.7% 20.7% Average household years of schooling 4.2 5.3 4.0 4.8 Average head years of schooling 3.9 3.9 3.3 3.2 Average years of agricultural experience 14.8 14.3 13.9 13.8 Early Treatment Late Treatment Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Note: A household with a head younger than 40 years old is considered a young household head. Table 13. Household size Table 14. Demographic characteristics of the families Final Evaluation Report PROGRESA-Caribbean Page 54 of 112 in 4 years, which is more than the average poverty reduction rate according to the last available official information37 . During the life of the program, families increased their monetary resources, compared to baseline, which could explain the improvement in indicators like nutrition, education, and housing. Although migration may have had some impact on household size, there could be other reasons for reduction in average household size (natural deaths, marriages, etc.), and therefore, would require further follow up to determine direct causality between participation in the program and the incidence of poverty. It is important to mention that while the poverty incidence of the country increased during 2018 and 2019 as a result of the economic crisis (20% in 2017 to 30% in 2019), the poverty incidence of the families in the program reduced. The panel data allowed the estimation of transition matrices of poverty between the baseline and end-line38, using poverty quintiles. In other words, the data allowed the classification of individuals according to their poverty probability and observe if the probability changed over time. The results suggested a greater mobility towards more favorable poverty ranges. At baseline, 50 percent of households had a poverty probability of 80-100 and 45 percent that had a poverty probability of 61-80 transitioned to lower poverty probability quintiles. 37 The National Institute for Development Information (INIDE, by its acronym in Spanish) estimates that poverty incidence in Nicaragua changed from 29.6 percent in 2014 to 24.9 percent of the population in 2016; this represent a reduction of 4.7 base points in the mentioned period. 38 The transition matrix is a Markov process that estimates the conditional probability of finding a household in a specific range of poverty at end-line, conditioned on the fact that this household was in another range at the baseline. Table 15. Socioeconomic outcomes Baseline Endline Baseline Endline PPI general (%) 60.11 52.9 66.6 55.13 HDDS (number of food groups) 7.1 7.7 6.8 7.5 MAHFP (months) 11.2 11.7 11.1 11.6 % of families with insufficient food 34% 14% 33% 17% Early Treatment Late Treatment Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Table 15. Socioeconomic outcomes Final Evaluation Report PROGRESA-Caribbean Page 55 of 112 The poverty reduction was driven primarily by 3 factors. First, there was an increase of 13 points in the number of households that had sanitary facilities, reducing the number of households that did not have any type of sanitary services. Second, the share of households sending all school age children to school went from 33 percent to 51 percent. Lastly, at end-line only 18 percent of the households had 7 or more members, a reduction of 9 points compared to baseline. To complement the analysis, food security indicators such as the number of Months of Adequate Household Food Provisioning (MAHFP) and Household Dietary Diversity Score (HDDS)39 were also collected. There were no statistically significant changes between the baseline and the end-line in these indicators for both, early and late treatment. Lastly, the number of families that underwent food shortages at some month during the year changed from 34 percent at the baseline to 14 percent at the end-line in the case of the early treatment group, and from 33 percent at the baseline to 17 percent at the end-line in the case of the late treatment group. This means that for both groups, the number of families that underwent food shortages at some month during the year was reduced by half. 5.1.10.3. Sociopolitical context of Nicaragua In April 2018, Nicaragua faced an abrupt socio-political crisis that has since affected the economy. The country has suffered an economic contraction of 3.8 percent and the agriculture sector, in particular, has suffered a deceleration, including the cattle sector which experienced a contraction of more than 5 percent. The effects of the 2018 socio-political crisis has created the second worst economic contraction in the history of the country after the 2008/2009 contraction driven by the international financial crisis. Nicaragua’s current economic crisis has affected a significant portion of the population. By the end of 39 The HDDS accounts for the number of nutritional food groups out of the 12 recommended the household consumed in a typical day. Table 16. PPI (%) transition matrix 0-20 21-40 41-60 61-80 81-100 0-20 64.3 10.7 14.3 8.9 1.8 100 21-40 44.4 17.8 24.4 11.1 2.2 100 41-60 21.4 15.2 35.7 24.1 3.6 100 61-80 10.6 6.5 27.7 48.2 7.1 100 81-100 3.3 0.7 7.1 39.6 49.4 100 19.2 8.0 21.8 33.5 17.5 100 Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. End-line Baseline Total Total Table 16. PPI (%) transition matrix Final Evaluation Report PROGRESA-Caribbean Page 56 of 112 2018, FUNIDES estimated that more than fifty thousand people were left unemployed, and 23 percent of the population now lives in poverty. In addition, more than seventy thousand people fled the country. The consortium partners decided to explore how the sociopolitical context affected their producers with the application of a short survey collected at end-line. The purpose of the survey was to try to measure if the families were subject to economic shocks and possible responses to those shocks. It is important to emphasize that there was no comparison point or a contrafactual, so no inference can be made about the real impact of the socio-political crisis. First, to measure asset acquisition, the survey asked about land sold and purchased. For both cacao and cattle producers, about 18 percent acquired new land over the evaluation period. As expected, most of the cacao producers who bought land acquired agricultural land, versus the cattle producers who acquired cattle and pastureland. The survey also asked about different family events. In 27.5 percent of the producer’s households, someone worked outside the farm during the evaluation period. A slightly smaller share of cattle producer’s households had a member working outside the farm. When asked if someone in the household had to quit school during the evaluation period, for both cattle and cacao producers, 6.8 percent of the households reported at least one person quit school. Exploring the reason why members had to quit school, around 20 percent attributed the socio-political crisis as the main reason. Other reasons included sickness and financial issues. However, around half of the cases were attributable to other reasons including lack of interest and the need to work. One of the main effects of the 2018 socio-political crisis was migration. At least one person migrated in 21.1 percent of the cacao producers’ households. For cattle producers’ households, 19 Graph 12. Land Transactions Final Evaluation Report PROGRESA-Caribbean Page 57 of 112 percent of the households had at least one person migrate. Migration was primarily international, largely to Costa Rica. It’s worth noting that the survey asked about someone migrating, regardless if the person had returned at the time of this survey. 5.2. Producer organizations performance The performance evaluation of the producer organizations was based on the application of the ADA Learning Cycle self-assessment and the LINK tool. The importance of the application of these tools relied on the fact that the different stakeholders (partners, leadership, contracted personnel or collaborators) that comprise these rural cooperative enterprises understand the status of their organization, their business relationships and the progress that it experiences over time. 5.2.1. Management of producer organizations (ADA Tool) ADA enabled the producer organizations to carry out a quick analysis of their business management and organizational processes. The methodology is based on a participatory self-assessment process by the leadership bodies, partners, and the management, administrative and technical teams of the cooperatives. Graph 13. Socioeconomic effects Final Evaluation Report PROGRESA-Caribbean Page 58 of 112 At the end-line, the ADA assessment was applied to 15 cooperatives. The Evaluation Team did not apply this tool to 8 organizations because they left the program or did not have baseline data40 . The cooperatives assisted by the program obtained an average score of 71.6 points out of 100 in the ADA tool at the end-line. This score represents an increase of 70 percent compared to baseline results. The increase was a result of the improvement in all the areas that the program addressed with the 15 cooperatives. 40 According to CRS MEAL team, attrition amongst cooperatives was associated with failing to maintain or secure their registration with the government; opportunity to join government or other NGO funded programs, and economic situation in the country. More information about attrition is available in Annex V. Table 17. Producers organizations selected for ADA tool Producer organization Municipality COOSEMUVIS Bocana de Paiwas COOSEMUP Bocana de Paiwas COOMULBAN Bonanza COOPROCAFUC El Castillo COOSEMUCRIM El Castillo ASIHERCA El Castillo COODEPROSA El Castillo COOPROCAR El Rama COMPROMUB Muelle de los Bueyes La Pradera Muelle de los Bueyes COOPELACME Nueva Guinea COOSAGRO Nueva Guinea UCA Ahmed Campos Nueva Guinea COOMUNSOL Siuna Nueva Waslala Waslala Source: PROGRESA Caribe. Table 17. Producers organizations selected for ADA tool Final Evaluation Report PROGRESA-Caribbean Page 59 of 112 The tool was applied to five areas of assessment: 1) strategic orientation; 2) business management; 3) technical services; 4) structure and functionality; and 5) governance in partnership processes. The areas with greater scores compared to baseline were technical services (91%), business management (83%) and strategic orientation (77%). The remaining two areas, governance (50%) and structure and functionality (37%), also showed an improvement at end-line. Despite that structure and functionality's score had the lowest variation rate compared to baseline, it was the area with the highest score with 78 points. This outcome is explained by the strength in the structure of the organizations and their fulfillment of the legal requirements needed to operate. Furthermore, the cooperatives had a score of 76.3 points in the business management component at end￾line and the results in the evaluation of the management, accountability system, and human resource management were all better than those obtained at baseline. The score obtained by the cooperatives in the strategic orientation component was 71.7 points. They showed improvements in the vision and mission of the organization, the strategic links with their partners and the market, and the strategies to achieve medium and long terms goals. The component of governance showed a similar score with 71.3 points as a result of enhancement of partners’ participation, inclusion, and organizational practices. Table 18. Score of ADA tool by producer organization Producer organizations Baseline Endline Variation rate ASHIERCA 32.4 68.9 112.9% COMPROMUB 34.3 19.7 -42.5% COODEPROSA 41.5 96.3 131.8% COOMULBAN 14.2 71.4 403.8% COOPELACME 15.5 64.5 317.4% COOPROCAFUC 59.1 75.9 28.5% COOPROCA 38.0 69.5 82.9% COOSAGRO 41.6 88.6 113.1% COOSEMUP 44.3 69.5 57.1% COOSEMUVIS 32.1 52.0 61.9% COSEMUCRIM 65.6 82.7 26.2% LA PRADERA 66.5 84.7 27.5% NUEVA WASLALA 54.4 77.5 42.5% UCA AHMED CAMPOS 71.3 79.7 11.8% COOMUNSOL 19.3 72.4 274.7% Project 42.0 71.6 70.4% Source: Evaluation team's calculations based on on ADA tool (panel data) – PROGRESA Caribe. Table 18. Score of ADA tool by producer organization Final Evaluation Report PROGRESA-Caribbean Page 60 of 112 The provision of Technical Services was the component with the lowest score (59.1 points). Nonetheless, the cooperatives showed an improvement in this area compared to baseline. The cooperatives that increased their score, have focused their efforts on the provision of services demanded by their members’ productive activities, but the resources were insufficient to satisfy the entire demand. Moreover, some cooperatives formed by cattle producers did not provide technical services because the market already satisfied their needs. 5.2.2. Business relationships with buyers (LINK Tool) The LINK tool was used to analyze the business models of cooperatives and the relationship with their main buyers in order to understand the inclusion of the business relationships between cooperatives and buyers. At the end-line, the tool was applied to 11 cooperatives and their main commercial partner. Table 19. Score of ADA tool by area Component Baseline Endline Variation rate Strategic Orientation 40.6 71.7 76.6% Business Manageement 41.7 76.3 83.0% Technical Services 31.0 59.1 90.7% Structure and Functionality 57.0 78.0 36.7% Governance 47.7 71.4 49.8% Project 42.0 71.6 70.4% Source: Evaluation team's calculations based on on ADA tool (panel data) – PROGRESA Caribe. Table 20. Producer organizations selected for LINK tool Producer organizations Municipality Buyer COOSEMUVIS Bocana de Paiwas NILAC COOSEMUP Bocana de Paiwas NILAC COOPROCAFUC El Castillo Ritter Sport COOSEMUCRIM El Castillo Ritter Sport ASIHERCA El Castillo Ritter Sport COODEPROSA El Castillo Ritter Sport COOPROCAR El Rama Ritter Sport La Pradera Muelle de los Bueyes NILAC COOPELACME Nueva Guinea LALA UCA Ahmed Campos Nueva Guinea Ritter Sport Nueva Waslala Waslala Ritter Sport Source: Evaluation Team. Table 19. Score of ADA tool by area Table 20. Producer organizations selected for LINK tool Final Evaluation Report PROGRESA-Caribbean Page 61 of 112 On average, the relationship between cooperatives and buyers scored 51.5 point out of 100 in the LINK tool at end-line. This score represents an increase of 3 percent compared to baseline results. Most of the cooperatives surveyed indicated a decline in the commercial relationship with its main buyer due to the socio-political and economic situation in 2018. This tool assessed six principles: 1) collaboration among stakeholders; 2) effective market linkage; 3) transparent and consistent governance; 4) access to services; 5) inclusive innovation; and 6) measurement of results. Most components presented an increase in scores compared to baseline. Collaboration among stakeholders was the component with the highest score (74.3 points) and it is the only one with a score above 70 points. This component presented an increase of 12 percent compared to baseline. This result is explained by the strengthened links between the cooperatives and their buyers due to inclusive participation in the joint decision making. The cooperatives also showed an improvement in the indicators used to measure the results with its business partners. This component had a score of 67.3 points, 13 percent greater than the score obtained during baseline. The cooperatives and their buyers constructed clearer indicators to follow-up the commercial relationships based on short, mid and long terms goals. The component of effective market linkage scored 60.4 points. This score did not show a statistical change, as the relationship of the cooperatives and their buyers did not increase the access to new markets to the former. On the other hand, the transparency and consistent governance component had a score of 59.9. This component had an increase of 7 percent due to the improvement in the participation during the decision making among stakeholders. Table 21. Score of LINK tool by producer organization Producer organizations Baseline End-line Variation rate ASHIERCA 34.6 26.2 -24.2% COODEPROSA 50.6 77.5 53.3% COOPROCAFUC 62.7 55.4 -11.7% COOSEMUP 44.4 41.2 -7.1% COOSEMUVIS 45.3 41.6 -8.0% COSEMUCRIM 52.8 40.3 -23.6% NUEVA WASLALA 65.9 62.6 -5.0% UCA AHMED CAMPOS 43.1 66.9 55.2% Program average 49.9 51.5 3.1% Source: Evaluation team's calculations based on LINK tool (panel data) – PROGRESA Caribe. Table 21. Score of LINK tool by producer organization Final Evaluation Report PROGRESA-Caribbean Page 62 of 112 Access to services was the only component which presented a decrease compared to baseline. Most of the services resulting from the stakeholder relationships were closed due to the economic crisis in the country. Finally, the inclusive innovation component scored 19.2 points, 19 percent greater than baseline. Despite the increase, this component was not the main goal in the relationship between cooperatives and buyers. Their relationship was shaped as a productive value chain where the commercial benefits were the most important aspect. 6. Conclusions and lessons learned The Program for Rural Enterprise, Health and Environment – Caribbean Zone (PROGRESA Caribbean) was a five-year (2014–2019) value chain strengthening program for the cacao and livestock (dual purpose) sectors in the Caribbean Coast Region of Nicaragua: North Caribbean Coast Autonomous Region (RACCN, by its acronym in Spanish), South Caribbean Coast Autonomous Region (RACCS, by its acronym in Spanish), Río San Juan, Jinotega and Chontales. The Food for Progress program, financed by the United States Department of Agriculture, had a total budget of US$10,254,800 and was implemented by Catholic Relief Services (CRS) in consortium with TechnoServe (TNS) and Lutheran World Relief (LWR). The program assisted 5,108 small cacao and/or livestock farmers from 14 municipalities of the Caribbean Coast, Río San Juan and Chontales. The program also worked on strengthening 15 producer organizations and associations and promoted 11 commercial relationships under inclusive business models. The project activities aimed at fulfilling two strategic objectives: 1. Increase agricultural and livestock productivity. 2. Expand trade of cacao and cattle products. Table 22. Score of components of LINK tool Component Baseline End-line Variation rate Collaboration among stakeholders 66.3 74.3 12.0% Market Linkage 59.4 60.4 1.7% Transparent and consistent governance 55.9 59.9 7.1% Access to services 42.3 27.8 -34.2% Inclusive Innovation 16.1 19.2 19.2% Results measurement 59.4 67.3 13.3% Source: Evaluation team's calculations based on LINK tool (panel data) – PROGRESA Caribe. Table 22. Score of components of LINK tool Final Evaluation Report PROGRESA-Caribbean Page 63 of 112 The long-term outcome of the program was to increase household income and reduce poverty in the intervention areas. 6.1. The relevance of the program The program targeted the RACCN, RACCS and Río San Juan, which are the areas of the country with the highest incidence of poverty and historically, they have received fewer resources from both government institutions and international cooperation for development projects. The beneficiaries were small producers who work on dual-purpose cattle, cacao or both value chains, and whose production is fundamental to their livelihood strategy. These value chains have a high regional impact in the intervention zones, not only because of their high presence in the area but also because of their potential growth in demand for workers. According to data of the National Agricultural Census (2010), the Caribbean Coast accounts for 39 percent of the country's cattle and 53 percent of the area planted with cacao. Additionally, the program targeted socioeconomically disadvantaged producers, allowing them to gain access to knowledge and resources that otherwise would have been difficult to access. The program made every effort to prepare a learning curriculum in accordance to the relatively low educational level of the producers, reducing barriers with more participative learning environments, applying learning by doing methodologies, and hiring technical personnel locally. The results of the beneficiary satisfaction survey of both the early and late treatment groups showed high satisfaction levels among the participants. However, the satisfaction module uncovered some degree of heterogeneity across territories, mostly regarding terminology and the methodology of the knowledge transmission. CRS explained that the intervention approach was diverse due to different partner approaches and the territorial extension of the program (more than 200 communities in 14 municipalities). For future interventions and from an evaluation point of view, it is recommended to harmonize terminology and methodologies amongst partners at the beginning of the program. The program clearly fits within the Government of Nicaragua’s economic and agricultural development policies and programs. The Caribbean Coast is a region of high interest for the Government and PROGRESA Caribbean’s components are in accordance with other programs executed by the Government such as NICADAPTA and ProCacao, financed by other international cooperation agencies. 6.2. Efficiency of the program The Evaluation Team considered that it is not possible to explore whether the same results could have been obtained with fewer resources or alternative approaches, for three main reasons: 1. The program adjusted 4 of 11 original activities to better fit the context of the different areas of intervention, introducing some degree of variation during implementation. 2. The activities delivered by the program were not the same for all the producers treated. Final Evaluation Report PROGRESA-Caribbean Page 64 of 112 3. During the intervention, producers faced external shocks like Hurricane Otto in November 2016. In January 2017, the program carried out a survey of 27% of the producers located in Río San Juan, the area most affected by Hurricane Otto. The results of this survey showed that 98.5% of the sample of cacao producers were affected by the hurricane in this area. On average, each producer had damages to 17% of their production area. Those areas were mainly affected by the fall of shade trees and the overturning of the cacao trees, both because of wind and water saturation. Efficient strategies were implemented in terms of financial and human resources. In financial terms, the cost-share system that was designed for the provision of goods for production enabled the program to reach more producers than originally planned. This system provided a subsidy of between 40 percent and 60 percent, depending on the type of asset, the value chain, and the socioeconomic status of the producer. The producer paid the difference and this strategy helped encourage the producers to value the investment and put more effort into the production activity. In terms of human resources, the program aimed to provide more coverage to the beneficiaries through workshops, farmer field schools and informational talks. However, it could not be determined whether the different methodologies had a differentiated impact (whether it diluted the effects or not), as there was no comparison group. Regarding the monitoring system, it provided relevant key information in a timely manner. The digital information system allowed data management to be efficient. In the future, the Evaluation Team recommends the integration of automated consistency checks in the forms, this will reduce potential measurement errors in the data. Additionally, the Evaluation Team considers that there is more adequate software now available for data collection, specifically designed for survey data such as CSPro, Commcare and SurveyCTO. 6.3. Effectiveness and impact of the program As for the effectiveness and impact of the program, the Evaluation Team considers that the program’s overall performance was positive. The cacao producers increased their productivity and their total output significantly due to the intervention. These increases were not only due to higher yields but also due to more area under production. In terms of cattle, although no significant effect was found in the value chain productivity due to the intervention, all the fundamental intermediate outcomes were improved, which is where the program had a direct intervention. That said, the cattle sector experienced an even bigger contraction than the economy, which impacted the price of inputs and outputs, making cattle production less profitable and influencing producers’ motivation and incentives to invest in their cattle business, and ultimately participate in the market. Final Evaluation Report PROGRESA-Caribbean Page 65 of 112 For all three products, cacao, milk, and cattle commercialization rates increased, meaning that more producers were selling their production compared to the baseline rather than consuming it. This translates to an important productive shift. More importantly, all welfare characteristics of the farmers and their families improved. Poverty incidence changed from 60.1 percent at the baseline to 52.9 percent at the end-line in the case of the early treatment, and from 66.6 percent at the baseline to 55.1 percent at the end-line in the case of the late treatment. In other words, poverty incidence was reduced by at least 7 points in 4 years, which is more than the average poverty reduction rate according to the last available official information, as mentioned in previous sections. During the life of the program, families increased their monetary resources, in comparison to baseline, which could explain the improvement in indicators like nutrition, education, and housing. Although migration may have had some impact on household size, there may have been other reasons for reduction in average household size (natural deaths, marriages, etc.,), and therefore, would require further follow up to determine direct causality between participation in the program and the incidence of poverty. It is important to mention that while the poverty incidence of the country increased during 2018 and 2019 as a result of the economic crisis (20% in 2017 to 30% in 2019), the poverty incidence of the families in the program reduced. Additionally, the number of families that underwent food shortages at some month during the year changed from 34 percent at the baseline to 14 percent at the end-line in the case of the early treatment, and from 33 percent at the baseline to 17 percent at the end-line in the case of the late treatment. This means that for both groups, the number of families that underwent food shortages at some month during the year was reduced by half. As mentioned in the midterm report, welfare changes mostly take place over longer periods. PROGRESA Caribbean program had positive effects on the family’s welfare considering the duration of the program and the national context at the end of the program. 6.4. Good agricultural and commercialization practices The application of agricultural practices reflects the knowledge, skills, and abilities that the farm families acquired either on their own or from the implementation of the program. The proportion of producers applying different types of practices evaluated in cacao and/or cattle increased between baseline and end-line. In cattle, there were notable increases in the following practices that were promoted by the program: rotation of pastures, mastitis testing, and cattle identification. In cacao, there were significant changes in the proportion of producers practicing: rehabilitative pruning and disease management. Following the recommendation made by USDA at the mid-term evaluation, an additional survey was carried out by the Evaluation Team at end-line to take a more in-depth look at the practices adopted by the farmers, focusing not only on the adoption but also on the intensity and quality of the practice. Final Evaluation Report PROGRESA-Caribbean Page 66 of 112 The analysis revealed an association between improved nutrition and production and increase in yields in beef, but not milk. The study was less conclusive in identifying the same association in cacao due to the number of practices and nature of the crop, taking longer time to develop. This additional survey allowed a better understanding of the different levels of the application of the agricultural practices. For future interventions, the Evaluation Team recommends including this type of survey during the baseline to contribute to better understanding and comparing technical improvement over time. 6.5. Sustainability The program design focused on knowledge transmission, promoting investments at the farm and cooperative levels, strengthening of producer organizations, and the creation of alliances. Producers confirmed that they gained useful knowledge during the program, and that they want to continue adopting and improving the lessons learned. Additionally, the program included goods transfers where the producers contributed to a portion of the cost, thus creating a sense of ownership of the investments. Even though some producers expressed concerns related to resource constraints and lack of technical assistance going forward, the Evaluation Team concluded that the learning, adoption of new practices, private-private alliances created, cooperatives strengthened, and trained community promoters will help ensure the sustainability of the program. 6.6. Key recommendations for future interventions Based on the results of the evaluation, the Evaluation Team outlines the following set of recommendations for future interventions: 1. Ensure that the objectives of the intervention and the plausible time frame to achieve the goals are realistic. For example, cacao is a crop that takes longer to come into production. 2. To the extent possible, develop more harmonized methodologies when working with more than one partner, from the terminology to the actual materials and “know how” to be transferred. This will reduce an unnecessary source of variation. 3. Increase local support systems in the communities for when the program ends. Local promoters should play a bigger role in the program and increase their recognition in the communities. Additionally, even if training sessions are held collectively, future programs could increase individual follow up visits with increased staffing/promoters. 4. It is important that quality adoption of the main agricultural practices is measured from the baseline, especially interventions focused on adoption of new knowledge and behaviors. Additionally, the following recommendations will contribute to improving the monitoring and evaluation of future interventions. Final Evaluation Report PROGRESA-Caribbean Page 67 of 112 1. Data collection instruments must be designed, side by side with the intervention design. Ongoing changes must be carefully considered to allow comparability between waves of information. 2. Adopt a better data collection software, specifically designed for survey data that allows for ongoing quality checks. Additionally, develop a set of rules for consistency to improve the data quality testing. Final Evaluation Report PROGRESA-Caribbean Page 68 of 112 References Archibald, T., Sharrock, G., Buckley, J., & Cook, N. (2016). Assumptions, Conjectures, and Other Miracles: The Application of Evaluative Thinking to Theory of Change Models in Community Development. Evaluation and Program Planning, 59, pp. 119–27. Archibald, Thomas; Sharrock, Guy; Buckley, Jane and Cook, Natalie. Assumptions, Conjectures, and Other Miracles: The Application of Evaluative Thinking to Theory of Change Models in Community Development. Evaluation and Program Planning 59, pp. 119–27. Bamberger, M. (2012). Introduction to Mixed Methods in Impact Evaluation (Impact Evaluation Notes No. 3). InterAction and The Rockefeller Foundation. Bonbright, D. (2012). Use of Impact Evaluation Results (Impact Evaluation Notes No. 4). InterAction and The Rockefeller Foundation. Caliendo M. & Kopeinig S. (2008). Some Practical Guidance for the Implementation of Propensity Score Matching. Journal of Economic Surveys, 22(1), pp. 31-72. Caliendo M. and Kopeinig S. (2008). Some Practical Guidance for the Implementation of Propensity Score Matching, Journal of Economic Surveys, No. 22, pp. 31-72. Carter, M. R., Toledo, P., & Tjernström, E. (2012). The Impact of Rural Business Services on the Economic Well-being of Small Farmers in Nicaragua. Mimeo. Catley, A., Burns, J., Abebe, D., & Suji, O. (2007). Participatory Impact Evaluation or Assessment. A Design Guide. Somerville: Feinstein International Center. Duflo, E., Glennerster, R., & Kremer, M. (2008). Using Randomization in Development Economics Research: A Toolkit. In Schultz, T., & Strauss, J. (Eds.), Handbook of Development Economics (pp. 3895-3957). Amsterdam: Elsevier. Heckman, J. J., Ichimura, H., & Todd, P. (1998). Matching as an econometric evaluation estimator. The Review of Economic Studies, vol 65 (2), pp. 261-294. Gertler, P., Martinez, S., Premand, P., Rawlings, L., & Vermeersch, C. (2011). Impact Evaluation in Practice. Washington DC: The World Bank. Gottret, Junkin and Ugarte (2011). Self-assessment provided for the Management of Rural Enterprises. CATIE. Gottret, M., Junkin, R., & Ugarte, C. (2011). Autoevaluación facilitada para la gestión de empresas asociativas rurales. Turrialba: Centro Agronómico Tropical de Investigación y Enseñanza (CATIE). Guijt, I. (2014). Participatory Approaches (Methodological Briefs, Impact Evaluation No. 5). Florence: UNICEF, Office of Research. Instituto Nacional de Información de Desarrollo (2015). Resultados de la “Encuesta Nacional de Hogares sobre Medición de Nivel de Vida - 2014”. Instituto Nacional de Información de Desarrollo (INIDE) (2015). Resultados de la “Encuesta Nacional de Hogares sobre Medición de Nivel de Vida - 2014”. Managua: INIDE. Lundy, M., Becx, G., Zamierowsli, N., Amrein, A., Hurtado, J., Mosquera, E., & Rodríguez, F. (2012). LINK Methodology: A participatory guide to business models that links smallholders to markets (CIAT Publication No. 380). Cali: Centro Internacional de Agricultura Tropical (CIAT). Maluccio, J., & Flores, R. (2004). Impact Evaluation of a Conditional Cash Transfer Program: The Nicaraguan Red for Social Protection (Discussion Paper 184). Washington DC: International Food Policy Research Institute. Ravallion, M., & Jalan, J. (1999). Does piped water reduce diarrhea for children in rural India? The World Bank. Final Evaluation Report PROGRESA-Caribbean Page 69 of 112 Schreiner, M. (2013). A Simple Poverty Scorecard for Nicaragua. Microfinance Risk Management, L.L.C. Schreiner, M. (2013). Simple Poverty Scorecard. Poverty-Assessment Tool Nicaragua. Microfinance Risk Management, L.L.C. Winters, P., Salazar, L. and Maffioli, A. (2010), Designing Impact Evaluations for Agricultural Projects, SPD Working Papers 1007, Inter-American Development Bank, Office of Strategic Planning and Development Effectiveness (SPD). Winters, P., Salazar, L. and Maffioli, A. (2010). Designing Impact Evaluations for Agricultural Projects (SPD Working Paper 1007). Inter-American Development Bank, Office of Strategic Planning and Development Effectiveness (SPD). Final Evaluation Report PROGRESA-Caribbean Page 70 of 112 Annexes Annex I. Changes in the activities of PROGRESA-Caribe Original text from the cooperative agreement - Attachment A How the activity is developed or would be developed during the life of the program Adjustments Approved by USDA Capacity Building: Trade Associations CRS will assist with the formation of and strengthen existing non-profit and for-profit national trade associations for both cacao and livestock products. CRS will integrate trade associations into existing value chains, facilitate value chain discussions and collaborations, provide training in value chain analysis, manage each link in the chain, and assist with forging relationships with business actors. CRS will assist trade association stakeholders to develop competitiveness plans and advocate for support of those plans from secondary actors such as the government, universities, investigative centers, and service providers. CRS will help trade associations to increase the quantity and timeliness of market information for their members, including pricing trends, sales volumes, buyer tenders, and quality requirements. CRS will assist the strengthening of existing non-profit national trade associations that promote cacao and dairy products. CRS will integrate non-profit trade associations into the existing value chains. For example, it is working with FAGANIC and CONAGAN to strengthen their linkages and services to local cattle associations. It is also working with APEN and CANICACAO with cacao farmers and their cooperatives. CRS will also facilitate value chain discussions and collaboration between the actors linking the different levels in the chains through the participation in fairs, congresses, and commercial missions. Additionally, CRS will assist trade association and cooperative stakeholders to develop competitiveness plans and advocate for support of those plans from secondary actors such as the government, universities, investigative centers, and service providers. CRS will help trade associations and cooperatives to increase the quantity and timeliness of market information for their members including pricing trends, sales volumes and quality requirements. CRS proposes not assisting with the formation of new non-profit and for-profit national trade associations since the program has found at mid-term that it is more efficient to strengthen those that already exist such as CANICACAO, APEN, CONAGAN, and FAGANIC. In addition, at mid-term, CRS proposes to not directly “manage each link of the value chain”, but to instead “facilitate linkages between the different levels of the chain”. Financial Services: Facilitate Agricultural Lending CRS will establish a loan facilitation fund that helps buy-down financial institution risk, yielding more CRS will promote the co￾investment in private-private partnerships. CRS will also train As discussed previously with USDA representatives, CRS proposes eliminating the activity Final Evaluation Report PROGRESA-Caribbean Page 71 of 112 Original text from the cooperative agreement - Attachment A How the activity is developed or would be developed during the life of the program Adjustments Approved by USDA affordable interest rates. This fund may be used for partial loan guarantees, co-financing of collateral, buy-down of interest charges, and the co-investment in private-private partnerships. CRS will also train producer cooperatives in the basic business skills necessary to obtain formal credit, including improved record keeping, basic production and business plans, and the preparation of bankable loan dockets. Additionally, CRS will coordinate with microfinance agencies to facilitate relationship￾building with producer cooperatives and help them to develop and offer appropriate loan products that will meet the needs of the producer organizations. producer cooperatives in the basic business skills necessary to obtain formal credit, including improved record keeping, basic production and business plans, and the preparation of bankable loan dockets. Additionally, CRS will coordinate with microfinance agencies to facilitate relationship￾building with producer cooperatives and help them to develop and offer appropriate loan products that will meet the needs of the producer organizations. related to the loan facilitation fund due to budget cuts at the beginning of the project. The original intent of the fund was to lower risk for financial institutions to invest in small-scale farmers. Even without the fund, we have found that the presence of the program team in the field and the program´s support for the activities at the farm level has helped to lower the perception of risk by financial institutions regarding program participants. Infrastructure: Post￾Harvest Processing CRS will facilitate the construction and improvement of post-harvest cacao processing infrastructure including fermentation trays, drying tunnels, and ovens through direct investments, co-investments with private companies, assisting with obtaining credit for equipment purchases, and providing technical training. CRS will also develop dairy cold chains, chilling stations, producer-level milking rooms and handling equipment, smaller milk collection centers closer to target producers, and improved cheese processing equipment by co￾investing with producer cooperatives and private investors. CRS will facilitate the construction and improvement of post-harvest cacao processing infrastructure including fermentation trays, drying tunnels, and ovens through direct investments, co-investments with private companies, assisting with obtaining credit for equipment purchases, and providing technical training. CRS will also develop chilling stations at the farm level, producer-level milking rooms and handling equipment, smaller milk collection centers closer to target producers, and improved cheese processing equipment by co￾investing with producer cooperatives. The effort will focus on investments in milking rooms at the farm level and developing collection centers through co-investment between private companies and cattle At mid-term, CRS proposes to not develop dairy cold chains. Private companies are establishing this kind of facility with their own funds in the area where they buy raw milk. New investments in the dairy cold chain depend on the demand of raw milk and, while prices are low, there is not much incentive for farmers to co-invest in this particular intervention. Final Evaluation Report PROGRESA-Caribbean Page 72 of 112 Original text from the cooperative agreement - Attachment A How the activity is developed or would be developed during the life of the program Adjustments Approved by USDA farmers’ associations or cooperatives. The program will invest in small cheese processing equipment in order to provide the cooperative with the opportunity of utilizing raw milk when the companies (LALA or NILAC) are not able to get to the collection center due to extreme weather or other events. Inputs: Develop Agrodealers and Other Input Suppliers CRS will assist producer organizations and associations with obtaining working capital to develop supply infrastructure, purchase inputs, and build business relationships with input suppliers to obtain inputs on credit. Additionally, CRS will identify individual producer input and service needs and negotiate for an entire group in order to increase bargaining power and reduce the cost of acquiring inputs. CRS will assist producer organizations and associations with obtaining working capital to develop supply infrastructure, purchase inputs, and build business relationships with input suppliers to obtain inputs on credit. Additionally, CRS will identify individual producer input and service needs and negotiate for an entire group in order to increase bargaining power and reduce the cost of acquiring inputs. CRS does not propose changes in this activity. Inputs: Increase Access to Improved Genetic Material CRS will establish cacao nursery and plant propagation sites within producer cooperatives and small private enterprises in order to increase the availability of improved genetic material in target municipalities. CRS will provide technical assistance, facilitate access to finance for start-up costs, and train nursery enterprises in basic business management skills. For the cattle value chain, CRS will improve bull breeding, expand stud farms, and promote the use of artificial insemination services by training independent specialists in improved techniques, small business CRS will establish cacao nursery and plant propagation sites within producer cooperatives and small private enterprises in order to increase the availability of improved genetic material in target municipalities. CRS will provide technical assistance, facilitate access to finance for start-up costs, and train nursery enterprises in basic business management skills. For the cattle value chain, CRS will promote the use of artificial insemination services by training independent specialists in improved techniques, small business As discussed previously with USDA, CRS proposes eliminating the bull-breeding activities and expanding stud farms. Given the realities on the ground and the time it takes to breed bulls and expand stud farms, CRS believes that it is better to focus on training farmers to use better criteria for the selection of bulls on their farms or the purchase of new animals that are going to be used as studs. Final Evaluation Report PROGRESA-Caribbean Page 73 of 112 Original text from the cooperative agreement - Attachment A How the activity is developed or would be developed during the life of the program Adjustments Approved by USDA management skills, and for-fee services to cattle producers. management skills, and for-fee services to cattle producers. Final Evaluation Report PROGRESA-Caribbean Page 74 of 112 Annex II. Summary of the results framework Results Chain – Strategic Objective 1 Final Evaluation Report PROGRESA-Caribbean Page 75 of 112 Results Chain – Strategic Objective 2 Final Evaluation Report PROGRESA-Caribbean Page 76 of 112 Annex III. Matrix of intermediate and Outcomes indicators Matrix of intermediate indicators Description Calculation Formula Measuring Frequency Means of Verification Liters of milk per cow per day This indicator measures the productive efficiency of cattle herd Baseline survey, mid-term survey, final survey Module: Livestock production Yields in the cacao production The total cacao production volume in (Metric Tons) by producers divided by the total number of production hectares Baseline survey, mid-term survey, final survey Module: Agriculture Average cattle weight per hectare This indicator will measure the total kilos of live animals that the producer has in the herd inventory, and are supported by USDA Baseline survey, mid-term survey, final survey Module: Livestock production Cocoa Production Costs Total production cost per hectare measured in dollars for the cacao category. All sales related with cacao (dry and baba) production are included, as well as genetic material Baseline survey, mid-term survey, final survey Module: Agriculture Production Cattle Costs Total cost of milk sales, derived from milk and cattle, by hectare of pasture and measured in dollars Baseline survey, mid-term survey, final survey Module: Livestock production Gross Margin per Hectare (cocoa) Increase in income attributable to investments made in improving production techniques and technology implementation Baseline survey, mid-term survey, final survey Module: Agriculture Gross Margin per hectare (livestock) Increase in income attributable to investments made in improving production techniques and technology implementation Baseline survey, mid-term survey, final survey Modules: 1) Livestock Production and 2) Cattle Sales Number of hectares managed under sustainable agricultural practices Number of hectares of cocoa and livestock managed under sustainable practices, such as using eco-friendly fertilizers and pesticides, erosion control, agroforestry, etc. Baseline survey, mid-term survey, final survey Modules: 1) Livestock Production and 2) Cattle Sales Indicator Intermediate Indicators *100 *100 Final Evaluation Report PROGRESA-Caribbean Page 77 of 112 Matrix of outcome indicators Description Calculation Formula Measuring Frequency Means of Verification Progress out of Poverty Index The PPI is a simple index rating of poverty that estimates the probability that a home has costs less than a certain poverty line. The index uses ten simple indicators for collection on-site. Baseline survey, mid-term survey, final survey Modules: Poverty Index PPI Family Income Family income received by the sale of production and other income from some family member(s) Baseline survey, mid-term survey, final survey Modules: 1) Agriculture 2) Sale of Livestock and 3) Food Security (socio￾demographic characteristics ** HDDS: Total number of food groups consumed by families Months of adequate supply of food in the household (MAHFP) Capture changes in the home ability to approach the vulnerability so as to guarantee the procurement above a minimum level throughout the year. Measuring the MAHFP has the advantage of obtaining the combined effects of a series of interventions and strategies, for a better agricultural production, storage and activities that increase the families’ purchasing power. Baseline survey, mid-term survey, final survey Modules: Food Security Return of the producer’s investment and processing of cooperatives activities It measures the reduction in production costs and the increase of income as a result of the quality achieved through the efficiency of processing activities. Evaluates the performance of the investment made by the producer or cooperative in operation (efficiency of spending to maximize sales) Annual Administrative Records Number of individuals receiving financial services as a result of USDA´s assistance Number of producers who have access to financial services, as a result of USDA´s support. Financial services include those that help identify and access funds through formal and alternative channels. Bi-annual Administrative Records Indicator Outcome indicators Household dietary diversity scores It collects information about the families´ food safety. Focuses on the composition of the regular family diet, if they suffer some kind of food shortage and in which months this scarcity predominates. Baseline survey, mid-term survey, final survey Modules: Food Security 100 *100 Final Evaluation Report PROGRESA-Caribbean Page 78 of 112 Annex IV. Detailed explanation of the evaluation design PROGRESA Caribbean final evaluation followed a mixed-methods approach; in other words, it incorporated both quantitative and qualitative analysis. Both methods are complementary to each other. Bamberger (2012) states that this approach allows for: 1.Triangulation of the results of the evaluation, which strengthens the validity and credibility of the results. 2. Using information from one method to develop the instrument for another. 3. Complementarity in the results. 4. Diversity in the value dimensions of the evaluation; and generate new knowledge about the evaluation results. 5. Generating new knowledge into evaluation findings. Furthermore, the evaluation followed an Evaluative Thinking approach as suggested in Archibald, Sharrock, Buckley and Cook (2016), which implied close interaction with the Monitoring, Evaluation, Accountability and Learning (MEAL) and the Knowledge Management Division of CRS. In this regard, the Evaluation Team conducted several sessions to exchange information about the program with CRS and its partners. This made it possible to more deeply understand the assumptions of the program set forth in the theory of change and in the mechanisms for the implementation of activities in order to respond to the evaluation questions, and to recommend future actions. This section is divided in two parts. The first part discusses the methodology to measure the impact of the program on the welfare of the families and the performance of the producers (quantitative approach). The second part addresses the participatory approach to evaluation (qualitative approach) that will assess the impact on organizations of both producers (cooperatives) and private companies as well as beneficiaries' satisfaction on the services provided by this program. The qualitative methods are complementary to the (quantitative) experimental evaluation of the impact of the program on the beneficiary’s outcomes. Quantitative design at farmers level41 Eligibility criteria, targeting and selection of beneficiaries The program targeted the RACCN, the RACCS, Río San Juan, Jinotega and Chontales, because these territories (which are mainly rural) present a high incidence of poverty and historically, these areas have received fewer resources for socio-productive projects. The target 41 This section is constructed based on section 4.1 of the Baseline Report of PROGRESA-Caribbean: Experimental Design. Final Evaluation Report PROGRESA-Caribbean Page 79 of 112 municipalities were selected after consultation processes with local organizations and implementing partners, local leaders and other key stakeholders. The program benefitted 5,108 smallholder producers from 14 municipalities: Waslala, Siuna, Bonanza and Rosita in RACCN; El Ayote, La Cruz de Río Grande, Muelle de los Bueyes, Bocana de Paiwas, El Rama, and Nueva Guinea in RACCS; San Carlos, El Castillo in Rio San Juan; and Santo Domingo in Chontales. The selection of beneficiaries was based on the following criteria: a) cacao growers, dual purpose cattle ranchers, and producers that worked on both value chains, whose production was fundamental to their livelihood strategy; b) producers that had between 5 and 100 head of cattle in production (and less than 20 hectares allocated to livestock production) and/or with less than 5 hectares for agriculture production; c) families that were willing to actively participate in the program; and d) identification of women farmers who worked in these chains in order to achieve 20 percent participation by women in the program. According to the program beneficiary selection criteria, beneficiaries could be supported in one or two value chains, if they fulfilled the selection criteria. The program also targeted farmers who had participated in prior interventions in the area with consortium partners and who met the selection criteria (repeated beneficiaries). Additional farmers who entered the program were identified by the consortium and implementing partners. In practice, the selection of the communities was based on the geographic location of the producers that met the eligibility criteria and who accepted being part of the program. The consortium partners originally targeted more than 4,000 farmers to participate in PROGRESA Caribbean. At baseline, a total of 4,041 producers were registered, and by the mid￾term evaluation the total number of beneficiaries was 4,134; representing 93 more farmers compared to the baseline. According to CRS monitoring data through May 201942, the numbers increased to a total of 5,164 active producers in the program, which is equivalent to 1,123 more farmers compared to the baseline. The distribution of beneficiaries by value chain and treatment is presented in Table A below: 42 The information in Table A was prior to the final evaluation. Final Evaluation Report PROGRESA-Caribbean Page 80 of 112 Table A. Distribution of the beneficiaries of PROGRESA-Caribbean 31.0 percent of the beneficiary producers were supported in the dual-purpose cattle value chain, 48.9 percent in the cacao value chain, and 20.1 percent in both value chains. Geographically, most of the selected cacao producers were in Waslala and El Castillo, while most of the livestock farmers were in the RACCS (mainly in the municipalities of Nueva Guinea and Bocana de Paiwas). Issues on the selection of beneficiaries The selection of the universe of beneficiaries of PROGRESA Caribbean generated two types of potential selection bias, due to: 1) administrative rule43 (based on observable characteristics) and 2) self-selection. The first was related to the fact that 14 percent of the targeted beneficiaries came from previous interventions of the consortium partners. These producers may differ in characteristics compared to those working for the first time with CRS and its partners44 , as they would have better outcomes and attitudes due to the capacities built in previous interventions like the current one. However, there was no reason to believe that both groups had differences in their motivations to participate in the program. In addition, another slight bias was identified due to observable characteristics, given that 16 percent of the non-random selected beneficiaries were women, who were encouraged to participate in the program. Another source of bias came from the selection – under the program criteria – of participants that were apparently willing to actively participate in the program (program placement bias). Finally, a potential self-selection bias might appear if some families with motivation to receive a benefit approached more actively local partners in order to get selected into the program. 43 Selection bias by administrative rule implies that beneficiaries are targeted based on observable characteristics, such as gender, location, socio-economic status, among others. 44 The evaluation team was not able to test for this during the baseline as there was no raw data on the characteristics of the universe of beneficiaries selected. Final Evaluation Report PROGRESA-Caribbean Page 81 of 112 The different sources of bias in the selection of the universe of beneficiaries or, in other words, the absence of a random selection of beneficiaries, makes it difficult to identify a causal relationship between program participation and the expected results. This is because the treatment status (being a beneficiary or not) is correlated to observable and non-observable characteristics of beneficiaries. Therefore, comparing beneficiaries to other populations does not identify the particular impact of the intervention as the expected result is not statistically independent of the treatment status. To be able to make a proper identification, the treated individuals must be compared with individuals that have the same characteristics on average (statistically equivalent); in other words, where the only difference that exists between the two groups is that only one group benefits from the program. Despite that the identification of beneficiaries did not follow a random selection as in traditional experimental designs, the proposed strategy randomly identified a temporary comparison group and also randomized the duration of time in the program. It was determined that both treatment and comparison groups were likely exposed to the same environmental conditions, given that the targeting in the Caribbean Coast and Río San Juan45are locations where farmers tend to face the same environmental conditions. Impact evaluation design The key aspect of all impact evaluations is to identify what would be the outcome of the beneficiaries in the absence of the program. Obviously, it is not possible to observe the counterfactual outcome because individuals can only have one treatment status (being treated or not). In the case of PROGRESA Caribbean, the selection of beneficiaries did not follow an experimental design, because it was very complicated to have a “pure” counterfactual outcome in agricultural and livestock programs, as only certain producers can be selected to participate for ethical reasons, program objectives and budgetary issues (Winters et al., 2010). As program activities were not planned to begin with all producers at the same time, the Evaluation Team proposed an intervention strategy that made it possible to randomly identify a temporary counterfactual group and to randomize the duration of the program for each of the beneficiaries. This strategy is called randomized order of phase-in (Duflo, Glennerster and Kremer, 2007) and has been applied to the case of Nicaragua in the Social Protection Network Program (see Maluccio and Flores, 2004) and in the Rural Business Development program of the Millennium Challenge Account (see Carter, Toledo and Tjernström, 2012). 45 At the time the Evaluation Team designed the baseline evaluation, Jinotega and Chontales were not part of the targeted municipalities. Final Evaluation Report PROGRESA-Caribbean Page 82 of 112 The consortium partners implemented a two-phase intervention. The first phase started in October 2015 and the second phase in May 2017, both ending in September 2019. The selection of beneficiaries that received the treatment in the first (early treatment) and second (late treatment) phases was randomized. Nevertheless, one of the main problems in agricultural programs are the spillover effects (or contamination), i.e., late treatment could be indirectly exposed to the program (Winter et al., 2010). For example, if a producer is selected to be in the early treatment group while another from the same community corresponds to late treatment, the latter can learn from the producer that is receiving technical assistance and/or any other type of services in the early treatment group. This would generate a contaminated comparison group and could dramatically underestimate the program effects. The experimental approach in several agricultural programs has been more recently designed to capture spillover effects for eligible and non-eligible units (double randomization). However, PROGRESA Caribbean was not designed for this purpose. Therefore, in order to diminish the contamination risk in the program, it was decided to randomize the treatment allocation to 65 groups (clusters) of adjacent communities (cluster randomization). These clusters (adjacent communities) were created in coordination with the consortium partners based on geographical location criteria. Within each group, the communities were adjacent and not among the targeted groups, since the geographical distances were very far. With this number of groups, the communities within each group have the same probability of being selected so the risk of contamination would be marginal. The consortium partners stressed during the evaluation design that in order to meet the goal of reaching the number of beneficiaries per year they had to initiate activities with more than 50% of the beneficiaries. So, it was decided a (cluster) random selection of 65 percent for early treatment and 35 percent for late treatment, which generated an early and late assignment of 42 and 23 clusters, respectively. At beneficiary level, this implied 2,676 beneficiaries for early treatment (63%) and 1,571 for late treatment (37%). Since all clusters (adjacent communities) had a similar opportunity to be early treatment, a treatment group (early treatment) and a temporary comparison group (late treatment) were formed with similar characteristics, as reported in the baseline. For the final evaluation, the first cohort became the treatment group and the second one corresponded to the comparison group, since the first group was more exposed to the intervention. A successful experiment requires a clear identification strategy. In this regard, the Evaluation Team explained the experimental design and its challenges to the consortium and implementing partners, so that they could inform all beneficiaries and producer organizations about the randomized treatment allocation process. Final Evaluation Report PROGRESA-Caribbean Page 83 of 112 The diagram below presents the treatment selection process46 . Figure A. Treatment selection process The cluster47 random assignment generated the following beneficiaries’ distribution by treatment status, value chain, region, sex and repeating48 farmers: 46 Information regarding the excluded producers was not available because they did not comply with the selection criteria or declined their participation. 47 As explained earlier, the clusters selected are groups of adjacent communities. 48 Farmers who had participated in prior interventions in the area with the consortium partners, and who met the selection criteria. Final Evaluation Report PROGRESA-Caribbean Page 84 of 112 Table B. Beneficiaries by treatment assignment Participatory approach to evaluation The consortium partners used different participatory methodologies with producers, cooperative members and staff, territorial leaders and private sector representatives during the life of the project Participatory evaluation sometimes refers exclusively to obtaining qualitative information on the views of participants using methods such as maps or stories; however, this is optional. This type of evaluation allowed the measure of the impact of different indicators. Catley, Burns, Abebe and Suji (2007) explain that the use of participatory methods to assign scores and establish a hierarchy order allows the linkage of numerical information to the qualitative indicators (based on opinions or perceptions). Assigning a score provides a point of reference through which it is possible to measure the program impact over time. The World Bank indicates that not all qualitative methods are participatory, although many participatory techniques can be quantified. The indicators obtained using participatory methods were not the primary purpose of this type of exercise, but the process of reflection and analysis arising from the application of the instrument. Guijt (2014) mentions that participatory approaches can be applied in any type of impact evaluation design, as they are not unique to a specific method of evaluation. Thus, the following discussion explains the participatory methodologies that measured the influence of PROGRESA Caribbean on the performance of producer organizations and business models with the private sector, and of the satisfaction of the services provided by the program. The methodologies relied on assigning scores based on opinions and/or perceptions from the beneficiaries and stakeholders (producer organizations and enterprises). The qualitative Beneficiaries by treatment assignment F M F M F M F M F M F M F M F M Both chains 25 194 15 87 28 139 12 Cacao 32 171 11 81 38 242 75 296 17 127 65 175 3 5 Livestock 9 99 5 27 108 571 3 16 Total 66 464 31 195 38 242 0 0 75 296 17 127 201 885 6 33 Both chains 21 127 3 43 84 476 4 Cacao 17 94 7 61 36 161 43 124 21 43 10 Livestock 4 35 3 13 140 1 Total 42 256 10 107 36 161 0 0 43 124 21 43 97 626 0 5 Note: F = female, M = male. Source: Based on PROGRESA Caribe monitoring records. Early treatment Late treatment Value chain RACCN RACCS Río San Juan RACS New Repeaters New Repeaters New Repeaters New Repeaters Final Evaluation Report PROGRESA-Caribbean Page 85 of 112 methods applied were complementary to the (quantitative) experimental evaluation of the impact of the program on the beneficiary’s outcomes. ADA Learning Cycle self-assessment tool The final performance evaluation of the producer organizations was based on the application of the ADA Learning Cycle self-assessment tool49 (ADA, by its acronym in Spanish). This tool enabled the producer organizations to conduct a quick analysis of their business management and organizational process. Gottret, Junkin and Ugarte (2011) state that this methodology aims at a participatory self-assessment process by the leadership bodies, members, and the management, administrative and technical teams of the cooperatives. The tool has six areas of assessment: 1) strategic orientation; 2) business management; 3) technical services; 4) financial services; 5) structure and functionality; and 6) governance in partnership processes. It included126 criteria outlined in 34 indicators, which are detailed in Figure 2. Each criterion was assigned a score, from one to five, where five was the optimal status for the organization. The self-assessment scores were entered into an Excel file to estimate the average values for each thematic area and for the overall score. This tool was applied by the consortium partners in the baseline and mid-term evaluations and by the Evaluation Team for the end-line. 49 Facilitated self-assessment of the management of rural associative enterprises (RAE), in Spanish “ADA”, which was developed by the Learning Alliance in Nicaragua. Final Evaluation Report PROGRESA-Caribbean Page 86 of 112 Figure B. Qualitative indicators for the facilitated self-assessment tool The baseline assessment applied the original version of the ADA tool developed by Gottret, Junkin and Ugarte (2011). After some discussion, CRS and its partners decided to exclude “Financial Services” from the assessments in the mid-term and final evaluation due to the lack of activities focused on this area. For the interpretation of the results and trend analysis, a standard normalization index was constructed for the sub-areas stated in figure 2 that comprise each area, obtaining percentage values from zero to 100. 𝑧 = 𝑥 𝑥 00 The score for these areas was constructed through a simple average of the sub-areas that comprise them. Likewise, the overall score, henceforth called the Learning Alliance score, was obtained through a simple average of the five standardized areas considered in this analysis. 𝑦 𝐴 = Qualitative indicators for the facilitated self-assessment tool Source: Baseline Report of PROGRESA Caribe (2014). 1) Strategic Orientation Market Skills Strategic Management Skills Strategic Planning Business Plan Business Alliances Access and Use of Information 2) Business Management Economic Analysis Economic and Financial Management Administrative Management Commercial Management Human Resource Management and Gender 3) Technical Services Access and Coverage Partnerships for Innovatin in Service Delivery Investment for the Provision of Services Management for the provision of Services Satisfaction with Technical Services Skills development for Sustainable Production Provisions of Operational Services 4) Financial Services Financial Skills for Saving Planning for the Provision of Financial Services Partnerships for Financial Access Management of Financial Services Access and Coverage Satisfaction with Service 5) Structure and Functionality Legal Status Organizatiiona l Chart and Functions Rules and Regulations Transparency and Accountability Communicatio n Influence on Policies and Practices 6) Governance in Partnership Processes Organizational Skills Membership and Commitment Organizational Practices Resolution of Conflicts Final Evaluation Report PROGRESA-Caribbean Page 87 of 112 𝐿 𝐴 = 𝐴 𝐴 𝐴 𝐴 Thus, the tool makes it possible to compare the results of the self-assessments with the results obtained by other rural associative enterprises and among themselves over time. For the evaluation of the tool, the methodology suggests that a score greater than or equal to 70 points places the cooperative in a positive position. The methodology indicates that initially one has to form groups or have separate sessions with the board, management, administrative and technical staff of the cooperative and with the associates. Once this stage is finished, a meeting is carried out where the different groups share their results and decide by consensus the final scores for each area of assessment. These scores are then incorporated into an Excel file, which calculates the average value obtained. The role of the consortium partners was to act as facilitators of the process but not as evaluators or verifiers. The results obtained through this methodology allowed producer organizations to classify themselves according to their level of development in terms of business and key organizational capacities, identify needs, current and future opportunities for improvement, and design an action plan to attain specific goals. The participation of management, administrative and technical teams, and a representative sample of associates is very important not only for the validity of the self-assessment, but also for evaluating the management and organizational process periodically at the different levels of decision making. This exercise also provides a reflection about the institutional strengthening needs and identifies opportunities and actions to improve the capacity of the organization. Additionally, this tool facilitates planning sessions to establish clear strategies to achieve the objectives that were developed through the self-assessment and to establish follow-up mechanisms to guarantee the progress towards the fulfillment of those goals. At the end-line, the ADA tool was applied to 15 cooperatives. Evaluation of the stakeholders in the value chain: Link tool The stakeholders in the value chain are linked by commercial relationships in which each act as a buyer and seller for its neighboring links. The LINK methodology developed by CIAT (2013) was used to evaluate the degree of connection (inclusion) in the value chain. This methodology was mainly aimed at those stakeholders that play the role of facilitators for the processes between sellers and buyers, which was the role played by CRS and its partners. The LINK methodology includes the use of four tools: 1. The value chain map, which helps identify the key stakeholders, processes and services within a value chain and how each interacts with the rest. Final Evaluation Report PROGRESA-Caribbean Page 88 of 112 2. The business model canvas, which enables the organizations and/or enterprises to rapidly sketch their current situation as an entity and to identify areas for improvement and/or intervention. 3. Business model principles for inclusive businesses, to determine whether each business that links rural producers with buyers is truly inclusive. 4. The prototype cycle, which unites the three previous tools to undertake concrete strategies to expand what is already effective or to apply new innovations to promote the participation of smallholder farmers in the value chain. These tools facilitate better understanding of the significant stakeholders as well as the processes and the relationships within a value chain. For the final evaluation, as in the baseline and the mid-term report, the Evaluation Team and the consortium partners agreed to apply Tool #3. This tool of the Link methodology is a participatory instrument designed to analyze the principles for inclusive business models. The methodology serves a double purpose: helping the buyers as well as the sellers to assess their commercial relations under inclusive criteria and serving as a point of departure in the promotion of the inclusion of small-scale producers in the value chain in which they participate. This tool assesses six principles: 1. Collaboration among stakeholders: establishing the level of common goals that exists between buyers and sellers. 2. Effective market linkage: evaluating the relations that form among the stakeholders to access market information and quality products. 3. Transparent and consistent governance: establishing the level of sharing of the rules of negotiation. 4. Access to services: measuring the impact of the buyer company in collaborating with the cooperative to access diverse services. 5. Inclusive innovation: evaluating the joint promotion of innovations in the products and in good practices. 6. Measurement of results: incorporating customized indicators and monitoring plans to assess the health status of the commercial relations of a for-profit company, as well as its effectiveness in promoting development. Each principle is disaggregated in sub-indicators and the score for each of the principles in this case, is achieved through self-assessment based on the perceptions of the cooperative’s representatives (or the company´s representatives). The score ranges from zero to five, with zero signifying that the parties are in “strong disagreement” with the sub-indicator and five signifying Final Evaluation Report PROGRESA-Caribbean Page 89 of 112 that there is “strong agreement.” The scale is designed to not have a neutral score; in other words, each response will have a positive or negative inclination. There are some criteria that may not be applicable (N/A), which depends on the context in which the business model is developed. In the three rounds, the tool was applied by the Evaluation Team. The objective was to obtain different perceptions and opinions and a better understanding of the context of the value chain at the end of the project. The LINK tool was applied to 11 cooperatives. Survey on the satisfaction and perception of beneficiaries The success of the program greatly depended upon the efficient and high-quality provision of technical, business and financial services to the producer families and the producer organizations. In this regard, the beneficiaries' perception on the quality of the services provided during the program was addressed through a survey on satisfaction and perception of the program. This instrument was developed by the Evaluation Team and constituted a module of the final evaluation. It covered the most and least-liked aspects of the program and rated each of the major activities developed. This tool was complementary to the (quantitative) experimental evaluation of the impact of the program on the beneficiary’s outcomes and helped the Evaluation Team to get inputs for the conclusions and lessons learned. It is important to mention that results based on perception should be interpreted with caution. Although it may provide relevant information on the quality of the services, it can also show the degree of producers' understanding on the type and quality of services provided by the program. Also, the Evaluation Team cannot rule out the possibility of measurement errors in data collection. Annex V. Detailed explanation of the empirical strategy at farmers level The information for the final evaluation at farmers level came from a panel data that compiles the characteristics of producers and their families for two evaluation rounds: baseline and final. As explained later in this document, the mid-term data was not considered for the final evaluation due to comparison difficulties. The collected data allowed one to evaluate the evolution of intermediate and outcome indicators and to estimate the final impact of the program at farmers level. The aim was to identify lessons and recommend strategic actions to improve future related interventions. Sampling strategy The sampling strategy was designed to allow the collection of the necessary data to answer key evaluation questions and to analyze the effects of the intervention on intermediate and final outcomes. For this, it was necessary to have a representative sample of beneficiaries from the early and late treatment. To calculate a preliminary sample size, the identification of the following Final Evaluation Report PROGRESA-Caribbean Page 90 of 112 information was required: a) expected effect on outcome variable; b) outcome indicator standard deviation; c) confidence level; and d) statistical power. Regarding the expected effect on outcome variables, there was no official information on key variables such as the poverty indicator and family income in the areas of program intervention. Therefore, it was decided to rely on secondary information in order to obtain the reference values for the intermediate indicators of milk and cacao productivity. Given that there was a strong correlation between productivity and family income and incidence of poverty, the values of these indicators were used to obtain a preliminary estimate of the sample size. The cacao yields came from a combined sample of beneficiaries in two projects, one carried out in the municipality of Siuna by CRS and the other carried out in the municipality of Waslala, El Castillo, San Carlos and Rancho Grande by LWR. Milk yields data corresponded to information from the 2011 National Agricultural Census (CENAGRO) for the program´s intervention zones. Based on this information and the goals discussed with the partners, it was estimated a minimum level of impact for cacao of 0.22 metric tons per hectare, while for livestock it was estimated a minimum level of impact of 0.45 liters of milk per day. According to discussions held with the CRS monitoring and evaluation team, it was very likely that the impacts to be identified would be greater, so this conservative estimate would give a greater precision to the evaluation sample. In reference to the standard deviation of the outcome of interest, it was assumed that the variance between early and late treatment was the same and a confidence level of 95 percent and a statistical power of 80 percent would be used. A power of 80 percent means that the Evaluation Team will find an impact in 80 percent of the cases where one has occurred (Gertler et al., 2011). This information allows one to compute a preliminary sample size and power using the noncentral t-distribution. The experimental design was based on a cluster randomization with the aim to reduce the contamination risk; therefore, the estimated sample size was corrected by the cluster effect. The results of the baseline suggested a representative sample of 980 observations (490 for each treatment status). Assuming a rate of non-response (attrition) of slightly more than 20 percent, the total planned sample was 1,210 producers (605 for each treatment status). The sample design used the list of program participants as the framework. The selection of the sample took place at one stage and was based on simple random sampling. In addition, a random sample of 14 percent of replacements was estimated, which was distributed proportionally in the geographic areas of the program. In the next section, the final data sample is discussed, which turned out to be lower than the planned sample in the baseline because of attrition during the program. Final Evaluation Report PROGRESA-Caribbean Page 91 of 112 Data collection The information for the final evaluation at farmers level came from a data panel that compiles the characteristics of producers and their families for two evaluation rounds: baseline and final. As mentioned later in this document, the mid-term data is not considered for the final evaluation due to comparison difficulties with the final round and intermediate indicators such as yields and agricultural practices. Per the suggestion of USDA in the mid-term evaluation, the data collection for the final evaluation was exclusively the responsibility of the Evaluation Team. Logistics assistance during the field work was provided by CRS and its partners. The baseline and mid-term data collection on the other hand was conducted by the consortium partners at farmers level with technical assistance from the Evaluation Team50 . CRS developed the instrument to be used in the collection of information at farmers level. The Evaluation Team and CRS MEAL team jointly reviewed the instrument to ensure that the questions included all the elements necessary to conduct the evaluation, both those related to the key indicators (intermediate and outcome indicators), as well as the impact mechanisms. The Evaluation Team ensured the survey would not lose comparability with the baseline. The preparation for the field work included training workshops, preparation of a users’ guide, pilot trials, and the creation of route sheets for collection of the information. In addition, the Evaluation Team prepared a protocol for systemic review in order to filter data. This made it possible to reduce measurement errors and to ensure quality in the management of the information. The following databases were obtained from the household survey: 1. Progress out of Poverty Index (PPI) 2. Food security 3. Agriculture: production of cacao 4. Agriculture: sales of cacao 5. Cattle production 6. Cattle inventory 7. Cattle sales 8. Infrastructure 9. Credit 50 The Evaluation Team provided comments on the instrument and randomly supervised 10% of the field work. Final Evaluation Report PROGRESA-Caribbean Page 92 of 112 10. Employment 11. Socio-demographic information 12. Financial education 13. Perception about the program 14. Recession Assessment of the data’s quality and comparability over time The Evaluation Team ensured that during the process of field data collection the same procedures were applied for both the early treatment and the late treatment groups. The digital survey system made the standardized application of the survey possible. This, together with the protocol for systematic review for data cleansing, minimized measurement errors. An important aspect of data quality is the process of building and harmonizing the database. Once the field data was collected and prior to the construction of output and outcome indicators, the Evaluation Team conducted exhaustive reviews of the information collected and requested clarifications from the CRS MEAL team, to ensure the lowest possible data loss, to reduce measurement errors and atypical values in the information, and to ensure transparency in the database-building process. Nevertheless, the data collected was not free from outliers. Though the collected information had some measurement issues either because of overestimation by the farmers or errors in the collection process, outliers were adjusted by the 90 percent winsorizing method. This method designates values located in the 90th percentile to the observations located above said value. Likewise, the expenditure section was prone to measurement errors. These errors were corrected depending on the value chain analyzed. In the case of cattle, expenditures are reported as total costs; therefore, quantities and expenditures were gathered for the entire farm. These components were then converted to quantities per hectare (hectare of pasture), and finally, the winsorizing method was applied at 90 percent for each subsection (labor, supplies, tools, etc.). In the mid-term evaluation, the consortium partners decided to conduct the survey using an evaluation period (6 months) that was different from that of the baseline (1 year). For the final evaluation, the Evaluation Team used an evaluation period of twelve months (1 year). The analysis in this report does not include the indicators in the mid-term data, as the statistical adjustments made might have created a bias in some of the data collected (e.g., annualized sales, agricultural practices). Finally, as mentioned in previous evaluations (baseline and mid-term), the variable of average cattle weight was measured based on the producer's perception. The Evaluation Team considers that this variable may have a bias, as it was uncertain whether the producer´s perception Final Evaluation Report PROGRESA-Caribbean Page 93 of 112 in estimating the weight of cattle would be higher or lower than the true weight. Nevertheless, the data on number of animals and their mean weight in kilograms was collected by sex, age class and predominant breed. This disaggregation might reduce to some extent the potential measurement error. Data collected In the mid-term evaluation, the planned sample size was 945 producers, however, information was collected for 699 producers (74% of the planned sample). These 699 producers constituted the planned final evaluation sample to be used for building the two-period panel data. The Progress Out of Poverty (PPI)51 section was used as the basis for determining the final sample of 598 producer families (85% of the planned sample). To estimate the number of complete interviews, it was determined that the survey should have information on PPI and the inventory of livestock or the area of cacao under development or production52. Taking this into account, a total of 590 complete interviews were obtained, but when the final data was harmonized with the baseline, 53 observations were lost53. The final data was comprised of 537 complete interviews (77% of the planned sample), of which 313 (58%) corresponded to early treatment and 224 (42%) to late treatment. The non-response rate, which corresponds to the percentage of non-responses54 in comparison with the programmed sample, was 23 percent (162 observations). The construction process for the final sample is shown in the following table. 51 This criterion was established in the baseline because PPI was the program’s most important long-term indicator, and because it was the section with the greatest number of interviewees. 52 When the farmer works in only one value chain, the information required comes from the PPI and the production and sales of the respective value chain. 53 This occurs because for some of the previously described indicators, there were no baseline observations for some of the producers and vice-versa, which caused a slight loss of information. 54 The nonresponse is the sum of the non-interviewed plus the incomplete interviews plus the loss due to baseline harmonization. Final Evaluation Report PROGRESA-Caribbean Page 94 of 112 Table C. Final sample and nonresponse of Evaluation Survey During the process of constructing the two-period panel data, the Evaluation Team identified that the information predicted for each value chain changed because 105 producers moved among value chains. While the farmers in both value chains increased by 79 producers between baseline and end-line, the cattle value chain reduced by 69 producers during the same period. These variations are statistically significant. A similar situation arose with the baseline and the mid-term data, which reconfirms that there was migration among value chains impacting some of the results. More importantly, this Final sample and nonresponse of Evaluation Survey Baseline Mid-Term End-line (1) Target Sample 1210 945 699 (2) Initial target sample 1040 945 699 (3) Efective Sample 922 740 598 (4) Replacements 165 - - (5) Use of replacement 104 - - (6) Sample identified by TNS 172 - - (7) Not interviewed 12 205 101 (8) = (3) + (5) + (6) Final Sample 1198 740 598 (9) Incomplete Interviews 253 5 8 (10) Lost due to harmonization with baseline - 36 53 (11) = (7) + (9) + (10) Nonresponse 265 246 162 Nonresponse rate 22% 26% 23% (12) = (8) - (9) - (10) Completed Interviews 945 699 537 (13) Early treatment 534 414 313 Percent intervention 57% 59% 58% (14) Late treatment 411 285 224 Percent intervention 43% 41% 42% Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Final Evaluation Report PROGRESA-Caribbean Page 95 of 112 affected the “power” to assess the impact of the program and increased the error term of the estimations. Table D. Changes on value chain Women made up 14.6 percent of the sample; this proportion was similar for both early and late treatment groups. In terms of the geographical distribution of the sample, 46 percent of the farmers were located in the RACCS, 41.4 percent in the RACCN, and 12.6 percent in Río San Juan. There was relatively greater participation of the RACCS producers in early treatment, whereas in late treatment the largest proportion of producers were in the RACCN. Graph A. Geographical distribution of effective sample Changes on value chain Both Cocoa Catlle Total Both 53 1 2 56 Cocoa 12 218 0 230 Cattle 70 1 180 251 Total 135 220 182 537 End-line Baseline Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Final Evaluation Report PROGRESA-Caribbean Page 96 of 112 Attrition One advantage of the randomized order of phase-in is that it makes it possible for the researchers and those responsible for program monitoring and evaluation to have greater contact with the beneficiaries, which helps reduce potential attrition of the evaluation sample. A potential disadvantage is that the late treatment may be affected by future treatment expectation (Winters et al., 2010). For example, farmers may delay some activities that they normally would have undertaken if they were not part of the project (e.g., seeking technical assistance and access to external markets). In addition, it is possible that producers for late treatment become beneficiaries of other projects, which would then affect the comparison in the evaluation. That said, and as mentioned earlier, the 71 producers who left the program after the mid￾term were not interviewed. The program had dropouts from the evaluation sample for both early and late treatment, as documented in the mid-term report. Table E. Effective allocation by partners The Evaluation Team asked the CRS MEAL team to document the reasons for attrition in the evaluation sample. The reasons for attrition were varied but included: 1.Loss of interest while waiting to join the late treatment. 2. Initiation of work on the government’s ProCacao and NICADAPTA projects while waiting to join late treatment group. 3.Migration to other areas of the country or abroad. 4.Change of economic activity55 . 5.Distance and insecurity. 6.Natural Deaths. 55 These farmers decided to not grow cacao or raise livestock and instead focused on growing other agricultural products or work outside the farm. Effective allocation by partners Early Treatment Late Treatment Total Early Treatment Late Treatment Total Early Treatment Late Treatment Total 414 285 699 354 244 598 85.5% 85.6% 85.6% 59.2% 40.8% 100.0% 59.2% 40.8% 100.0% Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Scheduled sample Complete interview Accomplishment Final Evaluation Report PROGRESA-Caribbean Page 97 of 112 The attrition amongst cooperatives was associated with failing to maintain or secure their registration with the government, opportunity to join government or other NGO funded programs, and economic situation in the country. Box 2. NICADAPTA “Adapting to Markets and Climate Change Project in Nicaragua - NICADAPTA” is executed by the Ministry of Family, Community, Cooperative and Associative Economy (MEFCCA, by its Spanish acronym), with financing from the International Fund for Agricultural Development (FIDA) and the Central American Bank for Economic Integration (BCIE, by its Spanish acronyms). This project is being implemented in areas such as Nueva Segovia, Madriz, Estelí, Jinotega, Matagalpa, Boaco, Río San Juan, RACCS and RACCN. The project targets approximately 40,000 small-scale producers who own coffee or cacao plantations. The duration of the project is from 2014-2020. NICADAPTA aims to contribute to the improvement of rural families´ living conditions by assisting them in their production process and making it more complex (value added). The specific objectives of the project are: 1) strengthen the capacity of organizations to improve productivity and quality of coffee and cacao through improved business and productive management; 2) incorporate investments and management practices that allow improvement of the quality of raw material produced by reducing the impact on the environment, 3) increase the production of coffee and cacao plantations managed by the small producers and the community, and 4) incorporate good management practices and investment decisions that improve adaptation to climate change, including improved water management efficiency and reduced soil loss. The project works to accomplish these objectives through the following main activities. Firstly, by facilitating access to markets for value added coffee and cacao, introducing crop diversification. Second, by communicating to the participant of the program agro-climatic information. Third, improving the capacity of producer organizations through their training. Lastly, by collaborating with the government, cooperation agencies, and private sector to promote and strengthen the coffee and cacao industries. Source: IFAD (2013). “Adapting to markets and climate change project in Nicaragua (NICADAPTA)”, International Fund for Agricultural Development; MEFCCA (2016). “TDR, Proyecto NICADAPTA”, Ministerio de Economía Familiar Comunitaria, Cooperativa y Asociativa. Box 2. NICADAPTA “Adapting to Markets and Climate Change Project in Nicaragua - NICADAPTA” is executed by the Ministry of Family, Community, Cooperative and Associative Economy (MEFCCA, by its Spanish acronym), with financing from the International Fund for Final Evaluation Report PROGRESA-Caribbean Page 98 of 112 The highest distribution of attrition by intervention area, for both early and late treatment, was in the areas of Siuna, Rosita and Waslala, municipalities specifically targeted by NICADAPTA and ProCacao, after the PROGRESA Caribbean program began. When the attrition rates were identified, it was important to examine the potential differences in observable characteristics between producers who left the program and those who remained. Univariate tests were then performed for different baseline indicators between the deserter group and those remaining in the evaluation sample, separating the results by value chain and by assignment to treatment (early and late). Subsequently, probabilistic models were estimated using the baseline characteristics, which were also separated by value chain and the allocation to treatment. The following tables present the results of the mean differences between the groups of producers who left and those who remained in the evaluation sample. As for the socioeconomic well-being indicators, compared to the producers who left the project, the ones who remained in the program are not different in terms of poverty situation but have greater diversity in their diet. Box 3. ProCacao ProCacao is executed by the Nicaraguan Institute of Agricultural Technology (INTA, by its Spanish acronym), with financial support from the Swiss Agency for Development and Cooperation (COSUDE). The purpose of this project is to contribute to the Mining Triangle’srural inhabitants’ food security with a particular emphasis on gender equity, leading to increases in productivity and quality of the cacao value chain through a sustainable agroforestry system. One of the key goals of the program is to manage and implement a cacao clonal garden certification process with the relevant institution. Other objectives include certifying three clonal gardens established by INTA for future plant nurseries; establishing a research system with active involvement of other actors in the territories; devising and carrying out an economic sustainability strategy for the cacao research system; training producers and other participants on material related to value chains; and promoting agroforestry systems, among others. Source: INTA (2019). “ProCacao/Cosude”, Instituto Nicaragüense de Tecnología Agropecuaria. Box 3. ProCacao ProCacao is executed by the Nicaraguan Institute of Agricultural Technology (INTA, by its Spanish acronym), with financial support from the Swiss Agency for Development and Cooperation (COSUDE). This purpose of this project is contributing to the Mining Triangle rural inhabitants’ food sovereignty with a particular emphasis on gender equity, leading to increases in productivity and quality on the cacao value chain through a sustainable agroforestry system. One of the key targets is to manage and implement a clonal garden certification process with the relevant institution. Other objectives include certifying three clonal gardens devised by INTA for future plant nurseries; establishing a research system with an active involvement of actors in the territories; devising and carrying out an economic sustainability strategy for the cacao research system; training producers and remaining participants in contents related to value chains; and promoting agroforestry systems, among others. Source: INTA (2019). “ProCacao/Cosude”, Instituto Nicaragüense de Tecnología Agropecuaria. Table 23. PPI (%) transition matrix Box 3. ProCacao ProCacao is executed by the Nicaraguan Institute of Agricultural Technology (INTA, by its Spanish acronym), with financial support from the Swiss Agency for Development and Cooperation (COSUDE). Final Evaluation Report PROGRESA-Caribbean Page 99 of 112 Table F. General – Two-sample mean comparison tests (univariate analysis) For livestock producers, there was a statistical difference between the late treatment producers who stayed compared to those who left. Those who remained had, on average, higher gross income and more livestock and cows in production. Table G. Cattle – Two-sample mean-comparison tests (univariate analysis) General - Two-sample mean comparison tests (univariate analysis) Attrition Complier Difference Attrition Complier Difference Attrition Complier Difference PPI (%) 59.262 61.514 2.252 54.254 60.118 5.864** 64.732 63.589 -1.142 HDDS 6.688 6.987 0.298** 7.034 7.101 0.067 6.333 6.823 0.490** MAHFP (months) 11.144 11.161 0.016 11.179 11.189 0.010 11.109 11.121 0.012 Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Total Early treatment Late treatment Notes: *** Significant at the 1 percent level. ** Significant at the 5 percent level. * Significant at the 10 percent level. Indicator Cattle - Two-sample mean-comparison tests (univariate analysis) Attrition Complier Difference Attrition Complier Difference Attrition Complier Difference Gross income - cattle ($) 8837.75 8464.25 -373.49 7258.92 8424.57 1165.65 10839.08 8532.08 -2307.00** Milk yields (liters of milk per cow per day) 2.75 2.60 -0.15 2.74 2.47 -0.27 2.76 2.82 0.06 Total farm area (mzn) 70.82 76.05 5.23 68.02 76.35 8.33 74.17 75.55 1.38 Pasture area (mzn) 53.92 61.50 7.58 56.65 60.51 3.87 50.68 63.15 12.47 Total livestock (number of heads) 49.23 51.60 2.37* 50.52 49.68 -0.84 47.71 54.82 7.12** Cows in production (number of heads) 19.88 20.34 0.46 20.90 19.20 -1.70 18.61 22.26 3.65** Average cattle weight per hectare (kg/ha) 349.69 389.33 39.64 354.66 387.44 32.78 343.75 392.48 48.73 Improved pasture (% of producers) 0.35 0.32 -0.03 0.36 0.31 -0.05 0.33 0.32 0.00 Gross margin - Cattle (%) -42.78 -40.16 2.62 -33.97 -39.36 -5.40 -53.95 -41.51 12.44 Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Total Early treatment Late treatment Notes: *** Significant at the 1 percent level. ** Significant at the 5 percent level. * Significant at the 10 percent level. Indicator Final Evaluation Report PROGRESA-Caribbean Page 100 of 112 For cacao producers, there were differences in the proportion of area under production in the late treatment. The producers who left the late treatment had, on average, a greater proportion of area under production than those who stayed. Table H. Cacao – Two-sample mean-comparison tests (univariate analysis) To understand the determinants on attrition, the Evaluation Team estimated econometric models56 to measure the probabilities of remaining in the program. In the livestock value chain, when controlled by a set of socioeconomic and productive variables, it was observed that for late treatment producers there were differences between those who remained and those who abandoned the program. Specifically, based on the estimated models, the probabilities of remaining in the program were reduced for those producers with a higher proportion of pasture out of the total farm area, a larger farm size, and a greater average weight of cattle per hectare. By contrast, greater dietary diversity and a larger herd of cattle increased the probabilities of remaining in the program. It was also determined that belonging to a cooperative reduced the probabilities of remaining in the program. 56 Probabilistic models of the baseline characteristics. Cacao - Two-sample mean-comparison tests (univariate analysis) Attrition Complier Difference Attrition Complier Difference Attrition Complier Difference Gross income - cocoa ($) 744.58 808.37 63.80 654.36 779.00 124.64 824.60 849.35 24.75 Amount harvested (qq) 19.00 21.38 2.39 18.22 19.87 1.65 19.68 23.48 3.80 Cocoa in pulp yields per hectare (tm/ha) 0.69 0.77 0.08 0.71 0.78 0.07 0.67 0.75 0.08 Dry cocoa yields per hectare (tm/ha) 0.23 0.26 0.03 0.24 0.26 0.02 0.22 0.25 0.03 Total area of cocoa (mzn) 2.53 2.45 -0.08 2.53 2.15 -0.38 2.53 2.87 0.34 Amount of sales in dry cocoa (qq) 18.69 21.04 2.35 17.91 19.49 1.59 19.37 23.22 3.85 Value of production per hectare ($/ha) 542.74 614.52 71.78 562.27 633.80 71.53 525.41 587.62 62.21 Application of sustainable agricultural practices (%) 0.02 0.04 0.02 0.03 0.05 0.02 0.01 0.02 0.01 Proportion of area in production (%) 0.84 0.79 -0.05** 0.84 0.80 -0.04 0.84 0.78 -0.06** Gross margin (%) -216.97 -246.67 -29.70 -248.70 -292.56 -43.85 -189.66 -182.12 7.53 Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Total Early treatment Late treatment Notes: *** Significant at the 1 percent level. ** Significant at the 5 percent level. * Significant at the 10 percent level. Indicator Final Evaluation Report PROGRESA-Caribbean Page 101 of 112 Table I. Cattle – Probability of attrition (marginal effects) In contrast, slight differences were observed in the cacao value chain in both early and late treatment. Greater dietary diversity increased the probabilities of remaining in the program for those in the late treatment and belonging to a cooperative increased the probabilities of remaining in the program for those in the early treatment. Table J. Cacao – Probability of attrition (marginal effects) Cattle - Probability of attrition (marginal effects) Covariate Early Treatment Late Treatment General Water supply all year round (yes=1) -0.16 0.07 0.00 Area of pasture as proportion of farm (%) -0.31 -0.82** -0.43** Improved pasture area as a proportion of total pasture area (%) -0.14 -0.18 -0.13 PPI (%) 0.09** 0.03 0.06** Total area of the farm (mzn) -0.04 -0.53** -0.20* Non-agricultural income ($) 0.03** -0.02 0.01 HDDS 0.08 0.47*** 0.27** Total livestock (number of heads) 0.06 0.60** 0.23* Average cattle weight per hectare (kg/ha) -0.05 -0.42** -0.16 Milk yields (liters of milk per cow per day) -0.09 0.12 -0.01 Belongs to a cooperative (%) -0.03 -0.22** -0.12* Notes: *** Significant at the 1 percent level. ** Significant at the 5 percent level. * Significant at the 10 percent level. Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Cacao – Probability of attrition (marginal effects) Early Treatment Late Treatment General Proportion of area in production (%) -0.25 -0.26 -0.23 PPI (%) 0.04 -0.05 0.01 HDDS 0.05 0.25* 0.17* Amount harvested (qq) 0.00 0.03 0.01 Total area of cocoa (mzn) -0.12 0.03 -0.05 Cocoa in pulp yields per hectare (tm/ha) 0.03 0.03 0.04 Belongs to a cooperative (%) 0.12* -0.05 0.02 Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Notes: *** Significant at the 1 percent level. ** Significant at the 5 percent level. * Significant at the 10 percent level. Final Evaluation Report PROGRESA-Caribbean Page 102 of 112 From the above discussion, the Evaluation Team concluded that the observed attrition correlates with treatment allocation, at least for the case of the cattle value chain. In the late treatment "the best ones remained", that is, those farmers with the best productive performances. The implications of this are addressed in the next section. Revalidation of the evaluation design In the previous section it was verified that although the causes of attrition were mostly exogenous to the management of PROGRESA-Caribbean, there was a degree of correlation between attrition and treatment allocation. In this regard, when recalculating the baseline indicators with the sample of producers for the final two-period panel data, it was found that the indicators differed considerably. Thus, to avoid bias in time comparisons, the Evaluation Team recalculated a new baseline data for the purpose of this analysis. The Table below reports the average of the indicators for the late treatment sample, as well as the difference between early and late treatment. Table K. Re-estimation of baseline differences between early treatment and late treatment at program level Re-estimation of baseline differences between early treatment and late treatment at program level Late treatment mean Early treatment - Late treatment difference p- value Observations Late treatment mean Early treatment￾Late treatment difference p - value Observations Progress out of Poverty Index (PPI) - general (%) 64.13 -6.35 0.145 945 62.36 -1.35 0.564 535 Progress out of Poverty Index (PPI) - extreme (%) 25.26 -5.40 0.130 945 23.90 -2.65 0.138 535 MAHFP (months) 11.13 0.02 0.985 923 11.13 0.07 0.594 516 HDDS 6.57 0.46 0.115 923 6.85 0.21 0.160 516 Percentage of families with insufficient food (%) 0.31 0.04 0.670 923 0.32 0.02 0.704 516 Total sales (dollars) 4,714.77 -64.45 0.995 900 4,674.79 539.43 0.618 511 Cocoa sales (dollars) 850.62 -79.90 0.655 527 860.07 -96.22 0.481 282 Livestock sales (dollars) 9,668.53 -1800.00 0.705 442 8,385.01 356.58 0.845 270 Milk yields (lts of milk per cow per day) 2.76 -0.17 0.475 468 2.76 -0.28 0.243 290 Average cattle weight per hectare (kg / ha) 365.76 7.73 0.870 503 397.37 -12.05 0.814 301 Gross Income - cattle (dollars) 9,232.88 -1470.00 0.635 485 8,455.41 14.63 0.994 298 Value of production per hectare - cattle 174.56 21.74 0.395 440 208.67 5.91 0.810 294 Gross margin per unit of land - cattle (%) -44.44 5.87 0.605 485 -42.00 2.50 0.771 294 Dry cocoa yields (tm/ha) 0.23 0.02 0.610 532 0.25 0.01 0.750 285 Hectares of cocoa in production (ha) 1.45 -0.21 0.135 533 1.51 -0.38 0.018 2285 Hectares of cocoa in develpment (ha) 0.93 -0.11 0.270 241 0.95 -0.15 0.255 142 Gross income - cocoa (dólares) 858.15 -84.12 0.605 532 867.51 -97.48 0.514 284 Value of production per hectare- cocoa (dollars / ha) 565.04 71.50 0.670 527 594.67 33.71 0.613 282 Gross margin per unit of land - cocoa (%) -208.45 -52.86 0.650 532 -179.33 -120.43 0.298 282 Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Indicator Baseline 2015 New baseline (producers that remained in the project) Notes: The table reports the difference in each variable between the early treatment and late treatment groups. Inference is adjusted by bootstrap standard errors. Penultimate column reports the p-value of tests of differences for each variable. *** Significant at the 1 percent level. ** Significant at the 5 percent level. * Significant at the 10 percent level. Final Evaluation Report PROGRESA-Caribbean Page 103 of 112 The results suggest that there were no statistically significant differences between the characteristics of the early and late treatment group. This apparent "balance" in most sociodemographic and production indicators does not imply that there were no exogenous flaws in randomization. Rather, it may reveal an issue with the statistical power required to identify statistical differences between groups. In the baseline report, the Evaluation Team explained the potential presence of this problem, should a large amount of evaluation sample be lost. Although the level of attrition affected the randomization and the statistical power, having a baseline and a final follow-up survey (panel data) gives greater statistical power compared to a cross-section of data, which mitigates power losses to some extent, even though the final sample was smaller. Therefore, in order to not affect the internal validity of the evaluation, the analysis was carried out using the methodology of matching estimators combined with difference-in￾differences, as explained in the next section. This statistical method paired producers from the early treatment group with similar farmers from the late treatment group based on several characteristics not related to the intervention57 to determine the probability of selection in the program. Econometric method For the final evaluation, we aimed at estimating the Average Treatment Effect on the Treated (ATET). Under a standard evaluation of a binary treatment the ATET would be: 𝐴 𝐸 = 𝐸[∆ | = ] = 𝐸[𝑌1 𝑌0 | = ] = 𝐸[𝑌1 | = ] 𝐸[𝑌0 | = ] The key was to identify the counterfactual 𝐸[𝑌0 | = ], i.e., the potential outcome as untreated, for the treated ones. The intervention strategy made it possible to randomly identify a temporary counterfactual group and to randomize the duration of the program for each of the beneficiaries. This ensured that the treatment assignment was statistically independent of potential outcomes, then: 𝐴 𝐸 = 𝐸[𝑌1 | = ] 𝐸[𝑌0 | = ] 𝐴 𝐸 = 𝐸[𝑌1 | = ] 𝐸[𝑌0 | = 0] Randomization causes that the potential outcome as untreated for those who were actually not treated (𝐸[𝑌0 | = 0]) is a valid counterfactual. 57 The characteristics used to create the pairings were related to sociodemographic characteristics of the producers like age, years of schooling, members of the household and gender. Also, it accounted for credit access and control for geographic localization (municipality). Lastly, the estimation included an agricultural practices index. Final Evaluation Report PROGRESA-Caribbean Page 104 of 112 Alternatively, the difference-in-means estimator could be specified through the following regression: 𝑌 = 𝛽0 𝛽1 𝜀 Where i corresponds to the unit of analysis (the producer). Randomization ensures that is independently distributed from the unobservable characteristics (𝜀 ), so it accomplishes the zero￾conditional mean assumption (𝐸[𝜀 | = 0]). 𝛽1 measures the causal effect (or average treatment effect): 𝐴 𝐸 = 𝛽1 = 𝐸[𝑌 | = ] 𝐸[𝑌 | = 0] Given that the data was collected for two periods (baseline and end line), the Evaluation Team was able to exploit the panel data structure by computing the ATET in the final evaluation using the difference-in-differences methodology. This method consists of the double difference calculation: ∆̂ = (𝑌̅ , 𝑓 𝑌̅ , 𝑓 ) (𝑌̅ , 𝑓 𝑌̅ , 𝑓 ) The first difference corresponds to the difference in the after-and-before outcomes for the treatment group (early treatment), which controls for time-invariant characteristics. The second difference relates to the difference in the after-and-before outcomes for the comparison group (late treatment). Subtracting the first difference from the second, isolates the effect of time-varying factors, under the assumption that both treatment and comparison groups were exposed to the same environmental conditions (parallel trend assumption); this assumption is plausible in the context of PROGRESA Caribbean, given the targeting in the Caribbean Coast and Río San Juan are locations where farmers tend to face the same environmental conditions. This method provides a more precise estimate of the impact and allows the treatment and comparison group to differ in characteristics, provided that the parallel assumption trend is accomplished and that the unobservable characteristics of the beneficiaries are time invariant. This is very important, as the balance in observable characteristics between early and late treatment was affected by attrition (nonresponse) of the evaluation sample. The Evaluation Team found a bias in the evaluation sample, as the attrition was correlated with treatment status. In the potential outcome framework, the difference-in-differences parameter can be expressed as follows: 𝛿𝐴 𝐸 = 𝐸[𝑌 ,2 𝑌 ,1 | = ] 𝐸[𝑌 ,2 𝑌 ,1 | = 0] Where 𝑌 ,2 is the outcome of producer at the final evaluation and 𝑌 ,1 is the potential outcome of the producer at baseline. The potential outcome depends on whether the producer Final Evaluation Report PROGRESA-Caribbean Page 105 of 112 was assigned to be treated early ( = ), i.e. in the first 18 months of the program implementation, or if s/he was assigned to be treated late ( = 0), that is, from the 19th month. The difference-in-differences parameter can also be specified in the following regression: ∆𝑌 = 𝛽0 𝛽1 𝜆Δ𝜇 Δ𝜀 where 𝛥 reflects the change in the variable from baseline to final (at baseline, is 0 for all). 𝑌 represents the outcome of farmer in cluster at time . 𝛽1 corresponds to the impact coefficient, i.e. the final program impact would be given by the binary variable that captures the early (yes=1) and late treatment (no=0) assignment ( ). 𝜇 corresponds to time-invariant observable and unobservable characteristics of producer families, for example education and producer skills, sex, soil quality, etc. 𝜀 is a time-varying idiosyncratic error. The equation above was estimated using fixed effects. For the design, the difference-in-differences technique allowed the isolation of the self￾selection bias (previously discussed), as it was assumed that the motivation to participate in the program was a time-invariant unobservable characteristic. Randomized Control Trials (RCTs) are very challenging to implement, as usual problems include spillover and attrition, especially at large scale. As mentioned earlier, the Evaluation Team found a bias in the evaluation sample for the final round, as attrition was correlated with treatment status. This made the RCT strategy no longer viable. Therefore, it was decided to apply the combination of two techniques in order to isolate the bias: difference-in-differences (which isolates time invariant characteristics that might be correlated with treatment status) with matching estimators (which mitigates the bias by observable characteristics). Matching estimators are one of the most popular in the impact evaluation literature when the experimental design is affected by attrition; it is considered as a quasi-experimental method. This method evaluates the effect of an intervention by comparing outcomes between similar groups. The comparison group is determined to be a suitable match by measures of observed characteristics. Difference-in-difference combined with matching estimators compare the change (over time) in outcomes for treatment to the change in outcomes for the comparison group member (Ravallion & Jalan, 1999). In the context of this evaluation, the idea behind matching estimators was to statistically identify (based on observable characteristics) two “identical” individuals in the data, with the exception that one was treated earlier and the other individual was treated later, so that any difference in the result between the two may be attributed to the program. Final Evaluation Report PROGRESA-Caribbean Page 106 of 112 Matching requires the assumption that the selection in the program is only based on observable characteristics, which can also affect the expected results (conditional independence assumption). If this assumption is fulfilled, conditional on a set of covariates, the allocation of treatment is random. Therefore, it is assumed that two individuals with the same characteristics, one treated and the other not, have the same counterfactual (Caliendo and Kopeing, 2008). The key to this method lies on the careful identification of the “clone” for the treated individual. However, as mentioned earlier, there might be time invariant characteristics that are correlated with treatment status, so this technique was combined with the difference-in-differences estimation in order to avoid this potential bias. The first step was to estimate the probability of early participation in the program for the sample of early and late treatment, using observable characteristics as covariates, which jointly affect the probability of participation and the potential outcome; this is called propensity score. Estimating the propensity score requires choosing a set of 𝑋 conditioning variables not related to the intervention. In the context of this evaluation, producers were assigned into one of two groups: those assigned to an early treatment ( = ) and those assigned to late treatment ( = 0) on the basis of the propensity score: 𝑋 = = |𝑋 0 < 𝑋 < Where 𝑋 is a vector of control variables at baseline. The vector of coefficients is comprised of producers’ years of schooling and education, credit access, gender, number of members in the households (nonlinear) and municipal fixed effects. The following table presents the estimates for the selection equation for producers of the three commodities evaluated: cacao, milk and beef58 . 58 For the cattle production, the selection equation excludes producers age. Final Evaluation Report PROGRESA-Caribbean Page 107 of 112 Table L. Propensity Score: Preconditioning variables Using the estimated probabilities for each individual, the sample is restricted to a region of common support; in other words, to individuals in both groups (treatment and comparison) that have similar probability of being treated. It used a kernel regression (matching algorithm) estimator which chooses the weight so that the observation closer in terms of the distance receives greater weight (Heckman, Ichimura & Todd, 1998). The selected equation matched between 82 and 94 percent of the sample, satisfying the support property. The next step was to construct the matched outcomes: 𝐸 𝑌0 | 𝑋 , = 0 Once the matching was successful, The Evaluation Team computed the impact of the program, i.e. the difference in the mean outcome between the treated (early treatment) and the comparison group (late treatment) for the region of common support. The following table presents the main results from the difference-in-difference estimation. Cacao Milk Cattle Years of Schooling 0.060 -0.003 0.010 (0.029) (0.024) (0.022) Age 0.000 -0.003 (0.007) (0.007) Members 0.008 -0.017 -0.020 (0.009) (0.010) (0.009) Members square -0.163 0.183 0.228 (0.109) (0.120) (0.116) Credit Access 0.257 -0.564 -0.458 (0.211) (0.247) (0.234) Female -0.059 0.095 0.004 (0.242) (0.269) (0.259) Municipal Fixed Effects Yes Yes Yes N 257 235 248 Standard Errors in Parenthesis. Propensity Score: Pre conditioning variables Final Evaluation Report PROGRESA-Caribbean Page 108 of 112 Table M. Productivity Impact Estimation Results Annex VI. Index of agricultural practices The following tables show the practices, activities, and scores defined to calculate the Index of Practices for the Management of Livestock (IPMG) and the Cultivation of Cacao (IPCC). Both indexes have a scale from 0 to 100, where 0 represents the non-performance of any of the practices evaluated, and 100 represents that all practices were carried out according to the recommendations given by the consortium partners. To evaluate the integral performance of the surveyed producers, it is suggested that the threshold to consider that the producers have applied the practices correctly is 60 points in both indexes. This makes it possible to differentiate between producers that met at least half of the practices recommended by the program and performed them with good quality from those that did not. The IPMG was composed of 4 essential practices: nutrition, pasture management, vaccination and deworming. Each practice included a set of indicators that allowed an approximation of their quality. In the case of this index, equitable distribution of the score was made in each of the practices. The IPCC was composed of five practices: pruning, amendment establishments, fertilizer use, and shade management. Each practice was disaggregated by a set of activities that allowed an approximation of their quality. Also, each of the practices included in the index had different Cacao (MT/ha) Milk (lts/cow/day) Beef (kg/ha) Endline 0.0655 -0.0486 115.3* (0.0568) (0.220) (68.41) Early Treatment 0.0152 0.0938 0.760 (0.0567) (0.220) (68.41) Early Treatment*Endline 0.134* -0.126 -18.07 (0.0803) (0.312) (96.75) Individual Fixed Effects Yes Yes Yes Observations 498 452 474 R-squared 0.035 0.002 0.010 Mean control t(0) 0.243 2.518 409.1 Mean treated t(0) 0.258 2.612 409.8 Diff t(0) 0.0152 0.0938 0.760 Mean control t(1) 0.309 2.470 524.4 Mean treated t(1) 0.458 2.437 507.1 Diff t(1) 0.149 -0.0325 -17.31 Standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 Productivity Impact Estimation Results Final Evaluation Report PROGRESA-Caribbean Page 109 of 112 scores, which were assigned given the activities carried out within each practice and the importance of these. Table N. Index of Practices for the Management of Livestock (IPML) Activity Practice Criterion Score Improved grass Farm with improved grass manzanas 7.5 Concentration of animals Farm with at least 0.4 improved grass manzanas per cattle head 7.5 Mineral salts Producer provides mineral salts for cattle 2.5 Salt frequency Producer provides mineral salts for cattle at leat three times per week 2.5 Mineral salts and common salt ratios Salt mixture provided by the producer includes at leat 50% of mineral salts. 2.5 Salt quantity Saltpeters with residues at the end of day 2.5 Cattle have a natural and permanent water source at a distance of less than 500 meters 5.0 There are water containers in the pens 10.0 Number of pastures There are at least 16 pastures in the farm 10.0 Days of grazing Pastures are used for a maximum of 3 days 7.0 Days off Pastures has at leat 30 recovery days before being used again. 7.0 Pasture cleaning Cleaning is carried out with machetes or chemical substances 3.0 Fertilizer Fertilizers are used in the paddocks 3.0 Vaccine application Cattle are vaccinated 2.0 Anthrax vaccine application Cattle are vaccinated against anthrax at least twice in a year 2.0 Vaccine transfer Producer uses an ice pack or a flask to get vaccines transferred 4.0 Reuse of needles The same needle is used to vaccine several cattle 2.0 Application of deworming medications Oral or injected dewoming medications are applied 5.0 Deworming frequency Deworming medications are applied at least twice in a year 5.0 Washings Animals are washed 5.0 Washing frequency Animals are washed at leat twice in a year 5.0 100.0 Index of Practices for the Management of Livestock (IPML) Pasture management (30 points) Vaccination (10 points) Deworming (20 points) TOTAL SCORE Nutrition (40 puntos) Water availability Final Evaluation Report PROGRESA-Caribbean Page 110 of 112 Table O. Index of Practices for the Cultivation of Cacao (IPCC) Annex VII. Production value for the program Table P. Value of production at project level Activity Practice Criterion Score Pruning Pruning is carried out 5.6 Blunting is done 2.2 Made at least twice in a year 2.6 Sucker removal is done 2.2 Made at least twice in a year 2.6 Dead branches are cut off 2.2 Made at least twice in a year 2.3 Crossing branches are cut off 2.2 Made at least twice in a year 2.6 Use of appropriate tools No use of machete for pruning 8.9 Weed removal at the base of the tree is done 3.3 Made at least twice in a year 3.3 Amendments are set 5.6 Ash and lime amendments 7.8 Amendments set for at least 1 pound per plant 7.8 Fertilizer use (11.1 points) Fertilization Fertilizing is used 11.1 Shade handling is carried out 2.8 The producer considers that shades are well or regularly distributed 3.3 Thining Thining is done 3.9 Trimming the top of the shade trees are done 3.3 It is done at least twice at year 3.3 Trimming the lateral branches is done 3.9 It is done at least twice at year 3.9 Shade setting Shade setting is performed 3.3 100.0 Index of Practices for the Cultivation of Cacao (IPCC) Amendments (27.8 points) Weed removal at the base of the tree Amendments settings Pruning(33.3 points) Blunting Suckers removal Dead branch removal Crossing branches removal Shade management (27.8 points) Shade handling Trimming the top of the shade trees Trimming the lateral branches TOTAL SCORE Value of production at project level Baseline End-line Baseline End-line (A) (B) (C) (A×B) (A×C) Early Treatment 1789 15,902.7 12,330.9 28.45 22.06 Late Treatment 1312 21,974.1 18,102.1 28.83 23.75 Total 3101 18,439.2 14,772.7 57.18 45.81 Early Treatment 2333 1,337.3 2,164.6 3.12 5.05 Late Treatment 1868 1,889.7 2,152.0 3.53 4.02 Total 4201 1,583.0 2,159.0 6.65 9.07 Early Treatment 2882 10,919.5 9,406.7 31.47 27.11 Late Treatment 2282 14,184.9 12,173.5 32.37 27.78 Total 5164 12,362.5 10,627.4 63.84 54.88 Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Average productive value of the sample (US$) Total productive values of the universe (US$ millions) Cattle Cacao Project Chain Treament Universe of producers Final Evaluation Report PROGRESA-Caribbean Page 111 of 112 Annex VIII. Statistical tests Table Q. T-test for cattle indicators Baseline Endline Difference P-value Baseline Endline Difference P-value Baseline Endline Difference P-value Area (Ha) 44.0 38.7 -5.2 0.358 43.8 41.7 -2.1 0.737 43.9 40.0 -3.9 0.346 Average cattle weight (kg/Ha) 389.7 493.9 104.1 0.038 398.2 508.8 110.7 0.118 393.1 499.9 106.8 0.010 Milk yields (lts per cow per day) 2.4 2.4 0.0 0.876 2.8 2.4 -0.3 0.185 2.6 2.4 -0.2 0.276 Revenues ($) per ha 207.9 183.7 -24.3 0.267 212.3 183.5 -28.8 0.229 209.7 183.6 -26.1 0.108 Kg. per animal 261.5 252.7 -8.8 0.047 248.5 250.3 1.7 0.729 256.3 251.7 -4.6 0.174 Vaccination 0.8 0.8 0.1 0.035 0.8 0.9 0.1 0.003 0.8 0.9 0.1 0.001 Vitamin 1.0 1.0 0.0 0.178 1.0 1.0 0.0 0.653 1.0 1.0 0.0 0.203 Deworming 0.9 1.0 0.1 0.000 0.8 1.0 0.1 0.001 0.9 1.0 0.1 0.000 Cattle identification 0.2 0.6 0.4 0.000 0.2 0.6 0.3 0.000 0.2 0.6 0.4 0.000 Clean milking 0.1 0.3 0.2 0.000 0.1 0.3 0.1 0.003 0.1 0.3 0.2 0.000 Tuberculosis 0.0 0.2 0.1 0.000 0.0 0.2 0.2 0.000 0.0 0.2 0.1 0.000 Mastitis 0.1 0.4 0.3 0.000 0.0 0.4 0.3 0.000 0.1 0.4 0.3 0.000 Dehorning 0.6 0.8 0.2 0.001 0.6 0.8 0.2 0.001 0.6 0.8 0.2 0.000 Pasture rotation 0.7 1.0 0.3 0.000 0.7 1.0 0.3 0.000 0.7 1.0 0.3 0.000 Artificial insemination 0.0 0.0 0.0 0.654 0.0 0.0 0.0 1.000 0.0 0.0 0.0 0.780 Expenditure management 0.0 0.2 0.2 0.000 0.0 0.2 0.1 0.001 0.0 0.2 0.2 0.000 Savings management 0.0 0.1 0.1 0.001 0.0 0.1 0.1 0.018 0.0 0.1 0.1 0.000 Investment management 0.0 0.1 0.1 0.000 0.0 0.1 0.1 0.001 0.0 0.1 0.1 0.000 Debt management 0.1 0.1 0.0 0.104 0.0 0.1 0.0 0.122 0.0 0.1 0.0 0.025 Sales management 0.0 0.4 0.4 0.000 0.0 0.4 0.4 0.000 0.0 0.4 0.4 0.000 Inventory management 0.1 0.3 0.2 0.000 0.0 0.3 0.3 0.000 0.0 0.3 0.3 0.000 Improved pasture area (mzn) 16.6 25.1 8.5 0.013 13.6 18.8 5.3 0.112 12.4 18.1 5.7 0.004 Corrals 0.6 0.9 0.3 0.000 0.6 0.9 0.3 0.000 0.6 0.9 0.3 0.000 Agricultural tools 0.9 1.0 0.1 0.013 0.9 0.9 0.0 0.329 0.9 0.9 0.0 0.247 Pasture grinder 0.1 0.1 0.0 0.190 0.1 0.1 0.0 0.450 0.1 0.1 0.0 0.137 Milk jugs 0.5 0.5 0.0 0.672 0.4 0.4 0.0 0.797 0.4 0.5 0.0 0.623 Water tank 0.1 0.2 0.1 0.011 0.1 0.2 0.0 0.291 0.1 0.2 0.1 0.008 Water troughs 0.1 0.3 0.2 0.000 0.1 0.2 0.2 0.000 0.1 0.2 0.2 0.000 Milking parlors 0.0 0.2 0.1 0.000 0.0 0.2 0.2 0.000 0.0 0.2 0.2 0.000 t-test for cattle indicators Early Treatment Late-Treatment Project Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Final Evaluation Report PROGRESA-Caribbean Page 112 of 112 Table R. T-test for cacao indicators Table S. T-test for socio-demographic indicators Baseline Endline Difference P-value Baseline Endline Difference P-value Baseline Endline Difference P-value Production area 1.2 1.5 0.4 0.005 1.5 1.7 0.2 0.302 1.3 1.6 0.3 0.005 Dry cacao yields (MT/Ha) 0.3 0.5 0.2 0.005 0.2 0.3 0.1 0.006 0.3 0.4 0.1 0.000 Revenues ($) per ha 642.2 789.8 147.6 0.247 557.0 574.2 17.3 0.804 606.3 697.7 91.4 0.248 Buds elimination 0.8 1.0 0.1 0.000 0.8 1.0 0.2 0.000 0.8 1.0 0.1 0.000 Maintenance pruning 0.8 1.0 0.2 0.000 0.8 1.0 0.1 0.000 0.8 1.0 0.2 0.000 Rehabilitation pruning 0.1 0.7 0.6 0.000 0.2 0.7 0.5 0.000 0.1 0.7 0.6 0.000 Weed removal at the base of the trees 0.9 0.9 0.1 0.121 0.8 0.9 0.1 0.002 0.8 0.9 0.1 0.001 Bordeaux mixture 0.0 0.2 0.2 0.000 0.0 0.3 0.3 0.000 0.0 0.2 0.2 0.000 Foliar manure 0.1 0.2 0.1 0.179 0.1 0.1 0.1 0.041 0.1 0.1 0.1 0.023 Incorporation of green manure 0.1 0.2 0.2 0.000 0.0 0.2 0.2 0.000 0.0 0.2 0.2 0.000 Ground cover management 0.3 0.3 0.0 0.592 0.2 0.3 0.1 0.060 0.2 0.3 0.1 0.115 OCSA 0.1 0.1 0.0 0.490 0.0 0.1 0.1 0.054 0.0 0.1 0.0 0.096 Organic fertilization 0.1 0.2 0.1 0.017 0.1 0.3 0.2 0.000 0.1 0.2 0.1 0.000 Crop genetics 0.4 0.5 0.1 0.011 0.4 0.5 0.1 0.165 0.4 0.5 0.1 0.004 Pest management 0.1 0.3 0.2 0.000 0.1 0.4 0.4 0.000 0.1 0.4 0.3 0.000 Disease control 0.4 0.9 0.5 0.000 0.5 1.0 0.5 0.000 0.5 0.9 0.5 0.000 Expenditures management 0.1 0.2 0.1 0.130 0.1 0.3 0.2 0.001 0.1 0.2 0.1 0.001 Savings management 0.0 0.1 0.1 0.016 0.1 0.1 0.1 0.062 0.0 0.1 0.1 0.003 Investment management 0.1 0.1 0.1 0.015 0.1 0.2 0.1 0.015 0.1 0.2 0.1 0.001 Debt management 0.1 0.1 0.1 0.079 0.0 0.2 0.1 0.002 0.0 0.1 0.1 0.001 Sales records 0.1 0.5 0.4 0.000 0.2 0.6 0.4 0.000 0.1 0.6 0.4 0.000 Farm plan 0.1 0.2 0.0 0.561 0.1 0.2 0.1 0.050 0.1 0.2 0.1 0.089 Cost and production records 0.1 0.2 0.1 0.007 0.0 0.2 0.2 0.000 0.1 0.2 0.1 0.000 Production plan 0.2 0.4 0.2 0.000 0.3 0.4 0.1 0.323 0.3 0.4 0.2 0.000 Agricultural tools 0.9 0.9 0.1 0.073 0.8 1.0 0.2 0.000 0.8 0.9 0.1 0.000 Toolkits 0.3 0.5 0.2 0.000 0.3 0.5 0.2 0.001 0.3 0.5 0.2 0.000 Chainsaws 0.1 0.2 0.0 0.523 0.1 0.2 0.1 0.165 0.1 0.2 0.0 0.163 Fumigation pumps 0.7 0.7 0.0 0.727 0.7 0.8 0.0 0.520 0.7 0.8 0.0 0.494 t-test for cacao indicators Early Treatment Late-Treatment Project Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Baseline Endline Difference P-value Baseline Endline Difference P-value Baseline Endline Difference P-value Household Members 4.8 4.9 0.1 0.453 5.2 5.2 0.0 0.874 5.0 5.0 0.1 0.685 Large household 0.2 0.2 0.0 0.166 0.3 0.2 0.0 0.247 0.2 0.2 0.0 0.068 Head Age 45.3 49.2 3.9 0.000 47.4 51.5 4.1 0.002 46.2 50.1 4.0 0.000 Young Household 0.4 0.2 -0.1 0.002 0.3 0.2 -0.1 0.005 0.3 0.2 -0.1 0.000 Years of schooling 4.2 5.3 1.1 0.000 4.0 4.8 0.8 0.003 4.1 5.1 1.0 0.000 Years of schooling- Head 3.9 3.9 0.0 0.928 3.2 3.2 -0.1 0.799 3.6 3.6 0.0 0.840 Experience 14.8 14.3 -0.4 0.599 13.9 13.8 -0.1 0.952 14.4 14.1 -0.3 0.651 Food shortage 0.3 0.1 -0.2 0.000 0.3 0.2 -0.2 0.000 0.3 0.2 -0.2 0.000 Months with sufficient food 11.2 11.7 0.5 0.000 11.1 11.6 0.5 0.000 11.2 11.6 0.5 0.000 HDDS 7.1 7.7 0.6 0.000 6.8 7.5 0.7 0.001 7.0 7.6 0.6 0.000 Credit Access 0.2 0.2 0.0 0.834 0.2 0.1 0.0 0.201 0.2 0.1 0.0 0.556 PPI general 60.1 53.0 -7.2 0.001 63.6 55.1 -8.5 0.003 61.5 53.8 -7.7 0.000 PPI extreme 20.8 15.9 -4.9 0.001 24.7 18.6 -6.1 0.002 22.4 17.0 -5.4 0.000 Source: Evaluation team's calculations based on End-line Survey (panel data) – PROGRESA Caribe. Early Treatment Late-Treatment Project t-test for socio-demographic indicators