Photo Credit: Flowering sorghum field at Boucan Carré. Source: SMASH Program Year 6 Quarter 2 report, December 2018 PERFORMANCE EVALUATION OF SMALLHOLDER ALLIANCE FOR SORGHUM IN HAITI (SMASH) ACTIVITY FINAL EVALUATION REPORT May 2020 DISCLAIMER- The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i | SMASH FINAL EVALAUATION REPORT USAID.GOV PERFORMANCE EVALUATION OF SMALLHOLDER ALLIANCE FOR SORGHUM IN HAITI (SMASH) ACTIVITY EVALUATION REPORT May 2020 Evaluation Mechanism Number: AID-521-C-17-00002 Haiti Evaluation and Survey Services for USAID/Haiti USAID.GOV SMASH FINAL EVALUATION REPORT | ii ABSTRACT Social Impact’s Haiti Evaluation and Survey Services project conducted a final evaluation of the Smallholder Alliance for Sorghum (SMASH) Activity, a public-private partnership between Brasserie Nationale d’Haïti S.A. (BRANA), the Inter-American Development Bank, and United States Agency for International Development, administered by Papyrus S.A. SMASH’s goal was to improve livelihoods for up to 18,000 Haitian subsistence farmers by increasing sorghum yields to replace imported grain in BRANA’s Malta H beverage and strengthen the value chain. The evaluation report assesses whether SMASH achieved intended results and provides guidance for future programs. SMASH interventions to increase production were impeded by persistent effects of the 2014 aphid outbreak, natural disaster, and recurrent drought. SMASH training and extension services did not result in large-scale improvements of farmer sorghum production practices and yields. However, SMASH’s provision of aphid-resistant seed and pre-production financing positively contributed to sorghum production recovery in targeted communities. High logistics costs and low production quality led SMASH to consolidate its smallholder supplier numbers to a few hundred. Most smallholders continued to market their sorghum in local markets where prices rose due to high demand and low supply. SMASH opened a new, formal sector market for bulk sorghum, strengthened existing work on improved seed varieties, and created and sustained a central grain processing center. SMASH’s contributions to improving farmer income varied seasonally and were limited overall. SMASH had not developed or tested a clear sustainability strategy for many of its services at the time of the evaluation. iii | SMASH FINAL EVALAUATION REPORT USAID.GOV TABLE OF CONTENTS ABSTRACT II TABLE OF CONTENTS III LIST OF TABLES IV LIST OF FIGURES IV ABBREVIATIONS AND ACRONYMS V EXECUTIVE SUMMARY VII Evaluation Purpose and Evaluation Questions vii Activity Background vii Evaluation Design, Methods, and Limitations vii Findings and Conclusions viii Recommendations for USAID xi Recommendations for Alliance Management Committee Institutions (BRANA, IDB PAPYRUS, USAID/HAITI) xii Recommendations for BRANA xii INTRODUCTION 1 Background 1 Theory of Change 2 Activity Objectives and Interventions 3 Implementation Stages 4 Evaluation Purpose and Audience 5 Evaluation Questions 5 EVALUATION DESIGN 5 Sampling 5 Data Collection Methods 6 Analysis 9 Limitations and Mitigation Strategies 9 FINDINGS AND CONCLUSIONS 11 EQ1: To what extent and in what ways have SMASH’s interventions been effective in achieving key results, especially increasing farmers’ sorghum production and sales? 11 EQ2: To what extent did the project improve market access for small-scale farmers and in what ways has the SMASH program and other external factors affected the sorghum grain market in Haiti? 23 EQ3: To what extent and in what ways has the SMASH program been successful in strengthening the sorghum value chain in Haiti? 32 RECOMMENDATIONS 38 Recommendations for USAID 38 Recommendations for Alliance Management Committee Institutions (BRANA, IDB, PAPYRUS, USAID/HAITI) 38 Recommendations for BRANA 39 USAID.GOV SMASH FINAL EVALUATION REPORT | iv ANNEX A: EVALUATION STATEMENT OF WORK 40 ANNEX B. EVALUATION PROTOCOLS 48 ANNEX C: REFERENCES AND ENDNOTES 66 ANNEX D: LIST OF KEY INFORMANTS 68 LIST OF TABLES Table 1: SMASH Projected Outcomes, Key Results and Activities....................................................................... 4 Table 2: SMASH Intervention Zone Sample Selection Criteria.............................................................................. 6 Table 3: KIIs By Stakeholder Group And Gender..................................................................................................... 7 Table 4: FGDs By Location And Gender..................................................................................................................... 8 Table 5: Mini-Survey Respondents by Gender and Location .................................................................................. 8 LIST OF FIGURES Figure 1: Evolution of sorghum production in Haiti 2009-2016 in MT............................................................... 12 Figure 2: Comparison of number of farmers trained and number of farmers selling sorghum to SMASH by Activity year................................................................................................................................................................. 17 Figure 3: MTs of sorghum purchased by BRANA per Activity year................................................................... 20 Figure 4: Number of farmers who sold sorghum to SMASH/BRANA by Activity Year................................ 27 Figure 5: SMASH Cumulative and Absolute Volume of Purchases by Year...................................................... 29 Figure 6: Price of sorghum in three local markets 2013-2018.............................................................................. 30 v | SMASH FINAL EVALAUATION REPORT USAID.GOV ABBREVIATIONS AND ACRONYMS AMC ATL Alliance Management Committee Assistant Team Leader BRANA CASELI CHIBAS Brasserie Nationale d’Haïti S.A. Center for Support and Services for Local and International Businesses Centre for Research on Bioenergy and Sustainable Agriculture COR Contracting Officer’s Representative DEC Development Experience Clearinghouse EDN Etoile du Nord S.A. EGAD Economic Growth and Agriculture Development ESS Evaluation and Survey Services ET Evaluation Team EQ Evaluation Question FGD Focus Group Discussion GOH HA Government of Haiti hectare IDB Inter-American Development Bank IP KG Implementing Partner Kilograms KII Key Informant Interview LEVE Local Enterprise and Value Chain Enhancement MARNDR Ministry of Agriculture, Natural Resources and Rural Development MCC Micro Collection Center M&E Monitoring and Evaluation MOU Memorandum of Understanding MT Metric Tons NGO Non-Governmental Organization PPP Public-Private Partnership SI Social Impact, Inc. SMASH Smallholder Alliance for Sorghum in Haiti SOW Scope of Work TL Team Leader USAID United States Agency for International Development USD United States Dollars vii | SMASH FINAL EVALUATION REPORT USAID.GOV EXECUTIVE SUMMARY The United States Agency for International Development (USAID)/Haiti requested that Social Impact Inc.’s Evaluation and Survey Services design and conduct an independent final evaluation of the Smallholder Alliance for Sorghum in Haiti (SMASH), a public-private partnership (PPP) between Brasserie Nationale d’Haïti S.A. (BRANA), the Inter-American Development Bank (IDB), and USAID, and administered by Papyrus S.A., a Haitian management firm. This report is the evaluation team’s (ET) findings, conclusions, and recommendations. EVALUATION PURPOSE AND EVALUATION QUESTIONS The evaluation’s purpose is to guide the development of similar programming in support of Haitian agricultural value chains based on an assessment of the extent to which SMASH achieved intended results. The primary audiences for this evaluation are BRANA, IDB, USAID/Haiti, and Papyrus S.A. This final evaluation will answer the evaluation questions (EQs) below. 1. To what extent and in what ways have SMASH’s interventions been effective in achieving key results, especially increasing farmers’ sorghum production and sales? The evaluation should highlight gaps and lessons learned. 2. To what extent did the project improve market access for small-scale farmers and in what ways has the SMASH program and other external factors affected the sorghum grain market in Haiti? The evaluation should consider supply, demand and price in its market analysis. 3. To what extent and in what ways has the SMASH program been successful in strengthening the sorghum value chain in Haiti? The evaluation should include farmers’ income and sustainability in its analysis of the sorghum supply chain strengthening. ACTIVITY BACKGROUND SMASH’s goal was to improve the livelihoods of up to 18,000 Haitian subsistence farmers through a market-driven approach to increasing yields of high-quality, locally grown sorghum to replace imported malted barley in Malta H production, a BRANA-Heineken produced non-alcoholic beverage. BRANA started the alliance in 2013. In 2014, USAID/Haiti joined the alliance with a $1.7 million contribution, and in 2015, IDB’s Multilateral Investment Fund (MIF) joined with an investment of $2.4 million. Over six years, SMASH objectives were to: • Increase smallholder farmers’ sorghum production by at least 100 percent by the activity’s end; • Increase targeted smallholder farmer income from sorghum by 75 percent; • Organize the supply chain to enable smallholder farmers to supply commercial buyers such as BRANA with sorghum that meets the buyers’ internal quality standards. EVALUATION DESIGN, METHODS, AND LIMITATIONS The evaluation used a mixed-method, primarily qualitative evaluation design, consisting of document review, short farmer surveys, key informant interviews (KIIs), focus group discussions (FGDs), and direct observation. The ET conducted KIIs and FGDs in Port-au-Prince and in three out of six agricultural regions where SMASH engaged in activities (West, Central Plateau, and South). Key stakeholders included: PPP representatives, SMASH managers and field agents, sorghum farmers, Government of Haiti representatives, other value chain actors, and other donor-funded activities. USAID.GOV SMASH PERFORMANCE EVALUATION | viii FINDINGS AND CONCLUSIONS EQ1: TO WHAT EXTENT AND IN WHAT WAYS HAVE SMASH’S INTERVENTIONS BEEN EFFECTIVE IN ACHIEVING KEY RESULTS, ESPECIALLY INCREASING FARMERS’ SORGHUM PRODUCTION AND SALES? SMASH’S EFFECTIVENESS IN INCREASING FARMERS’ SORGHUM PRODUCTION WAS LIMITED, MOSTLY BY EXTERNAL FACTORS. Because of lack of quality baseline data, the ET cannot strongly state that SMASH interventions have been effective in increasing farmers’ sorghum production and sales. Short, non-representative survey results show that average sorghum yield of a sample of SMASH-affiliated farmers was more than 50 percent lower than the estimated yield before SMASH. However, survey results and farmer accounts suggest that SMASH services (training, seed, pre￾production financing) helped some farmers in selected areas increase production volume. The production level among farmers surveyed increased by an average 67 percent and farmers produced more in 2018 than in 2013, particularly farmers SMASH assisted to cultivate sorghum in humid plains in the south. Although SMASH trained 22,000 farmers, only a small proportion of these farmers received a full assistance package (approximately 500 farmers received aphid-resistant seeds; 118 received pre￾financing). The overall effect on sorghum production was likely limited given farmers’ difficulty in accessing quality inputs. Contextual factors affecting implementation include an aphid outbreak, which resulted in consecutive bad harvests and low yields starting in 2014, and Hurricane Matthew in 2016, which impeded SMASH interventions that sought to increase production. SMASH’S PROVISION OF APHID-RESISTANT SEED POSITIVELY CONTRIBUTED TO SORGHUM PRODUCTION RECOVERY IN INTERVENTION AREAS. Based on farmer and other value chain expert accounts, SMASH enabled farmers in intervention zones to return to sorghum production by distributing sorghum seed variety Papèpichon. Many farmers indicated that SMASH facilitated access to the seed, which encouraged them to plant sorghum despite the continued threat from aphids. For a more complete recovery, value chain experts highlighted the need to stabilize and improve aphid and drought tolerance of the existing seed variety and fill gaps in SMASH aphid-resistant seed distribution, which did not cover all former sorghum growing regions. SMASH TRAINING AND EXTENSION SERVICES DID NOT SUCCESSFULLY FOSTER LARGE￾SCALE IMPROVEMENTS IN FARMER SORGHUM PRODUCTION PRACTICES AND YIELDS. Although SMASH trained 22,000 farmers and established 84 demonstration plots, nearly meeting its targets, substantially fewer farmers than expected changed their production practices, largely because of the cost of applying new practices, the continued risk of poor harvests posed by aphids, and the extent to which farmers were willing to innovate and take risks. The ET could not determine the effectiveness of SMASH training and extension on increasing sorghum yields because of conflicting yield data as well as the negative effect of external factors, mainly the aphid infestation. SMASH PRE-PRODUCTION FINANCING EFFECTIVELY PROMOTED SORGHUM PRODUCTION. SMASH determined that because of continued low yields, most sorghum farmers were not financially able to manage credit. The Activity’s alternative, pre-production “in-kind service” financing (mainly access to mechanization for field preparation) was generally effective in motivating smallholder farmers to organize into production blocks (a requirement for pre-financing) and plant more sorghum. The practice was more effective in promoting sales to BRANA than other strategies because receiving services increased farmers’ sense of obligation to BRANA. Other “softer” strategies to strengthen trust between farmers and BRANA, such as BRANA’s post-hurricane farmer grants and supplier loyalty programs, were also effective in generating sales to BRANA. SMASH DID NOT SUCCESSFULLY ESTABLISH A LARGE SUPPLIER BASE FOR BRANA. Fewer farmers than expected became SMASH suppliers. Reasons include the Activity’s underestimation of market competition, overestimation of farmers’ valuation of supplier contracts, and BRANA’s evolving ix | SMASH FINAL EVALUATION REPORT USAID.GOV perception of cost-effectiveness and efficiency of having a large number of small producers in its supply chain. Because of these barriers, SMASH scaled back training and provided additional production services and follow-up support to smallholder farmers who demonstrated their interest in changing their farming practices and selling sorghum to SMASH. SMASH also integrated a greater number of larger scale farmers into the supplier network, which significantly reduced the number of farmers who regularly received SMASH services over time. SMASH AND OTHER SORGHUM GRAIN MARKET ACTORS CONTRIBUTED TO FARMERS SELLING A HIGHER PERCENTAGE OF THEIR HARVESTS. Despite the low levels of production and yields observed in the sorghum value chain, SMASH effectively stimulated some farmers to increase their sorghum sales. Short survey results show that 68 percent of SMASH-affiliated farmers sold a higher percentage of their harvest while only 40 percent of farmers did so before the SMASH intervention. The rising price of sorghum in local, informal markets as well as SMASH production incentives tied to sales led farmers to sell most of their harvest instead of saving a proportion for home consumption. EQ2: TO WHAT EXTENT DID THE ACTIVITY IMPROVE MARKET ACCESS FOR SMALL-SCALE FARMERS AND IN WHAT WAYS HAS THE SMASH PROGRAM AND OTHER EXTERNAL FACTORS AFFECTED THE SORGHUM GRAIN MARKET IN HAITI? SMASH CONTRIBUTED SUBSTANTIALLY TO THE CREATION OF A NEW, FORMAL MARKET FOR HIGH-QUALITY SORGHUM IN HAITI. BRANA’s local sourcing policy created a new market for high-quality bulk sorghum founded on a corporate commitment to integrating small-scale farmers into the supply chain to the extent that it would be a win/win arrangement. Before SMASH, smallholder farmers could not access formal agricultural markets. Generally, no formal market existed for bulk locally grown sorghum. When SMASH and USAID/Local Enterprise and Value Chain Enhancement supported the creation of Etoile de Nord (EDN), the central conditioning center, to meet BRANA’s requirements, it indirectly contributed to opening other new markets for locally grown, bulk sorghum such as bakeries and school feeding programs. SMASH DID NOT CHANGE HOW MOST SMALLHOLDER SORGHUM FARMERS MARKETED THEIR SORGHUM. Before and during SMASH, many smallholder farmers sold their sorghum through informal markets facilitated by traders, who were mainly women known as Madam Sara. The informal marketing system paid farmers in cash, on occasion offered cash advances, and handled sorghum consolidation and transportation to markets without requiring exceptional quality standards. Although this system was inefficient, it was largely acceptable to farmers based on their accounts. DESPITE SMASH’S EFFORTS TO FACILITATE AN INCLUSIVE SUPPLY CHAIN, HIGH LOGISTICS COSTS AND LOW PRODUCTION QUALITY LED SMASH TO INSTITUTE ELIGIBILITY REQUIREMENTS THAT LIMITED ACCESS BY SMALLHOLDER FARMERS. SMASH contributed to facilitating small-scale farmers’ access to BRANA’s market by creating storage facilities and managing grain transportation, and by piloting supply chain management and payment systems. However, these services failed to ensure that smallholder production met BRANA’s high quality standards, mainly due to the cost and complicated logistics of moving small sorghum quantities from the field to the processing center quickly. This factor and the overall high cost of working with many highly dispersed farmers triggered a change in SMASH’s local sourcing strategy that required smallholder farmers to form production blocks or to associate with proximate larger-scale producers. This decision, while not excluding smallholders, limited access to fewer smallholders. The evaluation found that within sorghum farming communities, SMASH interventions were more effective if they included males over females and older over younger farmers in BRANA’s supply chain. While the disproportionate involvement of older men was largely a reflection of Haitian sorghum farmer demographics, the Activity had no strategy to involve women traders in its supply chain nor did it have any targeted strategies to involve younger farmers. USAID.GOV SMASH PERFORMANCE EVALUATION | x BRANA’S AND EDN’S ENTRY INTO THE MARKET INCREASED NATIONAL DEMAND FOR BULK SORGHUM AND INCREASED MARKET PRICE, WITH POTENTIAL BENEFITS FOR SORGHUM PRODUCERS. Before the aphid infestation, SMASH sorghum purchases for BRANA may have affected the national market to a limited extent because of the small volume. Since supply declined, SMASH affected the market to a larger extent, particularly in 2018 when it purchased over 500 metric tons of sorghum. However, because of high competition for sorghum and SMASH’s own pricing policy, farmers received higher prices from Madam Sara than from SMASH. Most smallholder sorghum farmers gained more by selling in local, informal markets. Farmers who sold to SMASH likely did so because they wanted the Alliance’s services and felt an obligation to repay these in sales. EQ3 TO WHAT EXTENT AND IN WHAT WAYS HAS THE SMASH PROGRAM BEEN SUCCESSFUL IN STRENGTHENING THE SORGHUM VALUE CHAIN IN HAITI? BRANA’S DEMAND FOR HIGH QUALITY BULK SORGHUM DIRECTLY AND INDIRECTLY CONTRIBUTED TO POSITIVE CHANGES IN THE VALUE CHAIN. SMASH contributed to opening a new, formal sector market for bulk sorghum, strengthened existing work on improved seed varieties, engaged non-governmental organizations in improved seed multiplication, modestly strengthened local agricultural service providers, and created and sustained a grain central processing center. BRANA’S LONG TERM COMMITMENT TO SERVING THE HAITIAN MARKET AND SOURCING SORGHUM LOCALLY IS A POSITIVE FACTOR FAVORING SUSTAINABILITY. Although greatly reduced in scale (200-350 small-scale farmers), BRANA indicated it still intends to include small-scale farmers in its supply chain. Similarly, EDN, and BRANA’s part-ownership of it, promotes sustainability. As private sector entities with a vested interest in promoting dynamic value chains for locally grown sorghum, these businesses have shared interests with sorghum farmers: finding improved seed varieties to raise yields, increasing numbers of farms using improved farming techniques, and improving post￾harvest logistics. Because of these shared interests, continuation of BRANA’s local sourcing policy would likely be beneficial for sorghum farmers, including both those who sell to BRANA and others who may benefit indirectly from the development of improved seed varieties and additional buyers in the market. SMASH’S CONTRIBUTIONS TO IMPROVING FARMER INCOME FROM SORGHUM VARIED FROM SEASON TO SEASON AND WERE LIMITED OVERALL. A short survey (n=76) showed that 43 percent of surveyed farmers reported positive income from sorghum production. The average gain reported was higher than the average loss. However, income from sorghum made a relatively small contribution (8 percent) to farmers overall income. Among factors that affected sorghum growing’s contribution to farmers’ income were high production costs and low productivity. Lack of reliable baseline data prevents the ET from assessing the extent to which farmers’ income from sorghum changed since SMASH started. However, farmers in FGDs indicated that they have experienced an increase in income from sorghum production at least once since their involvement with SMASH. Income from sorghum production varies annually, which makes its contribution to farmers’ income unstable. SMASH HAD NOT DEVELOPED OR TESTED A CLEAR SUSTAINABILITY STRATEGY FOR MANY OF ITS SERVICES AT THE TIME OF THE EVALUATION. SMASH’s over-reliance on a project-based service delivery model (services provided directly by individuals employed by Papyrus and paid for by subsidies) instead of more sustainable, market-driven implementation modalities was the main factor limiting the sustainability of services. The table below summarizes factors that enabled and limited SMASH performance according to each evaluation question. xi | SMASH FINAL EVALUATION REPORT USAID.GOV Evaluation questions (EQ) Key enabling factors Key limiting factors EQ1. To what extent and in what ways have SMASH’s interventions been effective in achieving key results, especially increasing farmers’ sorghum production and sales? The evaluation should highlight gaps and lessons learned Promotion of high yield sorghum varieties; Promotion of aphid tolerant variety; Training and extension services; Pre-financing of production; Distribution of inputs Adverse weather conditions; Aphid infestation; Lack of appropriate pesticide use; Variability of the varieties promoted EQ2. To what extent and in what ways has the SMASH program and other external factors affected the sorghum grain market in Haiti? The evaluation should consider supply, demand and price in its market analysis Establishment of collect centers; Provision of transport of the grain from the field to BRANA; Training and support services to improve post￾harvest practices; Creation of production blocks; Use of mobile payment High costs of providing services to small-scale farmers; Reluctance of farmers to organize themselves for block production; Low grain quality; Competitiveness of market actors EQ3. To what extent and in what ways has the SMASH program been successful in strengthening the sorghum value chain in Haiti? The evaluation should include farmers’ income and sustainability in its analysis of the sorghum supply chain strengthening Collaboration with national research entities to test new varieties; Contribution to seed multiplication; Training of extension agents; Collaboration with EDN Limited income from sorghum; Lack of sustainability strategy RECOMMENDATIONS FOR USAID 1. FUND A SET OF INTEGRATED INTERVENTIONS AIMED TO BOOST SORGHUM PRODUCTION ACROSS HAITI. • Subsidize multiplication and distribution of improved seed varieties in all sorghum growing regions (once the seed varieties are field-tested and approved). • Sponsor farmer-targeted communication campaigns on effective measures to fight aphids, with improved seed distribution activities and information on crop rotation and safe insecticide use, including developing integrated pest management. • Focus on areas where sorghum was more prevalent before the aphid infestation. Conduct research to breed a seed variety that is aphid- and drought- resistant. 2. SUPPORT DEVELOPMENT OF OTHER MARKETS FOR BULK SORGHUM. • Support market development for bulk sorghum with lower-quality standards than BRANA to lower smallholder barriers to entry (school feeding programs, livestock production). USAID.GOV SMASH PERFORMANCE EVALUATION | xii RECOMMENDATIONS FOR ALLIANCE MANAGEMENT COMMITTEE INSTITUTIONS (BRANA, IDB PAPYRUS, USAID/HAITI) 3. CONTINUE ACTIVITIES TO INCENTIVIZE SMALL-SCALE FARMERS TO ORGANIZE IN BLOCKS AND IMPROVE THEIR FARMING PRACTICES, USING ACCESS TO BRANA’S MARKET AS AN INCENTIVE. • Intensify investment in the development and testing of aphid- and drought-tolerant sorghum seed varieties not only with Quisqueya University, which has accumulated substantial experience in this area over the last six to seven years, but also with other public and private institutions in Haiti with seed research capacity. • Adapt SMASH mobile to EDN needs and support its deployment. Consider feasibility of automating extension messages (i.e., time to weed) by interactive voice recording. • Support a small assessment on farmer attitudes and constraints to establishing production blocks to inform future strategies to manage change. • Support creation of decentralized drying and storage facilities closer to farms to improve quality of sorghum supplied by smallholders. • Develop targeted strategies to increase participation of youth in farming. • Pilot an initiative to integrate Madame Sara in BRANA’s supply chain. • Capture sound baseline data at the beginning of Activities and implement a strong monitoring and evaluation system to develop adequate analytical tools and better inform decision-making. For the implementation of these strategies, strengthen partnerships with universities, farmers’ associations, and grass roots organizations involved in rural and agricultural development programs. RECOMMENDATIONS FOR BRANA 4. TAKE STEPS TO PROMOTE SUSTAINABILITY IN LOCAL SOURCING STRATEGY. • Review price setting strategy to align more realistically with current yields and market conditions based on regular and comprehensive assessments of production costs. • Transition purchasing agent role from SMASH to EDN or other market-oriented intermediaries. During transition, document and share best practices, such as use of farmer liaisons as local support to purchasing agents. • Transition pre-production financing to local operators, farmers’ associations, and/or grassroots organizations and reinforce their capacity to maintain equipment and manage services. • Transition extension services to universities, vocational training centers, government extension services, and grassroots organizations involved in rural and agricultural development programs. 1 | SMASH FINAL EVALUATION REPORT USAID.GOV INTRODUCTION The United States Agency for International Development (USAID)/Haiti requested that Social Impact, Inc.’s (SI) Evaluation and Survey Services (ESS) design and conduct an independent final evaluation of the Smallholder Alliance for Sorghum in Haiti (SMASH) Activity, a public-private partnership (PPP) between Brasserie Nationale d’Haïti, S.A. (BRANA), a Heineken-owned brewery, the Inter-American Development Bank (IDB), and USAID/Haiti. Papyrus, a Haitian management firm implements SMASH and administers the Alliance Management Committee (AMC) comprising BRANA, IDB, USAID, and Papyrus. The purpose of this report is to present the Evaluation Team’s findings, conclusions, and recommendations. BACKGROUND Heineken became the majority owner of BRANA, Haiti’s first and only brewery, in January 2012. In line with Heineken’s “Brewing a Better Future” sustainability strategy, BRANA committed to source at least 20 percent of all raw materials for its brewed products locally. BRANA launched SMASH in July 2013, with the objective of replacing imported malt with locally grown sorghum in the production of its popular Malta H, a non-alcoholic beverage consumed in Haiti. The goal of SMASH is to improve the livelihoods of up to 18,000 Haitian smallholder farmers through a market-driven, value chain approach to sustainably increase yields of high-quality, locally grown sorghum. Through SMASH, the brewery committed to purchase at least 7,600 metric tons (MT) of Haitian sorghum over a six-year period. In 2014, USAID/Haiti joined SMASH, forming a PPP with BRANA. IDB joined the partnership in 2015. AGRICULTURAL PRODUCTION IN HAITI The Haitian economy is largely informal and heavily dependent on the agricultural sector dominated by small family farms. Growing rural population pressures; recurrent natural disasters; extreme weather events (e.g., droughts); and farmers’ limited access to information, modern technology, and up-to-date good cultivation practices have all constrained agricultural output and limited farmers’ income. Roughly 2.5 million Haitians live in extreme poverty (below $1.25 per day), predominantly in rural areas. 1 Haiti’s food supply is ensured through domestic production and imports from the global market (including cross-border imports from neighboring Dominican Republic). Haiti imports one billion United States Dollars (USD) worth of agri-food products, while local agriculture supplies approximately 55 percent of locally consumed food. 2 USAID/HAITI’S ECONOMIC GROWTH AND AGRICULTURE STRATEGIES SMASH is part of USAID/Haiti’s Economic Growth and Agriculture Development (EGAD) office portfolio. In the agricultural sector, USAID’s strategy focuses on sustainably increasing farmers’ incomes through measures that increase productivity, conserve the environment, and strengthen farmer access to domestic and international agricultural markets. USAID/Haiti programs also help farmers to process staple crops like corn, rice, beans, and plantains, as well as cash crops such as cacao and mango.3 USAID’s EGAD office is investing in creating more productive value-chains for medium and small enterprises in Haiti through a variety of strategies including workforce training, access to financing, and 1 USAID/Haiti (2017). Haiti Economic Growth & Agricultural Development Fact Sheet. https://www.usaid.gov/sites/default/files/documents/1862/FINAL_EGAD_Fact_Sheet_March_2017.pdf. 2 Chemonics International Inc (2018). HAITI Staple Food Market Fundamentals, Famine Early Warning Systems Network (FEWS). https://reliefweb.int/sites/reliefweb.int/files/resources/Haiti%20MFR_final_20180326%20%281%29.pdf. 3 USAID/Haiti (2017). Haiti Economic Growth & Agricultural Development Fact Sheet. USAID.GOV SMASH PERFORMANCE EVALUATION | 2 other business development support. For example, the Local Enterprise and Value Chain Enhancement (LEVE) Activity provides workers’ training and business development services to small- and medium￾sized enterprises, working primarily within value chains in three sectors – apparel, agribusiness, and construction – to increase outputs, incomes, and employment opportunities. USAID also leverages partnerships with the private sector entities, among them BRANA, to improve social and economic conditions as well as to deepen the Mission’s development impact.4 THEORY OF CHANGE SMASH did not explicitly articulate a theory of change; therefore, what follows is constructed based on Activity work plans and annual reports. The overarching goal of SMASH is to establish an inclusive, sustainable sorghum value chain in Haiti. Inclusive means that smallholder farmers, which SMASH defined as farmers working ten hectares (ha) or less, participate by selling their crops profitably. Sustainable means the value chain functions efficiently in the medium- to long-term without the need for subsidies. SMASH postulated that small farmers could earn a better living by growing and selling their sorghum to BRANA. By purchasing locally, BRANA would, likewise, win by saving on foreign currency expenditures and lowering its raw material costs. Once initial constraints within the value chain were overcome with corporate and donor support, the benefits to both the farmers and the brewery would be sustainable through normal market functions, without a need for subsidies. Ultimately, BRANA management intended that its investments would result in a sustainable sorghum supply at a price and quality level that would result in savings over the cost of comparable imports.5 Delving deeper into the Activity’s theory of change, SMASH postulated that smallholder sorghum farmers would improve their livelihoods if they could increase their sorghum yields per ha cultivated. To increase yields, SMASH believed that farmers needed to use better sorghum cultivation practices. The Alliance proposed to contribute by: 1. Training farmers on agronomic best practices for producing sorghum; and 2. Facilitating farmer access to improved seed varieties as well as other inputs such as pesticides and machinery. To increase farmer income per ha cultivated, SMASH reasoned that sorghum farmers also needed to lower production costs, reduce post-harvest losses, and/or market their crops more efficiently. To reach these objectives, SMASH proposed to: 1. Facilitate smallholder farmer access to credit and other financial services (such as pre￾production financing); and 2. Make it easier and less costly for smallholder farmers to sell their sorghum crop on the local market by committing to buy from smallholder producers, aggregating smaller producers into producer blocks, eliminating middlemen, and facilitating getting their product to market. SMASH also planned activities to make the value chain more efficient from field to processor. Because the overall approach was market-driven, SMASH hypothesized that to produce commercial quality sorghum sustainably, the whole value chain needed to be better organized. To achieve this objective, SMASH proposed to: 1. Strengthen logistics and transport at various points in the value chain; 4 Ibid. 5 SMASH Year 4 Annual Report. p. 2. 3 | SMASH FINAL EVALUATION REPORT USAID.GOV 2. Improve cleaning and processing facilities; 3. Establish and monitor quality standards; and 4. Identify and use good practices for determining fair market prices, contracting, and paying producers. KEY ASSUMPTIONS Even though locally produced sorghum was not sold in the bulk quantity and quality needed by the brewery when SMASH launched, BRANA believed that the potential for increasing sorghum production was high, since the crop was already grown in Haiti and was relatively drought tolerant. 6 SMASH likewise assumed that weather patterns in Haiti and conditions for growing sorghum would not be drastically altered. SMASH also assumed that political and economic stability in Haiti, which was improving at the time the activity launched, would not significantly degrade, would allow BRANA and other sorghum processers’ operations to continue without major disruptions. ACTIVITY OBJECTIVES AND INTERVENTIONS Over six years, SMASH objectives were to: • Increase smallholder farmers’ sorghum production by at least 100 percent by the Activity’s end, • Increase targeted smallholder farmer income from sorghum by 75 percent, and • Organize the supply chain to enable smallholder farmers to supply commercial buyers, such as BRANA, with sorghum that meets the buyers’ internal quality standards. SMASH planned to implement a variety of strategies to achieve these objectives. The table below is adapted from the project results framework 7 and work plans, and highlights SMASH’s main outcomes, planned outputs or intermediate results, as well as the main activities the Alliance implemented. 6 Eckert, Elizabeth and Latané, Anna, (2017). The Sorghum Value Chain in Haiti: A Mapping and Analysis, Research Triangle Institute (RTI). p. 4. https://www.rti.org/publication/sorghum-value-chain-haiti. 7 Annex 1 SMASH year 5 annual report, “Annex 1 -Performance Monitoring Matrix (Year 5: July 2017 –June 2018)”. USAID.GOV SMASH PERFORMANCE EVALUATION | 4 TABLE 1: SMASH PROJECTED OUTCOMES, KEY RESULTS AND ACTIVITIES OUTCOMES KEY RESULTS MAIN INTERVENTIONS • 10,000 farmers adopt improved technologies or practices • Participating smallholder farmers’ sorghum production/square ha increases by 100 percent by the end of the project • Targeted smallholder farmer income 8 from sorghum increases by 75 percent by the end of the project • 10,000 farmers provide sorghum to BRANA via the SMASH program • 29,300 farmers trained on improved sorghum production techniques • 114 demonstration plots established • 20 extension agents trained on “climate smart” sorghum production • 350 agricultural extension agents trained on improved sorghum production techniques • Improved seed varieties tested, replicated, and available for distribution to farmers • 10,000 farmers trained on good post-harvest practices • 620 farmers trained on good business and financial management practices • 20 collection points established • 320 farmers qualify for credit or other financial products • Mobile purchasing/traceability platform established • Central conditioning center established • 7,600 MT of sorghum purchased by BRANA over six years • Recruit smallholder farmers to participate in SMASH • Organize producer seminars for farmers who sign up with SMASH • Train agronomists who work with other sorghum producers in target regions • Establish demonstration plots • Import, test, store, reproduce, and distribute improved seed varieties • Organize smallholder farmers into production blocks • Establish collection centers, train operators, and coordinate transportation • Deliver post-harvest good practice training to farmers • Deliver business and financial management training to farmers • Develop and implement producer financing mechanism • Develop and implement mobile supply chain management tool • Provide short-term assistance to the central conditioning center, Etoile du Nord (EDN) • Prepare and implement transition strategy to move SMASH key activities to BRANA local sourcing department IMPLEMENTATION STAGES BRANA initiated SMASH in 2013 in one region. USAID/Haiti and IDB joined the Alliance in 2014 and 2015, respectively. BRANA committed $3.4 million to SMASH, with additional contributions of $1.7 million from USAID and $2.4 million from IDB’s Multilateral Investment Fund (MIF), for a total budget of $7.5 million. BRANA’s Memorandum of Understanding (MOU) with USAID/Haiti extended the Activity’s reach, allowing BRANA to source from three regions of the country. When IDB joined the alliance, SMASH’s reach was extended to four departments, namely the West, South, Artibonite, and the Central Plateau. Nippes later become the fifth after efforts to extend SMASH to the Northeast region proved unsuccessful. Over the Activity’s life, SMASH developed two work plans. The first covered the period from January 2015 to June 2016 and focused primarily on expanding the activity into new regions, training farmers and 8 Targeted farmers are farmers that have been reached either partially or fully and changed their practices. Income is defined in the profit that a farmer has from growing sorghum. The indicator is measured on the group of trained farmers that apply one or more trained techniques (plant density, thinning, weeding or replanting). 5 | SMASH FINAL EVALUATION REPORT USAID.GOV non-SMASH agronomists, testing new seed varieties and implementing a sustainable solution for seed reproduction and distribution, developing a technology application to facilitate purchasing and payments to farmers, establishing a network of micro collection centers to buy directly from farmers, and identifying and supporting a national-level sorghum collection and conditioning center. The second work plan, covering the period from July 2017 to June 2019, marked the beginning of SMASH’s transition from a separate, corporate and donor-funded project to a sustainable component of BRANA’s business activities. In its final two years, SMASH shifted its focus from providing extension services to increasing the production and commercialization of sorghum and improving the efficiency of its services to farmers. It also launched SMASH Mobile, a mobile supply chain management information system. EVALUATION PURPOSE AND AUDIENCE The purpose of this evaluation is to guide the development of similar programming in support of Haitian agricultural value chains based on an assessment of the extent to which SMASH achieved intended results/outcomes. The primary audiences for this evaluation are BRANA, IDB, USAID/Haiti, and Papyrus S.A. EVALUATION QUESTIONS This final evaluation will answer the evaluation questions (EQs) below. 1. To what extent and in what ways have SMASH’s interventions been effective in achieving key results, especially increasing farmers’ sorghum production and sales? The evaluation should highlight gaps and lessons learned. 2. To what extent did the project improve market access for small-scale farmers and in what ways has the SMASH program and other external factors affected the sorghum grain market in Haiti? The evaluation should consider supply, demand, and price in its market analysis. 3. To what extent and in what ways has the SMASH program been successful in strengthening the sorghum value chain in Haiti? The evaluation should include farmers’ income and sustainability in its analysis of the strength of the sorghum supply chain. The detailed statement of work for this evaluation is included in Annex A. EVALUATION DESIGN SAMPLING The Evaluation Team (ET) conducted data collection in Port-au-Prince as well as in three out of six agricultural departments where SMASH has been active. The ET selected West, Central Plateau, and South regions, which include locations where SMASH worked for varying durations, and experimented with various approaches in its work with farmers. In these regions, farmers were exposed to the main external challenges affecting Activity progress, and experienced varying degrees of success in meeting Activity indicators. The ET collected data on the perspectives and experiences of key stakeholders involved in SMASH and/or in the sorghum value chain. Key stakeholders consulted included: SMASH donors/members of the AMC, SMASH activity managers at the national and field levels, SMASH-affiliated sorghum producers, extension agents who participated in SMASH training, Government of Haiti (GOH) representatives from the Ministry of Agriculture, Natural Resources, and Rural Development (MARNDR), representatives of other value chain actors (processors, input producers, bulk buyers), and other organizations and donor￾funded activities working on sorghum. USAID.GOV SMASH PERFORMANCE EVALUATION | 6 TABLE 2: SMASH INTERVENTION ZONE SAMPLE SELECTION CRITERIA SELECTION CRITERIA WEST CENTRAL PLATEAU SOUTH Where there are the most farmers selling to BRANA X Where there are the least farmers selling to BRANA X Where SMASH has worked the longest X Where yields have improved the most X Where yields have improved the least X Where smallholder incomes have improved the most X Where SMASH worked with many, widely disbursed farmers X Where examples of work with smallholder producer blocks occurred X X X Where services like pre-production financing, access to production inputs, etc. have been provided X X X Zones affected by insect infestation X X X Zone affected by Hurricane Matthew X The ET selected participants using purposeful sampling techniques. Considering the time limitation of this evaluation, this approach enabled the team to prioritize interviewees with the most information, and ensure a diversity of interviewees where population sizes were larger (such as farmers). DATA COLLECTION METHODS The ET used a mixed method, primarily qualitative evaluation design. The team extracted quantitative data from SMASH’s monitoring and evaluation (M&E) systems. In addition, it gleaned quantitative data through short surveys with a small sample of farmers who were also participating in evaluation focus group discussions (FGDs). However, because of the small sample size, survey results are not necessarily representative of the larger population. The ET compared and complemented project and short survey data with qualitative data collected through various methods including document review, key informant interviews (KIIs), FGDs with participating farmers and field-level agronomists, and direct observation of local markets. The data collection protocols used to guide fieldwork can be found in Annex B. The purpose of the protocols was to: (1) ensure all key issues are covered during data collection, (2) elicit rich, sometimes unanticipated, information from respondents, (3) help organize information in a form that can be usefully and efficiently analyzed, and (4) ensure that sex-disaggregated and other interesting and relevant data are collected, and that information is gathered to assess the extent to which SMASH effectively addressed the specific concerns of both male and female stakeholders. The protocols consist of questions that address and derive from the EQs, as well as from the ET’s document review and its initial discussions with USAID. 7 | SMASH FINAL EVALUATION REPORT USAID.GOV DOCUMENT REVIEW The ET reviewed SMASH documents to understand the Activity design and implementation, extract findings relevant to answer all three EQs, and inform data collection protocol development so that instruments would appropriately supplement the information collected from background documents. Categories of documents are listed below and detailed in Annex C. These include: • Activity MOU and Collaboration Agreement with USAID • SMASH annual reports (years one to five) • SMASH quarterly reports (QI and Q2 for year six) • SMASH annual work plans • SMASH M&E plan • Other relevant statistical and background information produced by the GOH, USAID or USAID activities, and relevant research organizations. KEY INFORMANT INTERVIEWS The ET interviewed 36 individuals (9 females and 27 males) from various stakeholder groups. The team interviewed nearly all informants individually, except for agronomists, some of whom were interviewed in small groups of two to four people. Findings from KIIs contributed to the ET’s responses to all EQs. Table 3 shows the distribution of interviews by key stakeholder group and gender. Each KII was guided by an interview protocol adjusted for different types of respondents, and was conducted in French, Creole or English, depending on the stakeholder. The complete list of evaluation key informants is included in Annex D. TABLE 3: KIIS BY STAKEHOLDER GROUP AND GENDER STAKEHOLDER INTERVIEWS FEMALES MALES Alliance Management Committee Members 5 2 3 Implementing partner (IP) personnel-national level 6 1 5 IP personnel-field level 6 3 3 Other value chain, USAID Activity non-governmental organization (NGO) representatives 7 1 6 Larger-scale sorghum farmers 3 0 3 Agronomists 7 2 5 GOH Representatives 2 0 2 TOTAL 36 9 27 FOCUS GROUP DISCUSSIONS The ET organized 12 FGDs with smallholder sorghum farmers involving 121 participants (39 female, 82 male). The team organized three FGDs in the West, two in Plateau Central, and seven in the South. The ET segmented FGDs with farmers by the following characteristics: (i) farmers who received a SMASH service and sold sorghum to the program at least once, (ii) farmers who received a SMASH service but did not sell sorghum to the program, (iii) farmers who received business training (described as higher- USAID.GOV SMASH PERFORMANCE EVALUATION | 8 performing SMASH suppliers), and (iv) farmers who received pre-production financing. Among farmers who received a SMASH service and sold sorghum to SMASH, two FGDs were female-only. The ET selected focus group participants from lists provided by SMASH. TABLE 4: FGDS BY LOCATION AND GENDER LOCATION TOTAL FEMALES MALES West Department 3 (n=18) n=6 n=12 Central Plateau Department 2 (n=13) n=9 n=4 South Department 7 (n=90) n=24 n=66 TOTAL 12 (N=121) N=39 N=82 MINI SURVEYS ET members responsible for facilitating FGDs administered short, paper-based surveys to gather quantitative data to inform the answers to all three EQs. The team conducted the survey through face￾to-face interaction with individual farmers who participated in group meetings. The ET designed the survey to provide evidence on SMASH’s effect on respondents’ production, sales and revenue from growing sorghum following SMASH support. A copy of the survey is included in Annex B. The ET surveyed 76 farmers (23 females and 53 males). The greatest numbers of respondents were in the South, the region where the largest population of SMASH participants was found. All the participants were smallholder farmers, according the SMASH’s definition (farming less than ten ha). The average farm size was 1.51 ha (maximum size, nine ha, minimum size 0.06 ha); fifty-one percent of respondents farmed less than one ha, thirty-eight percent between one and three ha, and eleven percent more than three ha. TABLE 5: MINI-SURVEY RESPONDENTS BY GENDER AND LOCATION LOCATION TOTAL FEMALES MALES West Department 16 6 10 Central Plateau Department 5 1 4 South Department 55 16 39 TOTAL 76 23 53 DIRECT OBSERVATION ET members visited three agricultural markets in Fonds des Nègres, Kans, and Mirebalais. The te gathered information about market prices, observed the quantities of sorghum available, its sales form and quality, and the local traders and processors (such as artisanal millers) involved in the v chain. The ET also visited a SMASH micro collection center (MCC) in the South and observed its location relative to farms, how it was equipped, and any activity in the center at the time of the v FIELDWORK PROCEDURES The ET included two facilitators and four note takers to conduct the FGDs and short survey. Th held a training session for the facilitators and note takers to brief them on the evaluation objectiv questions. Significant time was spent on explaining the questionnaires for each FGD type and the survey. After the training, facilitators and note takers tested the FGD questionnaire and sort surv a group of farmers in Ganthier, who were not SMASH recipients. The ET revised the Creole ver the questionnaires to address issues identified during the pilot. am price, alue isit. e ET es and short ey with sion of 9 | SMASH FINAL EVALUATION REPORT USAID.GOV ANALYSIS ET members took detailed notes of KIIs and FGDs, and cleaned and shared electronic summaries on a rolling basis throughout fieldwork. Team members conducted internal debriefs regularly throughout fieldwork to discuss progress and make adjustments that were needed in the evaluation schedule. The team also discussed evidence collected to help answer the EQs, as well as to identify discrepancies. At the end of the fieldwork, interviewers for the short survey entered the data collected into a Microsoft Excel spreadsheet. The ET then performed dataset quality control and cleaning to check for data validity, inconsistencies, missing information, and duplicates. The ET employed several data analysis methods to identify key findings from the collected data, as well as to draw conclusions and make recommendations for Activity follow-up or future potential USAID programming like SMASH. 1. CONTENT ANALYSIS – The ET reviewed and coded KII and FGD data to identify and highlight notable examples of external or internal challenges affecting SMASH performance, SMASH achievements, and shortcomings, as well as participant and external expert perceptions of Activity interventions. 2. QUANTITATIVE ANALYSIS – The short survey collected quantitative data from farmers on production costs, sales, ha planted, and harvests to calculate income and yields. The team calculated yield (in kilograms (kg)/ha) as the total production divided by the area used to produce it. The following formula was used to calculate income. Income = [(VS/QS)*TP] – IC where, VS is the value of sales during the period considered; QS is the quantity of sorghum sold during the period; TP is the total sorghum production during the period; and IC is the total cash input expenses. 3. TREND ANALYSIS – The team examined SMASH M&E data on its indicators over time to identify trends in performance relative to its targets. 4. GAP ANALYSIS – The team analyzed where SMASH fell short of anticipated performance targets, and identified the likely factors contributing to these gaps. 5. COMPARATIVE ANALYSIS – The ET compared different stakeholder groups’ perspectives to assess either convergence or divergence. The ET identified findings that arose from the interviews and developed a tally sheet that cross-checked findings by EQ, respondent type, interview type (KII or FGD), and sex. This method enabled the ET to look for trends within and across sub-groups. 6. GENDER ANALYSIS – All data collected through KIIs and FGDs were disaggregated by sex and analyzed for effects on both male and female beneficiaries to show any significant differences. The Team Leader (TL) captured preliminary findings, conclusions, and recommendations in an Excel￾based matrix that categorizes analysis by EQ. The matrix: a) ensures the ET prepares a systematic and thorough response to each EQ; b) verifies that preliminary analysis accounts for gender and social dimensions; c) identifies any gaps where additional clarification or analysis may be necessary; and d) serves as the basis for developing the evaluation report. The ET triangulated data from KIIs, FGDs, and short survey to cross-validate the findings from the different methods used. The ET also triangulate short survey findings with secondary data to answer the evaluation questions. LIMITATIONS AND MITIGATION STRATEGIES USAID.GOV SMASH PERFORMANCE EVALUATION | 10 BIAS The ET was aware of several bias risks in data collection and analysis and developed mitigation strategies to overcome these: 1. RECALL BIAS: This evaluation collected data on farmer sorghum yields and profits, which relied on farmers’ accurate recall of information such as number of plots planted in sorghum, production costs, quantities sold, and price. To mitigate the risk of inaccurate recall, the ET only asked about the most recent planting season. In addition, rather than asking the respondents to estimate yields and profit/loss directly, the survey asked for the detailed inputs for the calculation of yields and profit/loss and performed the calculation using the information provided. 2. RESPONSE BIAS: Response bias is the risk that key informants may have been motivated to provide responses that would be considered socially desirable or influential in obtaining donor support or, alternatively, to discredit the Activity. To mitigate response bias, the ET formulated its findings based on multiple sources of information. Within stakeholder categories, the team interviewed several individuals. It also interviewed relatively disinterested technical experts and reviewed documents other than Activity documents to cross-check facts. The team also administered a clear consent script, which explained the evaluation purpose and highlighted the value of honest answers to help orient future interventions. 3. SELECTION BIAS: Selection bias is an inherent risk when Activity personnel help to facilitate contact with participants. For example, the ET required SMASH assistance to organize FGDs with farmers. To mitigate the risk of selection bias, the ET selected from among many possible communities where the Activity was active and identified the individuals to be interviewed using lists SMASH provided rather than leaving the choice up to SMASH personnel. OTHER LIMITATIONS AND MITIGATION STRATEGIES DATA AVAILABILITY Evaluation questions on changes in sorghum production and sales and the effect of SMASH on Haiti’s sorghum market required economic data on Haiti’s sorghum production and markets that were only partially available. For example, data on sorghum production are available up to 2016. In addition, there are two main sources of agricultural market data, MARNDR’s Agricultural Statistics and the Information Unit of the Food and Agriculture Organization (FAO). While production data are aligned up to 2013, there is substantial variation between the two sources from 2014 onward. The ET considers the national production data from the MARNDR statistical service were closer to reality from 2014 to 2016 than the FAO figures. DATA QUALITY SMASH M&E Data Quality During the evaluation kick-off meeting, USAID alerted the ET to potential limitations in SMASH M&E data, mainly because USAID had not had the opportunity to conduct a data quality assessment. Given the limited time, lack of reliable baseline data for some indicators, and the scope of this evaluation, the ET did not conduct a large-scale, independent survey of SMASH participants to gather additional data on key indicators such as changes in crop yields and livelihood improvement. To mitigate this potential evaluation limitation, the ET reviewed and commented on the methodology or methodologies SMASH used to gather data on key indicators in its findings. In addition, the ET conducted a short, paper-based survey with a small sample of SMASH-affiliated farmers to collect additional quantitative data on selected indicators such as sorghum yields, gross margin, and level of sales. The survey provided independently gathered perspectives on farmer 11 | SMASH FINAL EVALUATION REPORT USAID.GOV performance in growing sorghum with the contribution of SMASH, but this cannot be extrapolated to the larger population that participated in SMASH activities due to a relatively small sample size and non￾robust sampling methodology. Finally, the ET assessed the extent to which evaluation qualitative and quantitative data accorded or diverged from SMASH M&E data and presented the results in key findings, giving a nuanced picture of Activity achievements, as appropriate. Evaluation Data Quality To ensure the quality of evaluation data, the evaluation TL and Sector Specialist conducted one day of comprehensive qualitative and quantitative data collection training of all ET members. The ET organized a pilot data collection exercise in a SMASH community, which was overseen by the Sector Specialist. The pilot exercise was followed by a Sector Specialist-led day-long debrief session with the FGD teams to review notes and surveys and recommend corrective action to improve data quality. LOGISTICAL CHALLENGES Because of the security situation in some communities in the West, the ET changed locations of some planned FGDs and one group was cancelled due to last minute participant no shows. Planned FGDs with agronomists were adjusted to KIIs due to difficulty in finding and mobilizing participants in SMASH training for agronomists from other organizations. FINDINGS AND CONCLUSIONS This section synthesizes the evaluation’s findings and conclusions, which are presented by EQ. EQ1: TO WHAT EXTENT AND IN WHAT WAYS HAVE SMASH’S INTERVENTIONS BEEN EFFECTIVE IN ACHIEVING KEY RESULTS, ESPECIALLY INCREASING FARMERS’ SORGHUM PRODUCTION AND SALES? THE EVALUATION SHOULD HIGHLIGHT GAPS AND LESSONS LEARNED. In answering this question, the ET first analyzed the evolution of sorghum production and sales in Haiti before and during SMASH’s implementation period, highlighting the key external factors that affected SMASH’s success. The section then reviews SMASH’s actions to boost production and sales in the zones where it worked, its main achievements, as well as gaps and lessons learned. FINDINGS CONTEXT OF SORGHUM PRODUCTION IN HAITI Sorghum in Haiti was primarily a subsistence crop grown on drylands. Prior to SMASH, sorghum (Sorghum vulgare) was the third most widely grown staple grain in Haiti after rice and corn. It was grown in all regions in the country with 80 percent of the area planted concentrated in four departments: Centre, Artibonite, South, and West. It was primarily cultivated under rainfed conditions in water deficit areas in mountains and plains. 9 A small part of the area cultivated was in humid plains where there are irrigation schemes. 10 At its height, sorghum production 9 Fews Net (2018). Haiti: Staple food market fundamentals. 10 Bayard, B. (2012). Consultation pour le renforcement de la capacité d’analyse de la Coordination Nationale de la Sécurité Alimentaire (CNSA) dans les processus des achats locaux. USAID.GOV SMASH PERFORMANCE EVALUATION | 12 involved more than 300,000 producers 11 and was grown mainly in mixed plots with maize, pulses, and other crops.12 However, it was grown in monoculture on about 20 percent of the cultivated area. 13 Between 2009 and 2013, sorghum production fluctuated with an average volume of 124,208 MT per year. When SMASH started in 2013, sorghum production was estimated at 111,364 MT for an average yield of 878 kg/ha. 14 Considering an estimated 300,000 sorghum producers, an individual famer produced on average 371 kg of sorghum in 2013. The main constraints to sorghum production in Haiti at the time SMASH started were adverse climatic factors (late rains, drought, etc.), incidence of pests (insects, birds and rodents), use of low-yielding varieties, and most farmers’ limited knowledge and/or use of good crop management techniques. 15 Aphid infestation and unfavorable weather were significant external challenges to increasing farmer sorghum production and sales and discouraged sorghum growing. Starting in 2014, the sugarcane aphid (Melanophis sacchari) infestation wiped out successive sorghum harvests. Sugarcane aphid damaged sorghum by removing sap and covering plants with honeydew, causing general plant decline and yield loss. 16 Figure 1 shows that sorghum production steadily declined between 2014-2016 with a precipitous drop occurring at the start of the aphid infestation in 2014. According to Ministry of Agriculture data, sorghum production dropped from 111,364 MT in 2013 to 44,151 MT in 2014 and continued to decline through 2016. SMASH, MARNDR, and other agriculture experts, as well as farmers affirmed that after the infestation, there were fewer ha planted with sorghum in Haiti and that yields declined from pre-aphid levels. Official data indicate that 58,673 ha were cultivated in sorghum in 2016 compared to 185,000 ha in 2014. MARNDR estimated Sorghum yield at an average of 320 kg/ha in 2016. 17 Figure 1: Evolution of sorghum production in Haiti 2009-2016 in MT According to GOH, other value chain actors, and SMASH employee KIIs and FGDs, at the time of the evaluation, aphids remained a serious threat to sorghum production. When asked what their main obstacles to production were, in nearly all FGDs, farmers highlighted the aphids first. Many farmers indicated that after the aphid and before SMASH intervention in their communities, they were reluctant 11 Van Vliet et al. 2016. 12 Fews Net. Haiti: Staple food market fundamentals. July 2018. 13 MARNDR. Synthèse Nationale des Résultats du Recensement Général de l’Agriculture 2008/2009. Port-au-Prince, Haïti. 2012 14 Using MARNDR data Yield calculated = 111364 MT/126775 ha 15 HAITI Staple Food Market Fundamentals, Famine Early Warning Systems Network (FEWS), March 2018 https://reliefweb.int/sites/reliefweb.int/files/resources/Haiti%20MFR_final_20180326%20%281%29.pdf 16 https://agriengineering.conferenceseries.com/abstract/2017/the-sugarcane-aphid-hemiptera-aphididae-an-invasive-pest-of-sorghum-in-north￾america 17 MARNDR. Résultats des enquêtes nationales de la production agricole année 2016 13 | SMASH FINAL EVALUATION REPORT USAID.GOV to plant sorghum because of past crop failure as well as low yields. Some farmers said they were still reluctant to continue to grow sorghum. One larger-scale farmer in the Plateau Central affirmed that because of the negative effect of the pest on past harvests, “I will plant sorghum one more time, but if yields don’t increase, I won’t continue.” Climatic conditions during SMASH implementation were likewise unfavorable for sorghum production. According to the FAO, Haiti has been affected by episodes of drought and erratic distribution of rainfall almost every year since 2014. In addition, Hurricane Matthew – a Category 4 hurricane that made landfall in the fall of 2016, had catastrophic impacts on agricultural production in Grand’Anse, South, South-East, and Nippes. 18 According to SMASH program managers and farmer FGDs, Hurricane Matthew decimated SMASH-affiliated farmers’ production (and therefore sales) in the South in 2016, which hosted the largest numbers of farmers contracted to sell their production to SMASH. IMPROVED SEEDS: KEY RESULTS, GAPS, AND LESSONS LEARNED Although it exceeded targets for seed testing, SMASH was focused on drought resistant, higher yielding seed varieties rather than pest-resistant ones. Based on SMASH KIIs and document review, the Alliance did not anticipate the unprecedented, significant effect of pests on sorghum production, but had foreseen the importance of introducing improved seed varieties, mainly to boost yields and improve drought resistance. During its first two years, SMASH tested 44 seed varieties and distributed Dekabès (a high yield variety) and Papèsèk (a variety tolerant to drought) to farmers. According project reports and KIIs, initially SMASH worked on improved seed varieties with Centre for Research on Bioenergy and Sustainable Agriculture (CHIBAS), a foundation associated with Quisqueya University in Port-au-Prince. The early collaboration with CHIBAS ended prematurely because CHIBAS did not meet its reporting obligations. Subsequently, SMASH brought in its own experts to conduct seed variety tests. Despite having tested many seed varieties 19, SMASH did not have an aphid-resistant seed to propose to farmers when the devastating effects of the pest hit Haiti, largely because no one inside or outside the Activity could have anticipated the significant aphid infestation. SMASH distributed CHIBAS’ aphid-resistant seed and was credited with proving its effectiveness. Fortunately, CHIBAS identified an aphid-resistant seed line and produced a sorghum seed known as “Papèpichon.” With emergency support from the Canadian government, CHIBAS contracted a local firm (Double Harvest) to multiply and produce 300 MTs of seed. 20 In 2017, both MARNDR and SMASH distributed the new seed variety (SMASH distributed 100 MTs). Key informants involved in the operation highlighted that seed distribution was more successful in SMASH regions because it was done on time. Nearly all sorghum value chain stakeholders, including farmers, agreed that the new seed variety (Papèpichon) performed better than local seeds or other imported seed varieties and that the new seed was critical in the slow recovery of sorghum production. As of the third quarter of year six, based on Activity M&E data, SMASH distributed Papépichon to approximately 500 farms. The seed was provided free of charge. “The aphids had invaded our gardens and we could not harvest anything. After several failed harvests, many farmers abandoned sorghum production and planted maize, pigeon peas, 18 FAO (2019). Haiti Emergency Response Plan, March–May 2019. Rome. https://reliefweb.int/sites/reliefweb.int/files/resources/ca3648en.pdf. 19 As of the third quarter of year six, SMASH had tested 66 seed varieties. It had completed seven of eight planned rounds of testing and replicated 18 seed varieties. 20 CHIBAS was conducting research on sweet sorghum varieties when it inadvertently identified an aphid resistant variety https://lenouvelliste.com/public/index.php/article/187947/voir-grand-pour-le-petit-mil. USAID.GOV SMASH PERFORMANCE EVALUATION | 14 calalou and potatoes instead. Then the SMASH project appeared in the area in 2018 with the new variety (aphid-resistant) which revived sorghum production.” PLATEAU CENTRAL FARMER FGD “BRANA has provided us with other, more resistant seeds. When we used our seeds, aphids destroyed all our crops; although the aphid still exists with BRANA’s seeds, it no longer destroys the crops.” SOUTH FARMER FGD SMASH personnel and agriculture experts from various stakeholder groups cited specific ways Papèpichon should be improved. Several key informants indicated the seed’s effectiveness declined after 3 to 4 plantings, especially in drought conditions, suggesting it was not yet stable and required additional testing. Key informants likewise highlighted the need to produce a shorter variety suitable for mechanized harvesting, which is especially important for larger-scale farms. Informants also indicated that Papèpichon was not as tolerant of drought as aphids, which affected its viability in many areas, particularly in drylands where sorghum was predominantly grown in Haiti. At the time of the evaluation, SMASH was distributing Papèpichon, supporting Quisqueya University to test shorter varieties, and testing new, imported seed varieties. Other gaps: Distribution of aphid-resistant seed did not cover all former sorghum growing regions and lacked a formal seed certification process. Various key informants (SMASH personnel, other value chain actors, MARNDR officials) highlighted that the seed was in high demand because of farmers’ positive perception of its performance relative to other available options. They noted that after the initial distribution, only SMASH was still distributing Papépichon, but only in the zones where it currently supports suppliers, leaving a gap in other areas that had previously grown sorghum. Although it was not within the scope of SMASH to distribute improved seed to all sorghum-growing regions in Haiti, this factor limited overall sorghum production recovery. One stakeholder remarked, “They [SMASH] have a big problem to get the seed they need. They also have a difficulty to project beyond their own needs. If we don’t resolve the problem for the country, the price [of sorghum] is going to remain high.” One MARNDR official indicated that there was a need to strengthen the Ministry’s capacity to control and certify seeds. TRAINING AND EXTENSION SERVICES: KEY RESULTS, GAPS, AND LESSONS LEARNED SMASH delivered improved production and post-harvest practice training and established demonstration plots that reached more than 20,000 farmers. Training and extension services figured significantly in SMASH’s strategy to improve sorghum production and sales, especially during its first implementation phase (2013-2016). SMASH’s core training program featured five modules, with one to three focused on improved sorghum production techniques and four and five on good harvest and post-harvest practices. By the third quarter of year six, based on project M&E data, about 22,000 farmers (sex disaggregated data on whole dataset not available) participated in SMASH training, about three quarters of the anticipated target of 29,300 farmers. In addition to training, SMASH strategies to promote improved farming practices included the use of demonstration plots to model the productivity benefits of adopting new techniques. By the third quarter of year six, SMASH reported that it had established 84 demonstration plots, fewer than the 104 it had anticipated. Significantly fewer farmers than targeted changed their practices. In its own analysis of training and extension service effectiveness, SMASH first considered the extent to which training and extension services were effective in changing production techniques and improving farmer yields. Second, in line with the Activity’s objective to foster a sustainable and cost-efficient supply 15 | SMASH FINAL EVALUATION REPORT USAID.GOV of locally grown sorghum for BRANA’s production needs, SMASH assessed the extent to which the interventions facilitated sales to the brewery. According to SMASH M&E data, by the third quarter of year six, approximately 2,900 farmers had adopted SMASH-promoted new technologies or practices, 29 percent of the end-of-project target. 21 SMASH managers, agronomists, and farmers all agreed that overall, the outcome of SMASH training and extension programs on production practices was mixed: some farmers adopted new practices while many others did not. Cost and risk factors as well as general openness to innovation limited the extent to which farmers applied new practices. Key informants provided various explanations for farmers’ limited application of improved practices following training and extension services. Farmers highlighted the cost of applying new practices as a significant factor, especially labor-intensive recommendations such as more careful planting techniques and frequent weeding, because of high cost and the availability of farm labor. Some FGD participants likewise attributed their reluctance to invest in the techniques due to the aphid infestation, which made achieving higher yields from additional efforts less likely, a supposition, which is borne out in evaluation survey findings on yields (see below). "We put into practice all the techniques and they gave good results.” WEST FGD “These techniques work and should continue. Except, the high cost of plowing and weeding may be an obstacle in the future. Also, if we do not have seeds, we will not be able to continue.” CENTER FGD “The difficulty in applying the techniques is the price of labor. It costs a lot of money. Workers are forced to raise their labor costs because of the high cost of living.” SOUTH FGD “The aphid disease discourages us from applying the techniques.” SOUTH FGD SMASH managers and agronomists agreed with the former explanations and added that not all of the farmers trained were motivated to improve their farming and may have attended training for other reasons, such as a belief that the training would be accompanied by free handouts, a common NGO practice in Haiti. Agricultural experts outside the implementation team concurred that farmers’ mindset can be a significant factor affecting the success of training and extension services. One MARNDR key informant affirmed, “There is a difference between a farmer and a peasant. A peasant doesn’t want to take risks. The challenge of scaling up modern production techniques is finding a way for peasants to become farmers.” Although they remained committed to the practice, SMASH also reported that its demonstration plots, which were either managed by SMASH agronomists or local farmers, were only modestly effective in changing farmer cultivation practices to date. Specifically, for SMASH-managed demonstration plots, SMASH management highlighted that demonstration plots used production methods that were not in line with what most farmers could afford (mainly mechanization). For farmer-managed plots, one SMASH manager highlighted that some demonstration plot owners were not well-selected or did not 21 SMASH tracked and reported on this IDB and BRANA indicator. According to SMASH personnel, its field agents initially tracked the farmers who adopted new techniques, but as the number of trained farmers increased, SMASH personnel noted that it was unable to assess the extent that all the farmers it trained adopted new practices rather only those with whom it had a contract. USAID.GOV SMASH PERFORMANCE EVALUATION | 16 understand the purpose of the demonstration plots, citing as an example one farmer who did not want other farmers on his property. Effect of training and extension services on sorghum yields was not clear. SMASH expected that farmers who were trained and applied improved production practices would (i) achieve higher sorghum yields, (ii) increase their income, and (iii) have a multiplier effect by promoting the adoption of better farming practice by other farmers. The ET was not able to determine the effect of training and extension on sorghum yields because of conflicting data on yields as well as the negative effect of external factors (mainly the aphid infestation) on yields during the life of SMASH. Data on SMASH farmer sorghum yields were inconsistent. The SMASH M&E table reported that yields increased over the life of the Activity from a baseline of one and a half MT per ha. The reported rate of increase diminished during the implementation period, 22 but was nonetheless positive, meaning that farmers who applied recommended good farming practices achieved higher-than-the-baseline yield. 23 In contrast, in KIIs, SMASH personnel and other technical key informants reported that since the aphid infestation, yields rarely attained even one MT per ha. Through the short survey, the ET collected additional data on farmer sorghum yields. In addition, it reviewed yield data before and after the aphid outbreak. Information collected from SMASH participants show that sorghum yield is low in all regions surveyed. Sorghum yield averaged 408 kg/ha among surveyed farmers (n=76). Compared to yield data available in 2013 when SMASH launched its activities, the short survey suggests that SMASH-affiliated farmers were not able to improve sorghum yield. Sorghum yield decreased by about 54 percent from 878 kg/ha in 2013 to 408 kg/ha in 2018. However, the results show that sorghum yield has improved since the aphid infestation outbreak by more than 70 percent, from 237 kg/ha in 2014 and by 28 percent from 2016. 24 In terms of production, the survey results show that farmers produced an average of 619 kg of sorghum in 2018. Considering the estimated 300,000 sorghum producers and using Ministry of Agriculture data on production reported earlier, sorghum production per farmer was 371 kg in 2013. This suggests that some SMASH-affiliated smallholder farmers were able to increase their sorghum production by about 67 percent. To assess their perception of the change in their level of sorghum production, farmers were asked to state how the volume of sorghum they produce has changed since their participation in SMASH interventions. Approximately 53 percent of farmers surveyed (n=76) stated that they had seen an increase in their level of sorghum production, although this change would vary from one year to another depending on the level of pest infestation and weather conditions. The majority of those who reported an increase in production were from the South. Since the aphid outbreak, most farmers in other areas (West, Centre) have stopped planting sorghum. They resumed sorghum production with the assistance of SMASH, which provided seeds and other support services. When asked about change in farmers’ sorghum production, there was no clear-cut answer among KIIs and FGDs. While farmers in the South tend to agree that their sorghum production has increased, those in the West and Central Plateau saw no improvement. Most of the KIIs were not able to indicate clearly whether farmers’ sorghum production had changed. 22 In its M&E table, SMASH reported that yields increased by 147 percent in year 1 and 105 percent, 104 percent, 81 percent and 2.6 percent in years 2-5, respectively. 23 SMASH reported that its determined yields by selecting one meter square and counting the number of heads/pods. One manager postulated the high reported yields may have be the result of selecting the most productive section of a field to conduct measurements. 24 Using data on production and area from MARNDR statistics, yield = 44151 MT/185943 ha. 17 | SMASH FINAL EVALUATION REPORT USAID.GOV Significantly fewer farmers than targeted became SMASH suppliers and received follow-up services SMASH management also highlighted that among training participants, fewer farmers than expected became SMASH suppliers. Using SMASH M&E data, Figure 2 compares the number of farmers trained relative to the numbers who sold BRANA sorghum. 25 Activity personnel postulated that the difference reflects external production constraints and farmers’ perception of the benefits of selling sorghum to BRANA versus continuing to sell in informal markets. Starting in year four (FY 16/17), SMASH scaled down its training program and began providing additional services to increase production and sales to a much smaller number of farmers who had proven their willingness to both improve production practices and sell sorghum to BRANA. Figure 2: Comparison of number of farmers trained and number of farmers selling sorghum to SMASH by Activity year SMASH substantially reduced the number of farmers receiving Alliance services over time. According to AMC and SMASH personnel KIIs, the Alliance changed its approach to focus on a much smaller number of farmers since it observed that its large-scale training program was largely ineffective in changing farming practices and ensuring an adequate sorghum supply for BRANA. Several SMASH managers described trends in the Activity’s participant population as funnel-shaped, meaning that they started with many participants, who over time and partly through self-selection, reduced to a small core group of motivated suppliers. With this progression, SMASH indicated that follow-up training and extension services were better targeted and more effective at changing practices, but for a much smaller number of farmers. SMASH farm financial management training, known as “Jaden’w se Biznis ou26” was an example of training reserved for farmers who SMASH determined to be motivated to work with the Alliance and improve their practices. SMASH reported that by the third quarter of year three, the activity had delivered “Your farm is your business” training to 180 farmers (25 female, 155 male), less than half the number planned during the life of Activity. SMASH managers, AMC members, and many farmers affirmed the usefulness of SMASH’s farm financial management training. According to the training organizer, the training included information on tracking costs, sales revenues, and money-management. SMASH and AMC informants noted that the training was successful in reinforcing farmers’ understanding 25 Year six includes only the first three quarters of FY18/19 data. 26 Means “your farm is your business” in Haitian Creole. USAID.GOV SMASH PERFORMANCE EVALUATION | 18 of the importance of tracking production costs against revenue. They noted that the training also highlighted how to evaluate non-price benefits of working with BRANA, such as access to mechanization, which freed up time for other activities. Many farmers who participated in the training attested to its positive results during FGDs. Several farmers across different FGDs affirmed “now we manage our farms like a business.” SMASH AGRONOMIST TRAINING: KEY RESULTS, GAPS, AND LESSONS LEARNED SMASH exceeded targets for the number of agronomists it trained from other organizations, but results were unclear. In addition to hiring its own agronomists, SMASH trained 390 agronomists 27 from other organizations who were working in SMASH communities, exceeding its target of 350. Training objectives for the other agronomists were to publicize SMASH and improve extension services for sorghum farmers provided by other organizations. SMASH did not include any indicators to measure agronomist training outcomes nor, based on the absence of follow-up engagement with participants, did it appear to have a strategy to forge any long-term alliance with the agronomists trained or their organizations. In the initial evaluation design, the ET planned to hold focus groups with SMASH trained agronomists. In its efforts to contact them, the ET remarked that a relatively small percentage (roughly 20 percent) were still reachable using the contact information SMASH provided. In addition, out of the seven agronomists interviewed, only one described following-up on the training to support other farmers; the majority appeared to engage in sorghum cultivation on their own farms. CREDIT AND PRE-PRODUCTION FINANCING: KEY RESULTS, GAPS, AND LESSONS LEARNED SMASH decided to delay connecting farmers to credit providers. To address farmers’ challenge in mobilizing resources to cover up-front production costs (highlighted above as a significant constraint cited by farmers affecting their ability to apply good production practices), SMASH planned to facilitate access to credit for 300 farmers. To date, the Alliance has not made any progress against this objective. According to one SMASH manager and the potential credit support organization, SMASH discussed a plan for facilitating agricultural credit services with the Center for Support and Services for Local and International Businesses (CASELI), a Port-au-Prince-based NGO that specializes in promoting rural entrepreneurship. Because of the effect of persistent low yields on farmer income, SMASH did not move ahead with the plan. It assessed that most farms were not profitable enough to repay the loan with interest, especially the high interest rates commonly charged by micro finance institutions in Haiti. SMASH’s assessment was affirmed by evaluation survey data on farmer incomes, which showed that less than half of surveyed farmers (n=76) turned a profit on their sorghum production (more on this under EQ3). However, according to SMASH management and its latest progress report, the Activity is taking another look at the potential to connect a limited number of its most profitable suppliers (about 10) to credit services in collaboration with CASELI. SMASH pre-production financing to 118 farmers incentivized production and sales. Based on SMASH personnel and farmer accounts, following the aphid infestation and Hurricane Matthew, SMASH began providing in-kind services (mainly access to mechanized field preparation and in some cases, support for weeding) as a form of pre-production financing to be reimbursed in sorghum at harvest time. SMASH initiated this practice in the South to incentivize farmers to replant sorghum following their severe losses (Hurricane Matthew) and subsequently continued and replicated the practice in other zones based on its positive results. Before providing pre-production financing, SMASH indicated that it required most farmers to have sold sorghum to them in previous harvests and organize 27 The ET noted several non-agronomist participants on SMASH participants lists, which included participant profession. 19 | SMASH FINAL EVALUATION REPORT USAID.GOV in a production block to make mechanization services cost-efficient. SMASH managers set a cap on the value of pre-production financing at 25 percent of the value of projected sales, which they estimated was small enough to prevent farmers from carrying excessive debt in case of poor harvests. “The number of fields increases or doubles each season. Before SMASH, it was much more difficult to produce sorghum.” PRE-FINANCING RECIPIENT FGD SOUTH When SMASH tills a land, were required to plant sorghum, it is a factor promoting an increase of the quantity of sorghum planted in the community. FGD SOUTH SMASH has increased our ability to produce sorghum but sometimes due to delays, lack of equipment (tractors), we plant a bit late and it reduces the yield on the plots. PRE￾FINANCING RECIPIENT FGD SOUTH Basically, the advance given by BRANA allows us to work faster. But if our yields don’t increase, there is no real benefit. PRE-FINANCING RECIPIENT FGD WEST “We have kept our commitment to BRANA because of the support it has given us.” PRE￾FINANCING RECIPIENT FGD WEST Based on SMASH M&E data, a total of 118 farmers received pre-production financing, 89 in the South/Nippes and 29 in the West/Center. About 40 percent of recipients received pre-production financing for more than one season. According to outside agricultural experts (e.g. other value chain actors), SMASH field agents, and farmers, pre-production financing was effective in motivating smallholder farmers to organize in blocks and plant more sorghum because the services facilitated production. Most farmers who received pre-production financing affirmed that it saved them time. However, several farmers, including one larger-scale farmer, highlighted that pre-production financing was not enough to enable them to apply better production practices because some labor-intensive activities such as planting and weeding were not covered. To be effective, farmers and agronomists highlighted that pre-production services needed to be provided on time to limit some pests (especially sorghum midge) and enable block farmers to synchronize their production schedule. Several farmers in the South cited cases of SMASH’s late provision of in-kind services (field preparation, harvesting). SMASH agronomists acknowledged some services were delayed and indicated that the delay was caused by the unstable political situation in Haiti (February 2019) and in some cases, by the difficulty in mobilizing needed equipment. SMASH management key informants indicated that pre-production financing was an effective way to ensure farmers sold at least part of their crop to SMASH. Many farmers said they felt a sense of obligation to sell some of their production to SMASH because of its pre-production support despite having the opportunity to sell to traders at a higher price. SMASH STRATEGIES TO PROMOTE SALES TO BRANA: KEY RESULTS, GAPS AND LESSONS LEARNED According to SMASH M&E data, by the third quarter of year six, farmers had sold significantly less sorghum to BRANA than anticipated. Over the life of the Activity, BRANA planned to buy 7,600 MT of sorghum, or about 1,250 MT per year. As of March 2019, it has achieved 18 percent of its end-of- USAID.GOV SMASH PERFORMANCE EVALUATION | 20 Activity target for volume of sorghum sold to BRANA. However, as shown in Figure 3, sales jumped significantly starting in year five, reaching 500 MT. 28 Figure 3: MTs of sorghum purchased by BRANA per Activity year SMASH increased its purchasing price over the implementation period to attract more sales. Based on project documentation, KIIs and FGDs, SMASH raised the price it paid for sorghum several times during the life of the Activity. The price SMASH paid increased from 15,000 gourdes per ton when implementation began to 40,000 gourdes per ton paid at the time of the evaluation. According to SMASH personnel, it increased the price taking into consideration competition from local traders and the increasing cost of production, but did not match the local market price for several reasons: (i) local traders did not provide the same level of support to producers as SMASH, (ii) the Activity was wary of increasing speculation, (iii) it did not want to reward low levels of production (meaning it thought farmers would be more motivated to use better production practices to increase yields if increased yields were built into how SMASH determined its price) and (iv) it did not want to raise prices to a level that exceeded the cost of importing sorghum. Among its reasons for sourcing sorghum locally, BRANA wanted to save on currency fluctuation costs since the sales price of its products in Haiti are set in gourdes and do not fluctuate to the same extent as the currency. SMASH aimed to set the price it paid for locally sourced sorghum under the price of “landed” imported sorghum. The ET provides more detailed analysis of market changes in the price of sorghum under EQ2. Pre-production contracts were not effective guarantees of sales SMASH tried to encourage farmers to both produce more sorghum and sell more of their production to SMASH by signing pre-production contracts with farmers. The logic behind contracting farmers was intended to be win-win. Contracts provided farmers with a guaranteed market for their production while enabling SMASH to forecast supplies in advance of the harvest. In practice, according to SMASH personnel, production contracts alone were not effective in increasing sales to BRANA. In its January￾June 2016 project status report to IDB, SMASH indicated that only about ten percent of farmers with SMASH production contracts respected the terms of the contract. SMASH personnel and other agriculture experts provided multiple explanations for why contracts did not work. These included the fact that farmers could sell sorghum for a higher price in the local market at the time of harvest, contracts were not enforceable, and farmers wanted to maintain their long-standing sales relationships with local traders, who also were eager to buy sorghum. This report discusses under EQ2 the likely 28 Year 6 only includes data from the first three quarters of the fiscal year 18/19. Assuming good harvests, the total volume of sales will likely increase by FY end. 21 | SMASH FINAL EVALUATION REPORT USAID.GOV effects of changes in supply and demand, and the effect of differences in pricing offered by SMASH and local buyers on farmer marketing decisions. Farmers in nearly all FGDs agreed that the attractiveness of signing contracts and selling at least part of their harvest increased once SMASH integrated pre-production financing into its sales contracts. Supplier loyalty program promoted sales to BRANA. In the latter years of implementation, SMASH tried to build trust and strengthen supplier loyalty through some “softer” approaches. A notable example SMASH management cited was BRANA’s support to its suppliers affected by Hurricane Matthew. In the disaster’s aftermath, BRANA offered small grants to farmers to enable them to plant the next season. The demonstration of goodwill reportedly strengthened the bond between BRANA’s core supplier group in the South and the company. Other examples of loyalty programs included BRANA-sponsored post-harvest celebrations and farmer VIP visits to the brewery in Port-au-Prince. Farmers reported appreciation for these goodwill gestures and said that the gestures encouraged them to replant sorghum and sell some of their production to BRANA. Contracting larger-scale farmers led to higher volume transactions, increased production and sales. According to SMASH and AMC KIIs, over time, SMASH shifted its procurement practices from sourcing mainly from smallholder farmers to a more diversified supplier group, including many larger-scale farmers. 29 SMASH’s primary reason for this shift was to meet BRANA’s production needs in terms of volume, quality, and stable supply (to be discussed under EQ2), an objective on which the viability of its local sourcing strategy hinged, according the SMASH reports. AMC KIIs also indicated that BRANA plans to lease land and produce its own sorghum in the future, further diversifying its local sourcing strategy and increasing production. “It has proven extremely difficult and challenging to source the required volumes of sorghum from smallholders only. The lesson learned at this point in the program is that our supplier base needs to be more diversified with larger producers and possibly our own sorghum production on demonstration fields. The smallholder farmers can then benefit from the structures and the supply chain will become more efficient overall.” SMASH JULY-DECEMBER 2016 PROGRESS STATUS REPORT According to the document review and KIIs, few large farms grew sorghum prior to SMASH, largely because demand for sorghum in bulk was previously limited (more on changes in the sorghum market in EQ2).30 According to AMC and SMASH personnel KIIs, SMASH sought out larger scale farms and motivated them to grow sorghum by offering production contracts and pre-production financing. The larger scale farm sourcing strategy was a significant factor in the increased volume of sales to BRANA reported in the last two years of the Activity, according to AMC and SMASH personnel KIIs. SMASH-supported farmers sold a higher percentage of their production, though not necessarily to SMASH. Traditionally a subsistence crop, small-scale farmers who produced the grain saved a portion of their harvest for home consumption, especially during the lean months, and sold the remainder in the local market for cash. A study conducted in the Artibonite and Cul-de-Sac (West) areas in 2009 indicated 29 In the farmer master list provided to the ET by SMASH, there were 68 farmers with greater than 10 ha and 18 farms larger than 50 ha. 30 Larger sorghum farms would have needed to sell large quantities immediately after harvest like all the small famers when price was low because they did not have enough storage facilities to keep the grain. Because they had lands, some large-scale farmers (3-5) produced sorghum when they had a contract with specific clients. USAID.GOV SMASH PERFORMANCE EVALUATION | 22 that small-scale farmers sold about 40 percent of the sorghum grain they produced. 31 After harvest, they dried their sorghum grain and sold this share in bulk in local markets or at the farm gate to local traders. 32 Local Madam Saras milled the grain in small mills located in the vicinity of local markets and sold the milled sorghum to other traders who distributed the product to retailers and consumers. SMASH-affiliated farmers surveyed for the evaluation indicated that they sold their sorghum to SMASH and to buyers in local markets. Approximately 28 percent (n=76) of survey respondents declared that they did not sell to anyone because their production was too low, so they kept it for their own consumption. The survey results show that a farmer sold on average 567 kg of sorghum, which represents about 68 percent 33 of the volume produced (with volume of sales ranging from 26 percent to 100 percent). This is a significant departure from the 40 percent farmers traditionally sold in the market. The survey showed that farmers sold 79 percent of the volume sold to SMASH and 21 percent to other buyers in local markets. When asked about the change in their sorghum sales since their participation in SMASH interventions, 55 percent of survey respondents declared that the volume they sold had increased. Forty-five percent of surveyed farmers indicated that their sales of sorghum had decreased or remained unchanged since their participation in SMASH interventions. As in the case of production, most farmers who declared that their sorghum sales had increased were from the South. According to FGD participants, while selling part of their production to BRANA, they also sold to local buyers because of the higher price they received from them. Some farmers point out that BRANA asks for higher-quality sorghum, while paying lower than market price for the product. CONCLUSIONS SMASH EFFECTIVENESS IN INCREASING FARMERS’ SORGHUM PRODUCTION WAS LIMITED, MOSTLY BY EXTERNAL FACTORS. Because of the lack of quality baseline data, the ET cannot strongly state that SMASH interventions have been effective in increasing farmers’ sorghum production and sales. Short and non-representative survey results show that average sorghum yield of a sample of SMASH-affiliated farmers was more than 50 percent lower than the pre-SMASH implementation estimated yield. However, survey results and farmer accounts also suggest that SMASH services (training, seed, pre￾production financing) helped some farmers in selected areas to increase their volume of production. The production level of the small sample of surveyed farmers increased by an average of 67 percent and suggested that farmers, particularly those cultivating sorghum in humid plains in the South who received assistance from SMASH, achieved higher volume production in 2018 than in 2013. Although SMASH trained 22,000 farmers, only a small proportion of these received a full assistance package (at the time of the evaluation (April/May 2019), approximately 500 farmers received aphid resistant seeds; 118 received pre-financing). The overall effect on sorghum production was likely limited given farmers’ difficulty in accessing quality inputs. Evaluation findings on the contextual factors affecting implementation show that SMASH interventions to increase production were impeded significantly by the persistent effects of the aphid outbreak (which, starting in 2014, resulted in consecutive bad harvests and low yields) as well as by natural disaster (Hurricane Matthew, which struck in 2016), and recurrent drought and floods. 31 Agroconsult Haiti S A, Etude des systèmes de production agricole et des associations paysannes dans les bassins versants de la rivière la Quinte et de la rivière Grise. Port-au-Prince. USAID/WINNER. 2009. 32 Fews Net. Haiti: Staple food market fundamentals. 2018. 33 The 68 percent quoted here is the average of the average proportion sold for each of the 76 farmers surveyed as part of the evaluation. If the average quantity of sorghum sold is divided by the average quantity produced (a methodology dictated in the FtF handbook but one that does not take into account outlier production quantities from large farms), the proportion is 92 percent. 23 | SMASH FINAL EVALUATION REPORT USAID.GOV SMASH’S PROVISION OF APHID-RESISTANT SEED POSITIVELY CONTRIBUTED TO SORGHUM PRODUCTION RECOVERY IN ITS INTERVENTION AREAS. Based on farmer and other value chain experts’ accounts highlighted in evaluation findings, SMASH contributed effectively to getting farmers in intervention zones back to sorghum production through the distribution of sorghum seed variety Papèpichon. Many farmers indicated that SMASH-facilitated access to the seed encouraged them to plant sorghum, despite the continued threat from aphids. For a fuller recovery, value chain experts highlighted the need to stabilize and improve aphid and drought tolerance of the existing seed variety and fill gaps in SMASH aphid-resistant seed distribution, which did not cover all former sorghum growing regions. SMASH TRAINING AND EXTENSION SERVICES WERE NOT SUCCESSFUL IN FOSTERING LARGE-SCALE IMPROVEMENTS OF FARMER SORGHUM PRODUCTION PRACTICES AND YIELDS. Although SMASH trained 22,000 farmers and established 84 demonstration plots, nearly meeting its targets, evaluation findings highlighted that substantially fewer farmers than expected changed their production practices, largely because of the cost of applying new practices, the continued risk of poor harvests posed by aphids, and the extent to which they were willing to innovate and take risks. The ET could not determine the effectiveness of SMASH training and extension on increasing sorghum yields because of conflicting yield data as well as the negative effect of external factors, mainly the aphid infestation. SMASH PRE-PRODUCTION FINANCING WAS AN EFFECTIVE INTERVENTION IN PROMOTING SORGHUM PRODUCTION. SMASH rightly determined that because of continued low yields, most sorghum farmers were not financially able to manage credit. The Activity’s alternative, pre￾production “in-kind service” financing (mainly access to mechanization for field preparation), was generally effective in motivating smallholder farmers to organize in production blocks (a requirement for pre-financing) and to plant more sorghum (since the services facilitated production). ET findings also showed that the practice was more effective in promoting sales to BRANA than other strategies, such as production contracts without services, because receiving services increased the farmers’ sense of obligation to BRANA. Evaluation findings also showed that other “softer” strategies to strengthen trust between farmers and BRANA, such as BRANA’s post-hurricane farmer grants and supplier loyalty programs, were effective in generating sales to BRANA. SMASH WAS NOT SUCCESSFUL IN ESTABLISHING A LARGE SUPPLIER BASE FOR BRANA. Fewer farmers than expected became SMASH suppliers. Findings suggested several reasons for this outcome, including the Activity’s underestimation of market competition, overestimation of farmers’ valuation of supplier contracts, and Brana’s evolving perception of cost-effectiveness and efficiency of having a large number of small producers in its supply chain. Because of this, SMASH scaled back training and provided additional production services and follow-up support to smallholder farmers who demonstrated their interest in changing their farming practices and selling sorghum to SMASH. SMASH also integrated a greater number of larger-scale farmers into its supplier network, significantly reducing the number of small-scale farmers regularly receiving SMASH services over time. SMASH AND OTHER SORGHUM GRAIN MARKET FACTORS CONTRIBUTED TO FARMERS SELLING A HIGHER PERCENTAGE OF THEIR HARVESTS. Despite the low levels of production and yields observed in the sorghum value chain, SMASH was effective in stimulating some farmers in intervention areas to increase their sorghum sales. Short survey results show that some SMASH￾affiliated farmers sold a higher percentage of their harvest compared to pre-SMASH levels (68 percent against 40 percent). Evaluation findings show that the rising price of sorghum in local, informal markets as well as SMASH production incentives tied to sales, led farmers to sell most of their harvest, a break with past practices of saving a higher proportion of production for home consumption. EQ2: TO WHAT EXTENT DID THE PROJECT IMPROVE MARKET ACCESS FOR SMALL-SCALE FARMERS AND IN WHAT WAYS HAS THE SMASH PROGRAM USAID.GOV SMASH PERFORMANCE EVALUATION | 24 AND OTHER EXTERNAL FACTORS AFFECTED THE SORGHUM GRAIN MARKET IN HAITI? To answer EQ2, the ET examined small-scale farmer market access to Haiti’s informal and emerging formal sorghum grain market. The team then examined strategies implemented by SMASH to integrate smallholders into BRANA’s supply chain and the factors that contributed to and hindered progress. The section finishes with analysis of how SMASH and other external factors affected the sorghum market in Haiti as well as how market dynamics affected SMASH implementation. FINDINGS SMALLHOLDER ACCESS TO HAITI’S SORGHUM GRAIN MARKET Smallholder informal market access was facilitated by traders, mainly women. According to the ET’s document review, prior to and, to a large degree, during SMASH, the sorghum grain marketing system in Haiti was mainly unstructured and informal, composed of a multitude of players (many smallholder producers, small-scale traders, and local merchants). 34 Part of the harvest was consumed at home and another part sold in local markets. “There are no marketing problems for sorghum. The "Madam Sara," who crisscross the cities and buy and resell produce, come to buy our sorghum even before we producers go to markets….” FGD SOUTH “There are no problems with the sale of sorghum. On the contrary, there is not enough to give to BRANA and Madam Sara.” FGD WEST To get sorghum from fields to market, a network of traders, mostly women known as Madam Sara, procured farmers’ sorghum at the farm gate and distributed the commodities across the country toward final consumers. 35 Madam Sara operate at different levels in the market. Some are local people who have long-standing relationships with farmers and sell in the local market while other traders buy in local markets and sell in regional markets or in Haiti’s largest market in Port-au-Prince. Madam Sara offer their marketing services and, in some cases, cash advances, in exchange for farmers’ business. While they have been described as sometimes cheating farmers on price, the services provided by Madam Sara (cash purchases, consolidating, transporting, milling, and selling) are critical to smallholder farmers’ market access. 36 SMASH opened a new, formal market for sorghum to smallholder farmers. According to other value chain actors and MARNDR KIIs and FGDs, before SMASH, smallholder farmers did not have access to larger-scale, formal sorghum markets. Starting in 2013, SMASH created a new, formal market for high-quality, bulk sorghum, which it intended to source from at least 10,000 smallholder farmers. Initially, SMASH forecasted its purchases at 1,250 MT of sorghum annually for BRANA (although according to AMC KIIs, at the time of the evaluation the company estimated yearly demand to be between 800 and 1,000 MT.) In addition, when EDN, the national conditioning center, was created through SMASH and USAID Haiti/LEVE support, other formal sector markets for sorghum including local bakeries and school feeding programs opened. Based on 34 Eckert Elisabeth and Anna Latané (2017). The sorghum value chain in Haiti: A mapping and analysis. RTI International. International Development Working Paper No. 2017-04. 35 Chemonics International Inc (2018). HAITI Staple Food Market Fundamentals, Famine Early Warning Systems Network (FEWS). https://reliefweb.int/sites/reliefweb.int/files/resources/Haiti%20MFR_final_20180326%20%281%29.pdf. 36 Stam, Tahitha (2012). From Gardens to Markets – a Madam Sara perspective. 25 | SMASH FINAL EVALUATION REPORT USAID.GOV SMASH documentation, BRANA intended that its local sourcing policy would be a win/win partnership with smallholder farmers. BRANA would obtain a cost-efficient supply of locally produced sorghum while smallholder farmers would benefit from a new, stable market for their production. EFFECTIVENESS OF SMASH INTERVENTIONS TO FACILITATE SMALLHOLDER MARKET ACCESS SMASH post-harvest services improved sorghum quality and facilitated marketing logistics. SMASH intended to avoid informal market channels (e.g., Madam Sara and other traders) by developing systems to buy directly from smallholders, either as individuals or in groups. SMASH interventions to facilitate smallholder access to BRANA’s market were designed to improve the grain quality and facilitate supply chain logistics from the field to the central conditioning center run by EDN in Port-au￾Prince. According to SMASH progress reports and farmer accounts, at harvest time, SMASH brought in mechanized threshing and provided material for stacking and bagging harvested sorghum. To consolidate smallholder production, as of the third quarter of year six, SMASH established 84 MCCs (out of 114 planned) near smallholder farms. The Alliance also transported the grain from the MCCs to EDN free of charge for the farmer. “There is a change our post-harvest practice. Before, we harvested and treated the sorghum with some insecticide against mites and stored it without separating the grain. Now we use a threshing machine right in the field and we keep the grain in bags provided by BRANA.” FGD PLATEAU CENTRAL "BRANA comes to take our harvest on the spot. We do not have to do anything." FGD WEST “We have no problem for the sale because BRANA comes to take it. But the price of BRANA (100 gourdes) is the reason why we cannot sell all our sorghum to BRANA. We also sell on the local market to be able to make a few extra pennies to take care of our children.” FGD SOUTH Based on FGDs, farmers were appreciative of SMASH support to improve their post-harvest practices and facilitate market logistics. Several farmers noted that the new practices improved sorghum quality by eliminating the small stones that commonly were found in the grain. BRANA quality standards constrained smallholder participation in the company’s supply chain. The previously described measures were only partially successful in overcoming challenges affecting smallholder farmers’ capacity to meet BRANA’s quality standards, according to AMC and SMASH personnel KIIs. BRANA quality standards 37 required getting the sorghum from the field to the processing unit as quickly as possible to avoid increasing aflatoxin levels. This issue affected smallholder farmers more than larger-scale farmers because the expense of transporting the small quantities they produced was prohibitive and resulted in sorghum accumulating in MCCs too long. According to AMC KIIs, because of smallholders’ difficulty in meeting its quality standards, BRANA’s production costs were inflated twice over: first from transportation charges and, second, because it agreed to sell back sub￾standard sorghum to EDN at 70 percent of its original purchase price. Based on SMASH KIIs and progress reports, the Alliance planned to address the problem by introducing drying facilities closer to the farmgate (to be known as “super MCCs” because they were to be fewer and better equipped than normal MCCs) but none were operational as of the third quarter of year six. At the time of the 37 BRANA required aflatoxin levels of less than four ppb which are significantly stricter than the US Food and Drug Administration (FDA) standards of less than 20 ppb. USAID.GOV SMASH PERFORMANCE EVALUATION | 26 evaluation, SMASH also was experimenting with alternative collection methods such as purchase of panicles to speed transportation of the harvest to the processing center. SMASH-piloted systems eased management burden, but other local sourcing costs remained high. SMASH also introduced systems to streamline supply chain management aimed to make working with many small producers more efficient. After initially transporting cash to the field for farmer payments, SMASH eventually outsourced the micro payments to Unitransfer, a cash transfer service. Initially resisted by farmers, acceptance for the payment system grew over time, according to SMASH management reports. “I think it [SMASH mobile] has made the value chain more efficient. It has formalized our relationship with suppliers. Also having good data improves the efficiency of decision-making. The quality of data coming out of the MCC has been drastically better.” SMASH PERSONNEL KII “It is a good system, even if it creates a bit of frustration. With a bit more effort, we could capitalize better on its potential. For now, the data is not used in a timely way.” SMASH PERSONNEL KII “SMASH Mobile is a useful application, but it still needs some improvements.” SMASH AGRONOMIST FGD SMASH also developed a mobile application known as “SMASH mobile” to improve its management efficiency. According to SMASH personnel, the application functions as three integrated modules. The first module holds supplier information, including farm data (number of fields planted), the supplier contract terms, and records SMASH-provided services. This module built in SMASH field agent workflow management by, for example, scheduling extension visits based on the sorghum production calendar. The second module holds information on SMASH post-harvest purchases, while the third tracks sorghum purchases from the MCC to BRANA through EDN’s conditioning center. To date, the main SMASH mobile users are SMASH field agents, their supervisors, and SMASH administrators. According to the SMASH employee in charge of supporting the application, about 90 percent of SMASH field agents use SMASH mobile effectively, a rate significantly higher than was his experience in previous projects using mobile data collection devices. According to one SMASH manager, the application has significantly improved the quality and efficiency of Activity M&E systems, which previously relied on at least 20 different paper-based forms. However, one field supervisor noted that the system was not yet fully exploited by management for decision-making. SMASH agronomists also highlighted that improvements were still needed to make the tool efficient to use. CHANGES IN SMASH LOCAL SOURCING STRATEGY REDUCED SMALLHOLDER SUPPLIER NUMBERS SMASH’s new supply chain model requires smallholders to create production blocks to be eligible for most services. According to SMASH personnel and AMC KIIs, despite its efforts to improve efficiency, the cost of providing both production and post-harvest services to many widely dispersed smallholders was significant relative to the sorghum that these farmers produced and sold to BRANA. One SMASH key informant recounted, “There was pent up demand for basic fieldwork services. We tried to build up the service capacity. But it required grouping farmers together.” Stakeholders inside and outside Activity management, affirmed to provide mechanized plowing and threshing efficiently, plot sizes needed to be at least ten ha while the average smallholder farm was one ha or less. 27 | SMASH FINAL EVALUATION REPORT USAID.GOV “The project advised us to group together because it would give us the advantage of producing more sorghum.” FGD SOUTH “We want to work together and organize ourselves so that those who have plows help those who do not. But up until now, we’ve found this hard.” FGD SOUTH “When you cluster farmers, there are new opportunities. It is the future of agriculture in our country – it reduces production costs, makes mechanization possible and creates incentives for farmers to help each other.” OTHER VALUE CHAIN ACTOR KII “It is extremely difficult to get Haitian farmers to work together. Not impossible but very difficult. It will be hard to get a mass of small farmers organized into blocks so that they can benefit from mechanization.” OTHER VALUE CHAIN ACTOR KII As noted in EQ1, SMASH decided that to create a sustainable supply chain, the Alliance would scale back its ambitions to source from thousands of smallholder farmers to a few hundred who were willing to farm in blocks or associate with a nearby large-scale producer. Also, as highlighted under EQ1, at the same time, SMASH diversified its supplier base to include larger-scale farms and potentially in the future, will also produce on BRANA-managed farms. Various KIIs with knowledge of Haiti’s agriculture sector highlighted the benefits of block farming while also underlying the challenges to spread the practice given long-standing farmers’ preferences to work alone. One MARNDR official said block farming required successful social engineering. Farmers likewise highlighted they found it challenging to organize in blocks while also acknowledging the benefits. NUMBER AND CHARACTERISTICS OF SMASH SMALLHOLDER SUPPLIERS To improve the cost-efficiency of providing services, SMASH progressively consolidated its smallholder supplier numbers to a few hundred. As of the third quarter of year six, SMASH had procured sorghum from approximately 4,500 farmers (sex disaggregated data not available for whole dataset), mostly smallholders. Figure 4 shows how SMASH supplier numbers evolved over time. According to SMASH management, BRANA plans to consolidate its supplier base at around 200-350 farmers during the planned second phase of SMASH38 while still pursuing new and on-going strategies to increase the number of smallholders who meet its eligibility requirements (sorghum quality, ease of providing needed services). Figure 4: Number of farmers who sold sorghum to SMASH/BRANA by Activity Year 38 Based on AMC KII, BRANA intends to continue investing the sorghum value chain in Haiti and plans a SMASH phase II. At the time of the evaluation, USAID was considering whether to support the second phase. The IDB indicated that it did not intend to support phase II. USAID.GOV SMASH PERFORMANCE EVALUATION | 28 SMASH’s core group of smallholder suppliers were mostly older men. Based on SMASH M&E data, men significantly out-number women among Activity participants. 39 Value chain stakeholders in all categories as well as activity reports indicated that the low number of women involved in SMASH was largely explained by customary gender roles in sorghum production. Key informants highlighted that men are mainly involved in production while women are mainly involved in marketing. They also highlighted that there are fewer women farming large plots of land. “Women are not really involved in sorghum growing but they help their husband. They mainly deal with the sale of sorghum in the local markets.” FGD SOUTH “There is no real difference between men and women…except that there is some work that women do not do. Plowing for example. And women are mainly the ones that market the product.” FGD SOUTH “Women (sorghum farmers] are fewer in number, so we must motivate them more. However, they have a lot more discipline. They are more open and interested. In the “jaden” training program, the women participated more, had better ideas and showed stronger understanding in general.” SMASH MANAGER While it works with many women farmers, SMASH has not worked with Madam Sara. According to SMASH managers, the Madam Sara’s methods are incompatible with BRANA’s quality standards because they are too slow getting sorghum to market. However, SMASH, AMC, and other value chain actors and KIIs all recounted a story of an industrious Madam Sara delivering a truckload of high-quality sorghum to BRANA after hearing about the program on the radio, which suggests that there may be capacity differences among the women traders. The ET likewise observed that SMASH hired many women agronomists but an all-male cadre of farmer liaisons to facilitate field agent purchasing from smallholders in the South, a role with some similarities to local Madam Sara. Age is another key distinguishing characteristic of SMASH-affiliated smallholder farmers. Based on the ET analysis of SMASH farm financial management training, a group that one SMASH manager indicated constituted it core supplier group, men over the age of 50 constituted 45 percent. Various value chain stakeholders described the challenge of attracting young people to farming because of urban migration and the attraction of easier trades, such as being a motorcycle taxi driver, were among the reasons provided. SMASH did not report any specific strategies to attract youth to its supplier network. SORGHUM MARKET DYNAMICS DURING SMASH BRANA’s market entry coincided with reduced supply and was a significant factor among others increasing demand. When SMASH entered the sorghum market, smallholder farmers were the main suppliers of sorghum for human consumption in the market. Based on sorghum farmers’ selling behavior and the level of production, market supply of bulk sorghum were estimated at about 45,000 MT in 2013.40 Although there were no barriers to enter the market, imports of sorghum for human consumption were 39 Because it used multiple data collection systems for M&E and has not consolidate these in one formal, the ET does not have comprehensive data disaggregated by sex. However, partial data sets show male producers significantly outnumbering female SMASH affiliated producers. 40 40 percent of the 111346 MT produced in 2013 according to MARNDR statistics. 29 | SMASH FINAL EVALUATION REPORT USAID.GOV practically unknown because local production satisfied the demand.41 Despite SMASH efforts to increase sorghum production and sales, the market supply was significantly down following the sharp decline of yields and production due to aphid infestation and other adverse environmental conditions. Sites visits carried out during the evaluation period showed that bulk sorghum was scarce in all markets. Milled sorghum was not as scarce as bulk sorghum. In Fond-des-Nègres and Mirebalais, the small quantities of bulk sorghum observed in the markets were offered by traders who bought the grain from farmers in mountain areas or from those who stored the grain while waiting for better prices. In Kans, farmers supplied the grain in the market during the site visit. Madam Saras interviewed in the markets indicated that only a limited number of farmers who planted sorghum early (December) were supplying the market at the time of the evaluation and said that they anticipated an increase in June/July. Before SMASH, sorghum demand for human consumption was primarily from low-income households, especially in rural areas. Small-scale traders were the main buyers of sorghum in the market. With a per capita consumption of sorghum of five kg per annum in 2013, demand for human consumption sorghum was estimated at about 100,000 MT, with a large proportion kept by farmers for home consumption. Local production was enough to meet this demand.42 As depicted in Figure 5, based on SMASH M&E data, through SMASH, BRANA purchased a cumulative volume of 1,341 MT as of the third quarter of year six. The highest volume of sorghum purchased was 511 MT in 2018. Although BRANA purchased less than anticipated, its total volume of purchases increased demand over pre-SMASH levels and had a larger than expected impact on the grain market because of the large fall in supply. EDN, which not only processed bulk sorghum for BRANA, but also milled the product to sell to other large buyers, also contributed to increasing demand. According to SMASH reporting, EDN sold sorghum to Food for the Poor 43, Epi d’Or, and supermarkets. Based on KIIs and FGDs across all stakeholder categories, stakeholders concurred that demand for processed sorghum for human consumption increased during SMASH implementation due to trends in the Haitian diet. Numerous stakeholders recounted that the popularity of eating sorghum among urban consumers had increased due to its purported health benefits. Figure 5: SMASH Cumulative and Absolute Volume of Purchases by Year 41 Fews Net (2018). Haiti: Staple food market fundamentals. 42 Fews Net (2018). Haiti: Staple food market fundamentals. 43 Based in KII, sales to Food for the Poor had stopped at the time of the evaluation due to the high price of sorghum relative to the price of imported rice. Representatives of the organization indicated it had wanted to support local production but chose not to continue when the price differential with other grains became too large. USAID.GOV SMASH PERFORMANCE EVALUATION | 30 Based on Activity reporting and KIIs, BRANA and EDN increased the availability of bulk sorghum in the country in 2017 and 2018 by importing sorghum. Representatives of both companies affirmed that given the level of supply and demand for locally grown sorghum in the market, competition was significant during SMASH implementation. Madam Sara interviewed by the ET during market site visits likewise affirmed that they competed with SMASH and EDN for sorghum during the harvest period. During SMASH, the price of sorghum rose sharply in informal markets, offering a higher paying alternative to selling to SMASH. As depicted in Figure 6, market prices collected by the Ministry of Agriculture Statistic Service with donors’ technical and financial support showed that the sorghum price increased over time in markets where farmers supply their stocks. Figure 6: Price of sorghum in three local markets 2013-2018 Market participants in all SMASH intervention areas acknowledged this change. Depending on the market, the price of unprocessed sorghum in local markets was 25 to 50 percent higher than the price paid by SMASH. At the time of the evaluation, during market site visits, SMASH was buying sorghum at 100 gourdes per “marmite,” whereas the price was 175 gourdes per “marmite” in Fond-des-Nègres and Mirebalais and between 150 to 175 gourdes in Kans. Based on SMASH documentation, one of the economic impacts expected from SMASH was for small farmers to receive a higher effective price per ton than if sold through Madam Sara or the open market. 44 However, based on the price data found in Figure 6, throughout the SMASH implementation period, the price of sorghum in local markets, where small-scale farmers usually sell their grain, was higher than the SMASH price. Despite this price difference, many farmers who participated in evaluation FGDs said they felt they had gained because they could sell their production in either or both markets. Because of market conditions, many SMASH suppliers were not satisfied with SMASH price and sales conditions. In FGDs, many smallholder farmers indicated they were not satisfied with SMASH prices for two reasons: their cost of production had increased with the overall fast-rising cost of living in Haiti and 44 Public Private Alliance Collaboration between USAID and BRANA. June 2014. 31 | SMASH FINAL EVALUATION REPORT USAID.GOV because they could obtain a better price for their production on the local market. Nevertheless, many farmers indicated they sold at least part of their production to SMASH in consideration of the pre- and post-harvest services provided by the Activity. In KIIs, large-scale farmers likewise indicated dissatisfaction with SMASH pricing (although the sample size was very small). One farmer strongly objected to being paid in gourdes based on the rate of exchange prior to planting. Because of the devaluation of the gourde, he insisted that the sales price of his sorghum should be determined based on the rate of exchange with the dollar on the day of sale, arguing many of his costs, like BRANA’s, were in dollars. Two other large-scale farmers indicated that SMASH’s current price was not enough to cover their costs given low yields. In Plateau Central, given the low performance of the seed varieties provided by SMASH and the price they received, one large-scale farmer indicated that he had incurred significant financial losses. CONCLUSIONS SMASH CONTRIBUTED SUBSTANTIALLY TO THE CREATION OF A NEW, FORMAL MARKET FOR HIGH-QUALITY SORGHUM IN HAITI. BRANA’s local sourcing policy created a new market for high-quality bulk sorghum which came with a corporate commitment to integrate small-scale farmers in the supply chain to the extent that it was a win/win arrangement. Before SMASH, smallholder farmers did not have access to formal agricultural markets and generally there was no formal market for locally grown sorghum in bulk. When SMASH and USAID/LEVE supported the creation of EDN (the central conditioning center) to meet BRANA’s requirements, SMASH and LEVE indirectly contributed to opening other new markets for locally grown bulk sorghum, such as bakeries and school feeding programs. FOR MOST SMALLHOLDER SORGHUM FARMERS, SMASH DID NOT CHANGE HOW THEY MARKETED THEIR SORGHUM. Evaluation findings show that before and during SMASH, many smallholder farmers sold their sorghum through informal markets facilitated by traders, mainly women known as Madam Sara. The informal marketing system paid farmers in cash, on occasion offered cash advances, and handled sorghum consolidation and transportation to markets without requiring exceptional quality standards. Although this system was not very efficient, it was largely acceptable to farmers based on their accounts. DESPITE SMASH EFFORTS TO FACILITATE AN INCLUSIVE SUPPLY CHAIN, HIGH LOGISTICS COSTS AND LOW PRODUCTION QUALITY LED SMASH TO INSTITUTE ELIGIBILITY REQUIREMENTS THAT LIMITED ACCESS BY SMALLHOLDER FARMERS. SMASH contributed to facilitating small-scale farmers’ access to BRANA’s market by addressing their logistic constraints (creating storage facilities and managing grain transportation) and easing local sourcing administrative challenges (by piloting supply chain management and payment systems). However, these services were not enough to ensure that smallholder production met BRANA’s high quality standards, mainly because of the cost and complicated logistics of moving small sorghum quantities from the field to the processing center quickly. This factor and the overall high cost of working with many, highly dispersed farmers triggered a change in SMASH’s local sourcing strategy requiring smallholder farmers to form production blocks or to associate with proximate larger-scale producers. The effect of this decision, while not excluding smallholders, limited the number of them that were willing or able to participate in the SMASH Activity. The evaluation found that within sorghum farming communities, SMASH interventions were more effective including males over females and older over young farmers in BRANA’s supply chain. Findings show that while on one hand the disproportionate involvement of older men was largely a reflection of Haitian sorghum farmer demographics, the Activity had no strategy to involve women traders in its supply chain nor did it have any targeted strategies to involve younger farmers. USAID.GOV SMASH PERFORMANCE EVALUATION | 32 BRANA AND EDN ENTRY IN THE MARKET INCREASED DEMAND FOR BULK SORGHUM IN THE COUNTRY AND CONTRIBUTED TO INCREASING MARKET PRICE, WITH POTENTIAL BENEFITS FOR SORGHUM PRODUCERS. Prior to the aphid infestation, SMASH would most likely have had limited effect on the national market because of the small volume it purchased. However, because of the decline in supply, its entry had a larger effect on the market, particularly in 2018, when SMASH purchased over 500 MT of sorghum. Evaluation findings indicate that because of overall high competition for sorghum and SMASH pricing policy, farmers received higher prices from Madam Sara than from SMASH and as a result most smallholder farmers who produced sorghum during the Activity implementation period would have gained more by selling in local, informal markets. Farmers who sold to SMASH likely did so because they wanted SMASH services and felt an obligation to repay these in sales. EQ3: TO WHAT EXTENT AND IN WHAT WAYS HAS THE SMASH PROGRAM BEEN SUCCESSFUL IN STRENGTHENING THE SORGHUM VALUE CHAIN IN HAITI? THE EVALUATION SHOULD INCLUDE FARMERS’ INCOME AND SUSTAINABILITY IN ITS ANALYSIS OF THE STRENGTH OF THE SORGHUM SUPPLY CHAIN. To answer EQ3, the ET describes SMASH interventions to strengthen the sorghum value chain, the changes that occurred, and the extent to which the changes are likely to be sustainable. The latter part of the section focuses on how SMASH affected farmers’ income and analyzes how results as of March 2019 may affect the medium- and long-term sustainability of SMASH results. FINDINGS SMASH INTERVENTIONS TO STRENGTHEN SORGHUM VALUE CHAIN BRANA’s demand for high quality bulk sorghum induced broader changes in the value chain. Value chain stakeholders in all categories largely agreed that BRANA’s local sourcing policy and SMASH’s efforts to implement it was a positive force driving changes in the sorghum supply chain in Haiti. At the processor level, many key informants cited EDN’s emerging success in finding new markets for bulk sorghum. Many farmers affirmed they changed practices when they were faced with new buyer requirements. One farmer in the West highlighted, “To meet BRANA’s requirements, we had to change our methods of drying, sorting, cleaning, and harvesting.” In the same group, another farmer remarked that the changes affected all of his production, not only the portion sold to BRANA, “The quality of sorghum improved overall because we kept almost all the same practices that we used to produce for BRANA.” Other farmers similarly affirmed they organized in production blocks, a desirable evolution in Haitian farming practices according to multiple KIIs, because doing so was a requirement to receive SMASH services. “Over the years, based on learnings, we have changed our strategies to address climate, farmer loyalty, price and other challenges…More and more I think we are starting to understand what is working and what is not working. Now we see that things are working better and moving in the right way.” AMC KII “BRANA has shown it is ready to invest….it made the decision to renew the SMASH without waiting for the response of donors about funding SMASH II.” SMASH PERSONNEL Other key informants highlighted lighthouse effects from SMASH. On the promotion of improved seed use, one key stakeholder in the sector remarked, “With the presence of SMASH, the return [to sorghum] went a lot better than where they were not. Sorghum almost returned to its former strength. I am convinced that SMASH played a role.” Although EQ1 shows, in general, not as many farmers changed their practices as was foreseen, several thousand farmers did adopt new practices, many more than the number who were regularly selling sorghum to SMASH at the time of the evaluation. 33 | SMASH FINAL EVALUATION REPORT USAID.GOV BRANA has a long-term commitment to sorghum value chain improvements in Haiti. In KIIs, BRANA affirmed a long-term commitment to sourcing sorghum locally and indicated its intention to continue to participate in the SMASH Activity, whether it receives additional donor resources or not, although management also highlighted that they hoped their partnership with USAID would continue. Based on KII accounts, BRANA’s local sourcing program is its largest activity within its corporate social commitment portfolio in Haiti. Key informants within the company highlighted Heineken’s corporate commitment to the strategy and multiple business reasons for wanting it to succeed. Business reasons cited by BRANA management included to help manage local currency devaluation risk, to develop the economy, including consumer buying power for BRANA’s products, and to promote good relations with the GoH. In BRANA’s advertising for Malta H, the ET observed that the company is publicizing the local origins of the sorghum used to produce the beverage. BRANA key informants also cited successful examples of the strategy working in other countries after a similar period of trial and error. BRANA management expressed optimism that the company is learning from its challenges and moving toward a sustainable and successful strategy. Underlining its commitment, the company indicated its intention to diversify its line of sorghum-based products in Haiti. Stakeholders outside BRANA and SMASH affirmed their confidence in BRANA’s commitment. One MARNDR official remarked, “SMASH is a positive development. After all these years talking about the importance of agriculture, there has been very little investment by the private sector in the food value chain… I think if BRANA has started it, they will find a way to sustain the services to farmers.” BRANA plans future collaboration with CHIBAS on seed variety selection and testing. Earlier evaluation findings show that introducing new sorghum seed varieties was a critical factor to strengthening the sorghum value chain, which up to the time of the evaluation was severely menaced by pests and recurrent droughts and floods. EQ1 findings highlighted that SMASH collaborated with CHIBAS (Quisqueya University), the only national research institution involved in selecting and testing improved sorghum seed varieties in Haiti. At the time of the evaluation, the Alliance was providing follow-on funding to continue new seed research and testing. According to AMC and other involved stakeholders, in the next phase of SMASH, BRANA plans to strengthen its partnership with Quisqueya University’s seed research programs in order to contribute more effectively and consistently to the development of new seed varieties adapted to the Haitian context. BRANA affirmed that it is considering funding research positions at the University to reinforce its capacity in this area. “There is an almost total absence of agricultural research at the MARNDR level due to weak financial and human resources. The Ministry’s regional research centers that could do research and produce seeds are not functional.” MARNDR KII SMASH filled value chain gaps for seed multiplication and distribution but with limitations and concerns for future sustainability. According to SMASH personnel and AMC KIIs, currently SMASH supplies improved seed varieties to SMASH suppliers for free. Based on other agricultural experts and document review, customarily smallholder farmers do not spend their own resources to buy improved seed varieties, which results in a limitation on introducing better seed in the absence of subsidized distribution programs. SMASH findings showed it worked with NGOs involved in seed variety multiplication in Haiti. According to KIIs, historically these organizations mainly focused on beans and corn, but with support from SMASH, they also produced sorghum seed. USAID.GOV SMASH PERFORMANCE EVALUATION | 34 According to SMASH personnel KIIs, the Activity does not yet have a sustainability plan for how seed multiplication and distribution will be handled in its absence. One scenario put forth by a SMASH manager was that farmers, whose incomes it hoped would have improved as a result of using improved seed varieties, would purchase the seed themselves from local operators, if necessary. Several key informants interviewed by the ET indicated that there was unfulfilled demand from farmers for Papèpichon, suggesting some farmers might purchase the seed if it was available for sale. SMASH field agents played important double roles providing extension services and as purchasing agents, roles for which no post-SMASH substitute has been identified. Based on document review and ET interviews with agriculture experts (Government, NGOs), SMASH filled a gap in extension services in the sorghum value chain. These experts noted that government extension services are limited and under-resourced. At the time of the evaluation, other extension services were provided by community-based NGOs in some communities depending on funding and volunteer availability. In FGDs, many farmers said SMASH agronomist support was useful to them to implement improved farming techniques. According to SMASH personnel KIIs, currently SMASH employs eight agronomists (three males, five females) and 14 farmer liaisons (all male) to cover its current production zones. The largest number of agronomists are in the South (the region also hosting the largest number of SMASH smallholder suppliers). Based on SMASH reporting, the role of the SMASH agronomists evolved over time to include both extension support and purchasing. At the time of the evaluation, their principle tasks included identifying potential SMASH suppliers, raising awareness regarding SMASH sales conditions (for example what services BRANA offers, what is required to benefit from these services), signing sales contracts, providing extension services, and facilitating purchasing and sales logistics at harvest time. The role of the farmer liaisons, who live in the communities, was to assist the agronomist with his/her tasks while keeping a close eye on sorghum production, how much was being produced by which farmers, and to whom it was being sold. Both agronomists and farmer liaisons received a salary from SMASH. Based on KIIs with SMASH personnel, SMASH has not yet developed a strategy for how to sustain the role of SMASH agronomists and farmer liaisons after the SMASH Activity ends. According to AMC KIIs, BRANA currently does not intend to employ a similarly large number of agronomists to provide extension services to farmers. According to one key informant, BRANA intends to employ two agronomists to manage local sourcing and provide support to large producers. EQ1 findings show that although SMASH trained nearly 400 agronomists from other organizations, it does not work closely with these organizations in the field to provide extension services to farmers. A partial solution to sustain extension services provided by SMASH agronomists, based on AMC descriptions of SMASH phase two, includes plans by BRANA for collaboration with Haitian universities on demonstration plots to which it will bring smallholder farmers to observe good cultivation practices. SMASH mobile was not yet used extensively by EDN and not at all by BRANA. As highlighted in EQ2 findings, SMASH agronomists are the principle users of SMASH mobile. Based on KIIs, EDN and BRANA had not used SMASH mobile in their sorghum procurement operations as of April 2019. Based on SMASH personnel KIIs, information from SMASH mobile is used to communicate shipment information to EDN but at the time of the evaluation, the enterprise mainly relied on its own systems to manage transportation logistics. Based on KIIs, EDN and BRANA did not directly capitalize on the SMASH mobile supplier database or reporting capabilities to forecast sorghum production or actively use it to manage their supply chain operations outside SMASH. SMASH pre-production financing built local mechanization service providers’ capacity to a limited extent. 35 | SMASH FINAL EVALUATION REPORT USAID.GOV EQ1 findings showed that SMASH provided pre-production financing and post-harvest support, which enabled over a hundred smallholder farmers to access mechanized soil preparation, harrowing, and threshing. According to SMASH personnel KIIs, SMASH has purchased some of its own heavy equipment but does not yet have a plan for handing these over to another entity after the SMASH Activity ends. Based on its own and farmer accounts, SMASH also worked through local service providers for mechanization. “One of the weak points we have encountered during the past two planting seasons is the low availability of heavy agricultural equipment in the SMASH regions. The little equipment that’s available is often badly maintained and breaks down often during land preparation causing delays and sometimes missing the season.” SMASH PROJECT STATUS REPORT JULY￾DECEMBER 2017 Based on SMASH reporting, lack of tractor maintenance and repair capacity may be a significant barrier to sustaining some mechanization services in SMASH communities in the future, though SMASH has started to address this problem. Based on KIIs with SMASH personnel and progress reports, SMASH has three times provided tractor repair services by bringing in an international expert from Canada. The international expert trained Haitian mechanics during the second and third missions. In addition to equipment maintenance, one key informant also cited the lack of skilled tractor operators as a barrier to effective mechanization. Last year, SMASH also offered limited training to tractor operators on good practices, according to one key informant. According to SMASH personnel KIIs, SMASH has not yet developed a strategy for how to sustain pre￾production financing services after the SMASH Activity ends. One key informant indicated that over time, SMASH hopes that farmers will build up savings that will enable them to pay for their own pre￾production services procured through local operators such as those with which SMASH has worked. “I do not see local production increasing in the near term. We are stuck with poor production for now, but EDN will not close.” OTHER SUPPLY CHAIN ACTOR “The value chain works when there is a steady supply. It cannot work when there is a drought or an insect infestation and no grain is produced…. We solved this with grain importation.” OTHER SUPPLY CHAIN ACTOR EDN filled a significant gap in the sorghum value chain, making large, commercial sorghum sales feasible. EQ 2 findings indicated that before SMASH, there was no central conditioning center for sorghum and no local storage centers to consolidate grain destined for commercial markets. Based on Activity reporting and KIIs, through a competitive bidding process, SMASH and the USAID/HAITI LEVE program identified an entrepreneur and provided grants and technical assistance to establish the central conditioning center that became EDN. The role of EDN in the value chain is to receive, test, clean, sort, and package sorghum. It sells grain to BRANA and to other clients for other uses. Through LEVE, EDN received technical assistance to train the center operators who, according to EDN management and other KIIs, are currently able to run all the machinery and train their replacements, if necessary. “With EDN we have set up an effective way to locally clean, dry and store [sorghum]. With the micro collection centers, we connected farms to BRANA through EDN.” AMC KII USAID.GOV SMASH PERFORMANCE EVALUATION | 36 BRANA and EDN are committed to EDN’s business viability. According to AMC KIIs, BRANA is part owner of EDN and subsidizes its operations. BRANA took this step because the central conditioning center was a critical link in the brewery’s locally produced sorghum supply chain. One key informant indicated that BRANA plans to remain part owner but would like to reduce its share of ownership if and when EDN’s financial outlook has improved. Based on document review and SMASH and EDN KIIs, at the time of the evaluation, SMASH had not been able to purchase enough sorghum for EDN to work on a profitable scale. Although this was not a project objective, the former has had implications on EDN’s business model. The owner of EDN has a license to import grain and sorghum to keep its business running when quantities of locally produced sorghum are insufficient. 45 At the time of the evaluation, EDN had imported sorghum twice and was planning a third importation. According to KIIs, EDN, like BRANA, also planned to establish its own sorghum farm in the future to ensure a basic supply of locally grown sorghum. SMASH did not build EDN capacity to take over as BRANA’s purchasing agent. Based on EDN and SMASH management, SMASH—and not EDN—was charged with establishing supplier networks and purchasing from them for BRANA during the first phase of the Activity. At one point during SMASH implementation, BRANA asked EDN to cease purchasing sorghum in communities where SMASH worked. According to other value chain key informants, because of this, when the SMASH Activity ends, EDN will have to establish a purchasing agent relationship with SMASH suppliers if no other enterprise or organization assumes this role. According to the several KIIs, at the time of the evaluation, EDN was establishing a network of suppliers in regions where SMASH does not work (but currently did not offer the same pre- and post-harvest services that SMASH offers). One AMC key informant thought SMASH should have done more to build EDN’s capacity to source sorghum, stating that “There has been very little sustainable capacity building. They could have been strengthening EDN more.” During SMASH, farmers’ production costs increased while yields largely remained low, limiting improvements in farmer income. Lack of reliable baseline and other data prevented the ET having a full picture of how SMASH-affiliated farmers’ income from sorghum changed after SMASH started. In FGDs, farmers in all regions affirmed that their income from sorghum production was not steady mainly because of labor costs and yield variations from one season to another. Based on SMASH M&E data, farmers’ revenue from sorghum compared to the baseline decreased in years two and three, remained unchanged in year four, and increased by 112 percent and 78 percent, respectively, in years five and six. Based on the evaluation survey data (n=76), 43 percent of surveyed farmers reported positive income from sorghum production. SMASH-affiliated farmers’ income from sorghum averaged about 4,800 gourdes in 2018, representing about eight percent of their total income. However, the average benefit from producing sorghum largely exceeded the average loss, resulting in a net increase in income from sorghum production. The net income from sorghum production averaged 29,154 gourdes among those who had a gain in income and negative 13,882 gourdes for those who incurred a loss. The level of income depends on the costs of production and the yield. The cost of production also depends on the size of the plot in which the sorghum is produced. 45 According to AMC KII, SMASH supported the first importation of sorghum by EDN so that EDN could test newly installed equipment and safety and operational protocols, keep the mill running at scale during a period of sorghum shortages, and supply BRANA so that production could continue without altering recipes 37 | SMASH FINAL EVALUATION REPORT USAID.GOV “When we do a real calculation, we realize that despite the increase in yield, we do not make real profits. The money comes in all at once but does not mean much in terms of profits because the expenses are too high.” FGD SOUTH “Spending on sorghum production has increased in recent years. This is due to the various stages of soil preparation- now we are forced to plow the land, harrow and do ridging. Before we could do a single weeding but with these new seeds, we must weed several times. And the wages given to farm workers has also increased.” FGD CENTER Based on survey data, the ET estimated the cost of producing sorghum on one ha at about 14,000 gourdes. In terms of comparison, a study conducted in the Northwest in 2018 indicated that the cost of producing sorghum was 6,000 gourdes per ha, less than half the cost that SMASH-affiliated farmers incurred. Secondary literature reviews showed that small farmers producing sorghum traditionally minimized production costs by limiting the number of operations, using family labor and basic inputs, and by associating sorghum with other more valuable crops. According to FGDs, organizing farmers in production blocks helped with the mechanization of the soil preparation. While it increased plowing efficiency, it did not reduce production costs. Farmers highlighted that the new techniques increased demand for labor since farmers were encouraged to apply all the recommended practices (weeding, planting fewer seeds) to synchronize their operations within the block. Both farmers and SMASH managers highlighted the rise in farm labor wages (50-100 percent in recent years) and the cost of plowing services (which nearly doubled in the last two years), in part because of high demand in SMASH intervention zones. Yield was another factor that affected farmers’ income from sorghum based on survey data. EQ1 findings show that sorghum yield was low even for SMASH-affiliated farmers who received technical and financial assistance. Because of pests and weather conditions, the varieties promoted by SMASH did not reach their potential. Given the high cost of producing sorghum highlighted above, low yields reduced the benefits farmers obtained from growing sorghum. One large-scale farmer KIIs indicated that with a yield of less than one MT/ha, sorghum production was far from profitable at the price offered by BRANA. This farmer indicated that he will continue sorghum production if yields increase to two MT/ha or the price goes up. Despite sorghum’s low profitability, 59 percent of surveyed farmers declared that the sorghum’s contribution to their income has increased since their affiliation with SMASH. CONCLUSIONS BRANA’S DEMAND FOR HIGH QUALITY BULK SORGHUM DIRECTLY AND INDIRECTLY CONTRIBUTED TO POSITIVE CHANGES IN THE VALUE CHAIN. Evaluation findings show SMASH contributed to opening a new, formal sector market for bulk sorghum, strengthened existing work on improved seed varieties, engaged NGOs in improved seed multiplication, modestly strengthened local agricultural service providers, and created and subsidized a grain central processing center. BRANA’S LONG-TERM COMMITMENT TO SERVING THE HAITIAN MARKET AND TO SOURCING SORGHUM LOCALLY IS A POSITIVE FACTOR FAVORING SUSTAINABILITY. Although greatly reduced in scale (200-350 small-scale farmers), the company indicated it still intends to include small-scale farmers in its supply chain. Similarly, the existence of EDN as well as BRANA’s part ownership of the company is a positive sustainability factor. As private sector entities with a vested interest in promoting a dynamic value chain for locally grown sorghum, these businesses have shared interests with sorghum farmers: finding improved seed varieties to raise yields, increasing numbers of USAID.GOV SMASH PERFORMANCE EVALUATION | 38 farms using improved farming techniques, and improving post-harvest logistics. Because of these shared interests, continuation of BRANA’s local sourcing policy would likely be beneficial for sorghum farmers, both those who sell to BRANA and others who may benefit indirectly from the development of improved seed varieties and additional buyers in the market. SMASH CONTRIBUTIONS TO IMPROVING FARMER INCOME FROM SORGHUM VARIED FROM SEASON TO SEASON AND WERE LIMITED OVERALL. The evaluation’s short survey (n=76) showed that 43 percent of surveyed farmers reported positive income from sorghum production. The average gain reported was higher than the average loss. However, income from sorghum made a relatively small contribution (8 percent) to farmers overall income. Among factors that affected the contribution of sorghum production to farmers’ income were high production costs and low productivity. Lack of reliable baseline data prevent the ET from correctly assessing the extent to which farmers’ income from sorghum has changed since the SMASH Activity started. However, farmers in FGDs indicated that they have experienced an increase in income from sorghum production at least once since their involvement with SMASH. Income from sorghum production varies from one year to another, making its contribution to farmers’ overall income unstable. SMASH HAD NOT DEVELOPED OR TESTED A CLEAR SUSTAINABILITY STRATEGY FOR MANY OF ITS SERVICES AT THE TIME OF THE EVALUATION. SMASH’s over-reliance on a project-based service delivery model (services provided directly by individuals employed by Papyrus and paid for by subsidies) instead of more sustainable, market-driven implementation modalities was the main factor limiting the sustainability of services. RECOMMENDATIONS RECOMMENDATIONS FOR USAID 1. FUND A SET OF INTEGRATED INVENTIONS AIMED AT BOOSTING SORGHUM PRODUCTION ACROSS HAITI. • Subsidize multiplication and distribution of improved seed varieties in all sorghum growing regions (once the seed varieties are field-tested and approved). • Sponsor farmer-targeted communication campaign on effective measures to fight aphids, which may accompany improved seed distribution activities and include information on crop rotation and safe insecticide use including developing integrated pest management. • Focus on areas where sorghum was more prevalent before the aphid infestation. To implement this recommendation, research is needed to breed a seed variety that is both aphid and drought resistant. 2. ASSUMING THE ADOPTION OF EFFECTIVE SOLUTIONS TO EXTERNAL FACTORS CURRENTLY LIMITING SORGHUM PRODUCTION (ESPECIALLY BETTER SEEDS), SUPPORT DEVELOPMENT OF OTHER MARKETS FOR BULK SORGHUM. • Support market development for bulk sorghum with lower quality standards than BRANA to lower smallholder barriers to entry (school feeding programs, livestock production). RECOMMENDATIONS FOR ALLIANCE MANAGEMENT COMMITTEE INSTITUTIONS (BRANA, IDB, PAPYRUS, USAID/HAITI) 39 | SMASH FINAL EVALUATION REPORT USAID.GOV 3. CONTINUE ACTIVITIES TO INCENTIVIZE SMALL-SCALE FARMERS TO ORGANIZE IN BLOCKS AND IMPROVE THEIR FARMING PRACTICES USING ACCESS TO BRANA’S MARKET AS A CARROT. • Intensify investment in the development and testing of aphid and drought tolerant sorghum seed varieties not only with Quisqueya University which has accumulated substantial experiences in this area over the last six to seven years, but also other public and private institutions in Haiti with seed research capacity. • Adapt SMASH mobile to EDN needs and support EDN to deploy. Consider feasibility of automating extension messages (i.e. time to weed) by interactive voice recording. • Support a small assessment on farmer attitudes and constraints to establishing production blocks to inform future strategies to manage change. • Support creation of decentralized drying and storage facilities closer to farms to improve quality of sorghum supplied by smallholders. • Develop targeted strategies to increase participation of youth in farming. • Pilot an initiative to integrate Madame Sara in BRANA’s supply chain. • Capture sound baseline data at the beginning of Activities and implement a strong M&E system to develop adequate analytical tools and better inform decision making process in the value chain. For the implementation of these strategies, strengthen partnerships with universities, farmers’ associations, and grass roots organizations involved in rural and agricultural development programs. RECOMMENDATIONS FOR BRANA 4. TAKE STEPS TO PROMOTE SUSTAINABILITY IN LOCAL SOURCING STRATEGY. • Review price setting strategy to align more realistically with current yields and market conditions based on regular and comprehensive assessments of production costs. • Transition purchasing agent role from SMASH to EDN or other market-oriented intermediaries. During transition, document and share good practices developed, such as use of farmer liaisons as local support to purchasing agents. • Transition pre-production financing to local operators, farmers’ associations and/or grass roots organizations, and reinforce their capacity to maintain equipment and manage services. • Transition extension services to universities, vocational training centers, government extension services (where these exist), and/or grassroots organizations involved in rural and agricultural development programs. USAID.GOV SMASH PERFORMANCE EVALUATION | 40 ANNEX A: EVALUATION STATEMENT OF WORK STATEMENT OF WORK Performance Evaluation OF Smallholder Alliance for Sorghum in Haiti (SMASH) I. PURPOSE OF THE EVALUATION The purpose of this evaluation is to inform USAID/Haiti’s future similar agricultural activity in Haiti. Lessons learned and recommendations provided through the final report at the end of the evaluation process should guide our next generation of partnership within the sorghum value chain. The primary stakeholders for this evaluation include: USAID/Haiti, BRANA, Inter-American Development Bank (IDB), PAPYRUS which participated in the implementation of the project, and the Government of Haiti. II. SUMMARY INFORMATION Strategy/Project/Activity Name Smallholder Alliance for Sorghum in Haiti (SMASH) Implementer Brasserie Nationale d’Haïti S.A (BRANA) Cooperative Agreement/Contract # AID-521-A-14-CA0001 Total Estimated Ceiling of the Evaluated Project/Activity(TEC) $1,712,760 ,for USAID contribution, the total amount for the agreement was $5,158,330 Life of Strategy, Project, or Activity June 2014-June 2019 Active Geographic Regions N/A Development Objective(s) (DOs) DO2 : Increased food and economic security USAID Office Economic Growth and Agricultural Development III. BACKGROUND A. DESCRIPTION OF THE PROBLEM, DEVELOPMENT HYPOTHESIS (ES), AND THEORY OF CHANGE The goal of the SMASH program is to improve the incomes of up to 18,000 Haitian subsistence farmers through increased incomes and production of locally-grown sorghum in place of imported malted barley and other grainswheat in the production of BRANA’s product, Malta H, a non-alcoholic nutritional beverage. SMASH leverages a combination of entities to resolve constraints in the sorghum value chain in Haiti. This partnership combines the comparatives advantage of BRANA, a subsidiary of HEINEKEN USAID, and IDB to strengthen the sorghum value chain and leverage USAID’s existing work in Haiti through coordination with the other projects. This agreement provided a unique opportunity for local 41 | SMASH FINAL EVALUATION REPORT USAID.GOV farmers to sell their products and improve their agriculture production techniques through the education provided by the SMASH program. Over six years, SMASH objectives were to: • Increase smallholder farmers’ sorghum production by at least 100% by the end of the project • Increase targeted smallholder farmer income from sorghum by 75% • Organize the supply chain such that smallholder farmers are able to supply commercial buyers such as BRANA with desired quantities of sorghum at internal quality standards. Sorghum is the third most important cereal in Haiti behind corn and rice. Although precise statistics are not available, it has been estimated that annual sorghum production includes ~120,000 hectares involving more than 300,000 farmers across the whole country. Sorghum is multipurpose crop in the subsistence agricultural system of Haiti; where the grains are used for human and animal consumption, while the stalks are employed to feed animals after harvesting. Sweet sorghum varieties are also used by other farmers for grain production, while the juicy sweet stalks are pressed for the production of alcoholic beverage and syrup. The versatility of use and production of sorghum, which includes drought tolerance, high grain yields under low inputs conditions, animal feed, production of malts and alcoholic beverage, positioned it as promising crop for Haiti’s agriculture. Nevertheless, since 2015 sorghum production in Haiti has been significantly reduced by the presence of the sugarcane aphids. B. SUMMARY STRATEGY/PROJECT/ACTIVITY/INTERVENTION TO BE EVALUATED Through a market-driven approach, SMASH has contributed to increase income generating opportunities for up to 18,000 farmer households by raising yields through testing and by introducing new varieties, improving production, storing and transport practices, organizing the supply chain , enhancing skills through training, increasing sales, reducing losses, mitigating risk and reducing cultivation and financing costs. SMASH also includes activities that contribute to increase linkages between smallholder farmers and markets for sorghum that result in the gain of significant market efficiencies at all points in the chain. Value-added activities, improved agricultural techniques, increased long-term access to inputs and the introduction of better sorghum varieties stimulate rural economies through inclusive productivity growth degradation without degradation of natural resources. In addition, an open-source, evidence-based mobile application SMASH Mobil, was collaboratively developed with Dimagi, Inc. under the program to support sorghum supplier management, purchasing and inventory operations, supply chain logistics, product traceability and quality control, and monitoring and evaluation data collection, reporting, and analytics. Created specifically for low-resource settings, this web-based interface platform and cloud-hosted data system also features expanded end-user app design and launch capabilities for future customized mobile applications. Following are some key results of the SMASH activity; • 18,242 farmers and 294 agronomists and extension agents trained in improved climate-smart sorghum production techniques. • 97% increase in average sorghum yield per hectare prior to the devastating sugar cane aphid infestation. Current production is returning to more typical long term yields with the potential for significant gains with improved varieties and practices. • 162 sorghum farmers received pre-financing in the form of mechanized services or operating capital through a pilot initiative begun in 2017, with close to a 100% payback rate through grain reimbursement and final commodity sales payments effected via UNITRANSFER, a mobile commercial banking service. USAID.GOV SMASH PERFORMANCE EVALUATION | 42 • More than 4,000 sorghum growers sold grain to BRANA, with this past year’s purchase of 290 MT meeting the company’s annual target for locally-sourced raw materials for the first time. • 2,730 MT of sorghum processed at the EDN mill, 70% of which supported commercial markets outside BRANA’s core business, including grain wholesalers, food transformation industries, animal feed producers, and international donor food security and nutrition programs • C. SUMMARY OF THE PROJECT/ACTIVITY MONITORING, EVALUATION, AND LEARNING (MEL) PLAN SMASH has defined a set of performance indicators including a mix of outcome and output indicators to monitor project progress. Output indicators track the immediate products of project activities and provide feedback to managers on project performance to identify areas where implementation strategies may need to be adjusted. Outcome indicators measure the effects, or results, of project activities, at the higher levels of the project results framework. Indicators were selected based on the overall strategic approach of the project and closely reflect the project work plan, the Performance Monitoring Matrix and the IDB results matrix. IV. EVALUATION QUESTIONS 1. To what extent and in what ways have SMASH’s interventions been effective in achieving key results, especially increasing farmers’ sorghum production and sales? The evaluation should highlight gaps and lessons learned. 2. To what extent did the project improve market access for small-scale farmers and in what ways has the SMASH program and other external factors affected the sorghum grain market in Haiti? The evaluation should consider supply, demand and price in its market analysis. 3. To what extent and in what ways has the SMASH program been successful in strengthening the sorghum value chain in Haiti? The evaluation should include farmers’ income and sustainability in its analysis of the sorghum supply chain strengthening. V. EVALUATION DESIGN AND METHODOLOGY The methodological approach shall include a combination of qualitative and quantitative methods, with data collection from both primary and secondary sources. As part of the evaluation design document, SI shall propose for USAID’s review, a detailed methodological approach to be used to address evaluation questions. This methodology will specify the research design, as well as methods and procedures for sampling, data collection and data analysis. Efforts should be made to use multiple data collection methods and data sources, complementing literature review, interviews, and discussions to allow for triangulation of data and cross-validation of results. The evaluation matrix below provides a summary of the suggested data collection methods by evaluation questions. Questions Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods 1. To what extent and in what ways have SMASH’s interventions been effective in achieving key results, especially increasing farmers’ sorghum production and sales? Documents including performance monitoring data, expert knowledge, beneficiaries, Project staff… Key informant interviews, survey of participants, desk review, focus group discussions…… To be determined by evaluation team 43 | SMASH FINAL EVALUATION REPORT USAID.GOV Questions Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods 2. To what extent and in what ways has the SMASH program and other external factors affected the sorghum grain market in Haiti? Documents including performance monitoring data, expert knowledge, beneficiaries, Project staff… Key informant interviews, focus group discussions, desk review…… To be determined by evaluation team 3. To what extent and in what ways has the SMASH program been successful in strengthening the sorghum value chain in Haiti? Documents including performance monitoring data, expert knowledge, beneficiaries, Project staff… Key informant interviews, focus group discussions, survey of par8itcipants…… To be determined by evaluation team VI. DELIVERABLES AND REPORTING REQUIREMENTS 1. Evaluation Work plan: Upon receipt of this Activity Request, Social Impact (SI) shall submit within two weeks a draft work plan to the Contracting Officer’s Representative (COR). The work plan will include: (1) the anticipated schedule and logistical arrangements; (2) a list of the members of the evaluation team, delineated by roles and responsibilities with their level of effort; (3) the identification of other required personnel and relevant local subcontractors, their LOE, roles and responsibilities and qualifications; and (4) the deliverable schedule. 2. Evaluation Design: Within two weeks of approval of the work plan, SI must submit to Contracting Officer’s Representative (COR) an evaluation design (which will become an annex to the Evaluation report). The evaluation design will include: (1) a detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; (2) draft data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited; (4) known limitations to the evaluation design; and (5) a dissemination plan. USAID/Haiti will take up to 10 business days to review and consolidate comments through the COR. Once the evaluation team receives the consolidated comments on the initial evaluation design and work plan, they are expected to return with a revised evaluation design and work plan within 5 business days. 3. In-briefing / inception report: Prior undertaking field work, the evaluation team will have an in￾briefing with the USAID/Governance Team and the Evaluation and Survey Services (ESS) COR to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and to clarify any questions or logistic needs. 4. Evaluation Briefing/Presentation: The evaluation team is expected to hold a final presentation in person to discuss the summary of findings and recommendations to USAID within 20 business days after the conclusion of fieldwork. 5. Draft Evaluation Report: The draft evaluation report should be consistent with the guidance provided in Section IX: Final Report Format. The report will address each of the questions USAID.GOV SMASH PERFORMANCE EVALUATION | 44 identified in the SOW and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, the Governance Office will have 10 working business days in which to review and comment on the initial draft, after which point the ESS COR will submit the consolidated comments to the evaluation team. The evaluation team will then be asked to submit revised final draft report 5 business days hence, and again the Governance Office will review and send comments on this final draft report within 5 business days of its submission. 6. Final Evaluation Report: The evaluation team will be asked to take no more than 15 business days to respond/incorporate the final comments from the Governance Office. The evaluation team leader will then submit the final report to the COR. All project data and records (FGD and KII summary reports) shall be submitted in full and should be in electronic form in easily readable format, organized, and documented for use by those not fully familiar with the intervention or evaluation, and owned by USAID. VII. EVALUATION TEAM COMPOSITION The Evaluation Team shall be comprised of two Key Personnel positions: (i) a Team Leader, and (ii) an Assistant Team Leader. SI is strongly encouraged to sub-partner with a local Haitian firm for data collection purposes. The selected Haitian firms should demonstrate proven capacity in collecting data in the relevant fields (quantitative and qualitative data collection) and should have data analysis capabilities to perform all related data management functions. This approach is encouraged to build the local firm capacity and will also provide a Haitian perspective for the data collection and analysis. The Team Leader (TL) is ultimately responsible for the overall management of the evaluation team, coordinating the implementation of the evaluation, assigning evaluation responsibilities and tasks, and authoring the final evaluation report in conformity with this Statement of Work. The TL must be an experienced evaluation expert, with a documented track record of 10 years of experience in the field of evaluation. S/he should be fluent in French and English. S/he should have at least a Degree in Economics, Public Administration, International Relation or a related field. The Assistant Team Leader (ATL) helps the TL in the overall management of the evaluation team and the final products, in conformity with this Statement of Work. The ATL should be familiar with the Agriculture sector. S/he must possess excellent writing and interpersonal skills and must be familiar with USAID programs, objectives, and reporting requirements. S/he should have experience in designing and implementing evaluations in the agriculture sector and in conducting FGDs. Fluency in French is required. English and Haitian Creole are highly desirable, as is significant prior work experience in Haiti. A Master’s degree in Agricultural Economics, Political Science, Public Administration, or a related field is required to ensure that all areas of technical expertise required for the evaluation are effectively covered. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. The evaluation team shall demonstrate familiarity with USAID’s evaluation policies and guidance included in the USAID Automated Directive System (ADS) in Chapter 200. 45 | SMASH FINAL EVALUATION REPORT USAID.GOV VIII. EVALUATION SCHEDULE AND ESTIMATED LOE a) Schedule Timing (Anticipated Weeks or Duration) Scheduled Activities 4 Weeks Preparation of the work plan and evaluation design 3 Weeks USAID Approval of the Work Plan and Inception Report 1 week Within 5 business days of the inception report 4 Weeks Data collection 2 Weeks Data analysis and Evaluation Briefing with USAID, BRANA, PAPYRUS and IDB 2 Weeks Draft Report writing 2 Weeks USAID, IDB, BRANA and PAPYRUSreview of Draft Report 1 Week Incorporate USAID comments and prepare Final Report b) Estimated LOE in days by activity for a team of two Task LOE for Team Lead (TL) LOE for Assistant Team Lead (ATL) Total LOE in days Document review/desk review/work planning and evaluation design drafting 20 20 40 In-brief, evaluation design finalization (including meetings with USAID, BRANA, IDB and PAPYRUS) 1 1 2 Data collection 15 15 30 Data analysis 10 10 20 Out-Brief 1 1 2 Draft report 10 10 20 Final report 5 5 10 Totals 62 62 124 IX. FINAL REPORT FORMAT USAID.GOV SMASH PERFORMANCE EVALUATION | 46 The evaluation final report should include an abstract; executive summary; background of the local context and the strategies/projects/activities being evaluated; the evaluation purpose and main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations. For more detail, see “How-To Note: Preparing Evaluation Reports” and ADS 201mah, USAID Evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. The executive summary should be 2–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: • The Evaluation SOW; • Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; • All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; • All sources of information, properly identified and listed; and • Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of. • Any “statements of difference” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team. • Summary information about evaluation team members, including qualifications, experience, and role on the team. In accordance with ADS 201, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within three months of the evaluation’s conclusion. X. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report.46 46 See ADS 201mah, USAID Evaluation Report Requirements and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. 47 | SMASH FINAL EVALUATION REPORT USAID.GOV • Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. • Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. • The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. • Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. • Evaluation methodology should be explained in detail and sources of information properly identified. • Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. • Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. • If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. USAID.GOV SMASH PERFORMANCE EVALUATION | 48 ANNEX B. EVALUATION PROTOCOLS Informed Consent Agreement - KII Purpose: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called Social Impact, a company that is based in the United States. Our team is in Haiti to conduct a study about the Smallholder Alliance for Sorghum in Haiti (SMASH). SMASH is a public-private partnership (PPP) between USAID, the Inter-American Development Bank (IDB), and Brasserie Nationale d’Haïti (BRANA S.A.). BRANA launched SMASH in July 2013, with the objective of replacing imported malt with locally grown sorghum in the production of its Malta H. The goal of SMASH is to improve the livelihoods of up to 18,000 Haitian smallholder farmers through a market￾driven, value chain approach to increase yields of high-quality, locally grown sorghum. You have been asked to participate today so that we can learn more about your involvement in the SMASH program/learn from your knowledge and expertise on agricultural value chains in Haiti. We are meeting with over 200 individuals (SMASH managers, agronomists, farmers, and other value chain actors) who participated in the program either as implementers, recipients of program services and/or as independent experts. We would like your honest impressions, opinions and thoughts about various issues related to this activity’s implementation and outcomes. We are independent consultants who have no affiliation with those who implemented SMASH nor do we represent the GOH. Procedures: If you agree to participate, we ask you to discuss your experience and opinion of the activities and services implemented under the SMASH program. The interview will take about 1 hour of your time. Although we will publish our findings in a public report, all of your answers will be kept confidential. Nothing you tell us will be attributed to any individual person. Rather the report will include only a composite of all of the answers received by all of the individuals we interview. Although we may use quotes, none of the individuals interviewed will be named in the report. Risks/Benefits: There are no significant risks to your participation in this study. You will not receive any direct benefit or compensation for participating in this study. Although this study will not benefit you personally, we hope that our results will help improve potential future programs to support agricultural value chains in Haiti. Voluntary Participation: Participation in this interview is completely voluntary. You do not have to agree to be in this study. You are free to end the interview at any time or to decline to answer any question which you do not wish to answer. If you decline to participate in the interview, no one will be informed of this. Do you have any questions at this time? [Interviewer should answer any questions] Permission to Proceed I understand the purpose of the interview as outlined above and understand that I can withdraw from the interview at any time and for any reason. I agree to participate in the interview (Evaluator records). Yes No Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ 49 | SMASH FINAL EVALUATION REPORT USAID.GOV Informed Consent Agreement - FGD Purpose: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called Social Impact, a company that is based in the United States. Our team is in Haiti to conduct a study about the Smallholder Alliance for Sorghum in Haiti (SMASH). SMASH is a public-private partnership (PPP) between USAID, the Inter-American Development Bank (IDB), and Brasserie Nationale d’Haïti (BRANA S.A.). BRANA launched SMASH in July 2013, with the objective of replacing imported malt with locally grown sorghum in the production of its Malta H. The goal of SMASH is to improve the livelihoods of up to 18,000 Haitian smallholder farmers through a market￾driven, value chain approach to increase yields of high-quality, locally grown sorghum. You have been asked to participate today so that we can learn more about your involvement in the SMASH program/learn from your knowledge and expertise on agricultural value chains in Haiti. We are meeting with over 200 individuals (SMASH managers, agronomists, farmers, and other value chain actors) who participated in the program either as implementers, recipients of program services and/or as independent experts. You have been invited to participate in this discussion because you participated in at least one SMASH activity or because you farm sorghum in a region serviced by the SMASH Alliance. We would like your opinions and thoughts about SMASH services and activities. We are independent consultants who have no affiliation with SMASH or the GOH. Procedures: If you agree to participate, we ask you to discuss your experience and opinion of the activities and services implemented under the KONEKTE program. The FGD will take up to 2 hours of your time. Although we will publish our findings in a public report, all of your answers will be kept confidential. Nothing you tell us will be attributed to any individual person. Although we may use quotes, none of the individuals interviewed will be named in the report. However, as this is a group setting, to preserve confidentiality, we ask you not to share anything we discuss here today with anyone outside of this group. Risks/Benefits: There are no significant risks to your participation in this study. You will not receive any direct benefit or compensation for participating in this study. Although this study will not benefit you personally, we hope that our results will help improve potential future institutional capacity building programs for government institutions in Haiti. Voluntary Participation: Participation in this interview is completely voluntary. You do not have to agree to be in this study. You are free to leave the FGD at any time or to decline to answer any question which you do not wish to answer. If you decline to participate, no one will be informed of this. Do you have any questions at this time? [Interviewer should answer any questions] Permission to Proceed I understand the purpose of the interview as outlined above and understand that I can withdraw from the interview at any time and for any reason. I agree to participate in the interview (Evaluator records). Yes No Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ USAID.GOV SMASH PERFORMANCE EVALUATION | 50 Protocol for KII with USAID COR, USAID Technical Office and AMC Representatives A. Demographic Info: 1. Date: 2. Interview location: 3. Interviewer Name: 4. Primary Notetaker Name: 5. Respondent Name: 6. Respondent Title (during life of Activity): 7. Respondent Institution (during life of Activity): 8. Sex of respondent: B. KII questions for USAID 1. In which result area (increasing smallholder farmer yields, increasing smallholder farmer incomes, organizing the value chain) has SMASH achieved the most? In which result area has it achieved the least? What are examples of achievements? What are examples of shortfalls? (EQ1, EQ2, EQ3) 2. What factors (external or internal) best facilitated SMASH’s achievement of the activity goals and objectives? Which factors (external or internal) most constrained SMASH’s achievement of the activity goals and objectives? (EQ1, EQ2, EQ3) Probe: To what extent did environmental factors or in the general political and economic environment affect SMASH? To what extent did internal factors such as the effectiveness of the implementing organization affect achievement? 3. What unintended outcomes did you see, if any? (Cross Cutting) 4. How far advanced is SMASH’s transition from an externally financed, project￾oriented implementation strategy to a business-financed local procurement department within BRANA? (EQ3) Probe: What are the likely changes that will occur once external support for SMASH ends? What more needs to be done to ensure sustainability? In what areas, if any, do you think additional USAID assistance makes sense? 5. To what extent do you think that SMASH was effectively managed? What were management strengths? Weaknesses? What, if anything, might have been done better? (cross-cutting) Probe: What are the advantages and disadvantages of the PPP implementation modality? How effective was the AMC as a management oversight mechanism? Were the roles and responsibilities for project implementation clearly defined? Were activities carried out in a timely manner? To what extent were SMASH managers able to adapt and find solutions to unexpected challenges? 6. What examples are there (if any) of SMASH coordinating with other USAID or US Government funded programs? With other donor-funded programs? (Cross Cutting) 7. What lessons were learned in the implementation of the SMASH? (Cross Cutting Lessons learned) 51 | SMASH FINAL EVALUATION REPORT USAID.GOV Protocol for KIIs with SMASH Implementation team (including subcontractors) A. Demographic Info: a. Date: b. Interview location: c. Interviewer Name: d. Primary Notetaker Name: e. Respondent Name: f. Respondent Title: g. Respondent Institution: h. Sex of respondent: B. Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ C. KII Protocol for SMASH national implementation team 1. What is your role in SMASH implementation? Over which period did you perform this role? 2. To what extent was SMASH successful increasing smallholder farmer sorghum yields? What factors contributed to its success or lack of success? (EQ1) 3. To what extent was SMASH successful increasing smallholder farmer profits from sorghum? What factors contributed to its success or lack of success? (EQ1) 4. What factors affected the extent farmers did or did not adopt SMASH￾promoted production techniques and new technologies? What, if anything could have been done better by SMASH to foster higher rates of adoption? (EQ1) 5. Did SMASH adjust its intervention strategies to overcome environmental and other external constraints? If so, how? If not, why not? Have changes produced better results? What are some examples? 6. Were there significant differences in SMASH outcomes among some farmers versus others? Probe: Were there significant differences in SMASH outcomes between male vs. female producers. Smallholders vs. larger-scale farmers? Did SMASH propose effective solutions to overcome the factors affecting female producers differently than men? Factors affecting smallholders versus larger-scale farmers? If so, what are some examples? (EQ1) 7. To what extent has SMASH adequately incentivized smallholder (and medium and large-scale) farmers to be part of the Alliance? What are the principle strategies SMASH has proposed? Which strategies have been the most and least effective to date? USAID.GOV SMASH PERFORMANCE EVALUATION | 52 Probe: What non-price related factors affect smallholder farmer decisions about what they plant, in what quantity and who they sell their produce? 8. To what extent have SMASH interventions been effective overcoming smallholder market-access constraints? What are the principle strategies SMASH has proposed? Which strategies have been the most and least effective to date? Why? Probe: Has SMASH’s decision to source from larger farms affected the Alliance’s success contributing to an inclusive value chain? How or how not? 9. What market-related dynamics and or local practices (share cropping, keeping a portion of the crop for home consumption) affect the supply of sorghum produced by smallholder farmers and available for purchase by SMASH/BRANA? (EQ2) 10. To what extent have other market players (such as sorghum purchased for food aid, sorghum for bakery goods, animal feed, etc.) affected the market for locally grown sorghum in Haiti? (EQ2) Probe: Has market competition affected SMASH’s ability to maintain a stable network of smallholder suppliers willing to sell at the price set by BRANA? 11. To what extent has SMASH had any effect on the producers’ share of income/value-added created from recent developments in the sorghum value chain? Probe: What, if any, benefit have recent developments in the sorghum value chain offered to farmers? Have the benefits been equally distributed among small, medium and large-scale farmers? What more needs to be done to improve the benefits to farmers? 12. How effective is EDN as a market intermediary? How sustainable is it as a market intermediary? What are the factors affecting EDN’s effectiveness/sustainability? To what extent have these been addressed and what more needs to be done? (EQ3) Probe: Do you think that EDN is likely to be a profitable business? In what time frame? What are the main issues affecting its profitability? Do you think EDN will continue to buy from smallholder farmers after SMASH concludes? Why or why not? 13. How effective and sustainable is SMASH Mobile as a value chain management tool? What are the factors affecting SMASH Mobile effectiveness/sustainability? To what extent have these been addressed and what more needs to be done? (EQ3) 14. How successful has SMASH been in improving the overall efficiency of the sorghum value chain in Haiti? (EQ3) Probe: Are there significant bottlenecks affecting the efficiency of the chain in addition to those already being addressed by SMASH? 15. To what extent, if at all, do policy/regulatory issues affect sorghum value chain development? (EQ3) Probe: For example, do agricultural policies and macro-economic policies concerning agricultural imports, landownership law and practices, or food safety regulations have a significant effect on the sorghum value chain? How? And how have these policies/regulations changed, if at all, since the inception of SMASH? 53 | SMASH FINAL EVALUATION REPORT USAID.GOV 16. To what extent has SMASH been successfully transitioned from an externally financed, project-oriented implementation strategy to a business-financed local procurement department within BRANA? (EQ3) Probe: How do Alliance stakeholders perceive the transition? What are the likely changes that will occur once external support for SMASH ends? What more needs to be done to ensure sustainability? USAID.GOV SMASH PERFORMANCE EVALUATION | 54 Protocol for KIIs with representatives of other Donors/International Organizations/INGOs A. Demographic Info: 1. Date: 2. Interview location: 3. Interviewer Name: 4. Primary Notetaker Name: 5. Respondent Name: 6. Respondent Title (during life of activity): 7. Respondent Institution (during life of activity): 8. Sex of respondent: B. Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ 1. Please describe the role/mission of your organization and your main activities in Haiti. 2. Can you tell us a little bit about the state of the sorghum value chain in Haiti? How has it changed, if at all, in the last 5 years? What do you think has caused those changes? 3. Please describe your involvement (if any) with SMASH. Please characterize how informed you are about SMASH activities. Probe: Have you participated in any of the activities i.e. training, meeting, workshop, or events organized by SMASH? Did you interact with SMASH personnel during your work in Haiti? 4. What is your perception of the work done by SMASH? (EQ1, EQ2, EQ3)? Probe: Are there any examples of SMASH activities that have been especially relevant to the needs of smallholder sorghum farmers in Haiti (need for training, access to inputs, access to markets)? To improve the organization of the sorghum value chain in Haiti (logistics, information, creation of value-added markets for sorghum)? Are there any examples of SMASH activities that have not been useful? What more needs to be done? 5. To what extent did the work of SMASH coordinate with or complement the work of your organization (cross-cutting)? 6. What were the main factors enabling/constraining improvements in smallholder sorghum producer livelihoods (2013-present)? Enabling/constraining the organization of the sorghum value chain (EQ1, EQ2, EQ3)? 55 | SMASH FINAL EVALUATION REPORT USAID.GOV Protocol for KIIs with Government of Haiti representatives A. Demographic Info: 1. Date: 2. Interview location: 3. Interviewer Name: 4. Primary Notetaker Name: 5. Respondent Name: 6. Respondent Title: 7. Respondent Institution: 8. Sex of respondent: B. Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ 1. How long have you been with [government agency/organization]? What are the roles and responsibilities of your institution with respect to overseeing/regulating/promoting smallholder farmer productivity/agricultural value chain development in Haiti? What are your specific roles and responsibilities within your institution? 2. What have been the main challenges faced by your institution in its efforts to improve sorghum production/increase smallholder farmer incomes/develop agro￾industry value chains in Haiti in recent years? Has there been any progress and/or worsening of the situation in since 2013? If so, what are some examples? To what factors do you attribute these changes? (EQ1, EQ2, EQ3) 3. To what extent, if at all, do policy/regulatory issues affect sorghum value chain development? (EQ3) Probe: For example, what effect do agricultural policies and macro-economic policies concerning agricultural imports, landownership law and practices, or food safety regulations have on the sorghum value chain? 4. Have you been involved in any activities organized by SMASH or collaborated with the program in other ways? Please describe your collaboration. 5. Have SMASH activities (training, creation of collection centers, research, mobile applications, linkages between value chain actors) been effective in increasing smallholder farmer sorghum yields? How or how not? What, in anything, might have been done differently? What more needs to be done? (EQ1) 6. Have SMASH activities (training, creation of collection centers, research, mobile applications, linkages between value chain actors) been effective in meeting the specific needs/challenges of improving smallholder farmer access to (higher value￾added) agricultural markets? How or how not? What, in anything, might have been done differently? What more needs to be done? (EQ2) USAID.GOV SMASH PERFORMANCE EVALUATION | 56 7. Which groups of producers, if any, face additional challenges increasing their productivity and accessing markets? Probe: Do women producers face additional challenges increasing their productivity and accessing markets? If so, how? Have SMASH activities been effective addressing women producer needs/challenges? What, in anything, might have been done differently? What more needs to be done? (EQ1, EQ2, EQ3) 8. Have SMASH activities (training, creation of collection centers, research, mobile applications, linkages between value chain actors) been organizing the value chain? How or how not? What, in anything, might have been done differently? What more needs to be done? (EQ3) 57 | SMASH FINAL EVALUATION REPORT USAID.GOV Protocol for FGD with Farmers A. General Information 1. Date: 2. Number of Participants: 3. Sex of participants: __ male __ female 4. Location of FGD: 5. Interviewer Name: 6. Primary Notetaker Name: B. Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask FGD participants if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ C. Focus group questionnaire for participating farmers 1. What are the difficulties that farmers in your community face in the production of sorghum? (EQ1) 2. Have you received training or other support services to improve your sorghum crops in recent years? If so, what types of support did you receive? Who provided the support? (EQ1) 3. If you participated in training, to what extent have you been able to apply the suggested techniques? Were some techniques easier to apply than others? Which ones? In your opinion, what factors prevent you or any other farmers from applying the new techniques? (EQ1) 4. If you adapted your practices following the training and/or support services, do you feel that you will be able to continue to apply the techniques you learned from the in the future? Why or why not? (EQ1) 5. What have been the trends in your community in terms of the number of fields planted in sorghum? What factors have influenced how much sorghum is planted in your community? (EQ1, EQ2) 6. What have been the trends in sorghum yields per hectare in recent years? (yields decreases, yields unchanged, yields less than double, double, more than double). What factors have affected sorghum yields? (EQ1) 7. Have there been any changes in the way you process and store your sorghum after harvest in recent years? If so, what has changes? What factors contributed to these changes? (EQ1) 8. To what extent has the quality of the sorghum that farmers currently sell improved in recent years? What factors have contributed to or hindered improvements in the quality of sorghum sold? (EQ1) USAID.GOV SMASH PERFORMANCE EVALUATION | 58 9. What are the main problems that farmers in your community face in marketing their sorghum? (EQ2) 10. Have there been any improvements or deterioration in farmer access to markets in recent years? If so, what are examples of changes? What factors influenced these changes? (EQ2) 11. To what extent have SMASH/BRANA/EDN activities created more market opportunities for farmers to sell their sorghum in your community? (EQ2) 12. Have you sold your sorghum to SMASH/BRANA? Why or why not? (EQ2) 13. What other markets/buyers to you sell your sorghum to? (EQ2) 14. What is your preferred market to sell your sorghum? Why? (EQ2) 15. What have been the trends in recent years in terms of the prices farmers get for their sorghum? What factors have influenced the price farmers get for their sorghum? (EQ1, EQ2, EQ3) 16. What have been the trends in recent years in terms of the cost of growing sorghum? What factors have influenced the costs of growing sorghum? (EQ1, EQ2, EQ3) 17. What have been the trends in farmer income gained from sorghum in recent years? What factors have influenced farmer income? Are there any important differences between men and women farmers? If so, how do you explain the differences? (EQ1, EQ3) 18. What recommendations would you offer to better assist farmers in the production and marketing of sorghum in the community? (EQ1, EQ2, EQ3) 59 | SMASH FINAL EVALUATION REPORT USAID.GOV Protocol for FGD with Agronmists A. General Information 1. Date: 2. Number of Participants: 3. Sex of participants: __ male __ female 4. Location of FGD: 5. Interviewer Name: 6. Primary Notetaker Name: B. Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask FGD participants if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ C. Discussion Questions 1. What are the difficulties that farmers in your community face in the production of sorghum? (EQ1) 2. What have been the trends in your community in terms of the number of fields planted in sorghum? What factors have influenced how much sorghum is planted in your community? (EQ1, EQ2) 3. What have been the trends in sorghum yields per hectare in recent years? (yields decreases, yields unchanged, yields less than double, double, more than double). What factors have affected sorghum yields? (EQ1) 4. Have there been any changes in the way farmers process and store sorghum after harvest in recent years? If so, what has changes? What factors contributed to these changes? (EQ1) 5. To what extent has the quality of the sorghum that farmers currently sell improved in recent years? What factors have contributed to or hindered improvements in the quality of sorghum sold? (EQ1) 6. What are the main problems that farmers in your community face in marketing their sorghum? (EQ2) 7. Have there been any improvements or deterioration in farmer access to markets in recent years? If so, what are examples of changes? What factors influenced these changes? (EQ2) 8. To what extent have SMASH/BRANA/EDN activities created more market opportunities for farmers to sell their sorghum in your community? (EQ2) 9. What have been the trends in recent years in terms of the prices farmers get for their sorghum? What factors have influenced the price farmers get for their sorghum? (EQ1, EQ2, EQ3) 10. What have been the trends in recent years in terms of the cost of growing sorghum? What factors have influenced the costs of growing sorghum? (EQ1, EQ2, EQ3) USAID.GOV SMASH PERFORMANCE EVALUATION | 60 11. What have been the trends in farmer income gained from sorghum in recent years? What factors have influenced farmer income? Are there any important differences between men and women farmers? If so, how do you explain the differences? (EQ1, EQ3) 12. Did you receive training from SMASH? If so, what was the focus of the training? To what extent was the training you received adequate and relevant to enable you to assist sorghum farmers in the region to adopt improved cultivation techniques? 13. To what extent were you able to apply what you learned in the SMASH training for agronomists? What are some examples of follow-up actions that you have taken? (EQ1) 14. Have you participated in any activities other than training organized by SMASH? Is so, which activities? (EQ1) 15. What recommendations would you offer to better assist farmers in the production and marketing of sorghum in the community? (EQ1, EQ2, EQ3) 61 | SMASH FINAL EVALUATION REPORT USAID.GOV MINI-SURVEY QUESTIONNAIRE A. Demographics 1. Date: 2. Name of farmer: 3. Sex of: __ male __ female 4. Belongs to farmer’s association: __ yes __no 5. Has contract to sell to BRANA: ____ yes ____no 6. Commune and Locality: 7. Department: 8. Interviewer Name: B. Informed Consent Agreement Purpose: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called Social Impact, a company that is based in the United States. Our team is in Haiti to conduct a study about the Smallholder Alliance for Sorghum in Haiti (SMASH). SMASH is a public-private partnership (PPP) between USAID, the Inter-American Development Bank (IDB), and Brasserie Nationale d’Haïti (BRANA S.A.). BRANA launched SMASH in July 2013, with the objective of replacing imported malt with locally grown sorghum in the production of its popular Malta H, a non￾alcoholic beverage consumed here in Haiti. The goal of SMASH is to improve the livelihoods of up to 18,000 Haitian smallholder farmers through a market-driven, value chain approach to increase yields of high-quality, locally grown sorghum. You have been asked to participate today so that we can learn more about your involvement in the SMASH program/learn about challenges farming sorghum in Haiti. We are speaking with over 200 individuals (SMASH managers, agronomists, farmers, and other value chain actors) who participated in the program either as implementers, recipients of program services and/or experts. You have been invited to participate in this discussion because you participated in at least one SMASH activity or because you farm sorghum in a region serviced by the SMASH Alliance. We would like your opinions and thoughts about the SMASH services. We are independent consultants who have no affiliation with SMASH or the GOH. Procedures: If you agree to participate, we ask you to discuss your experience and opinion of the activities and services implemented under the SMASH program. The FGD will take up to 2 hours of your time. Although we will publish our findings in a public report, all of your answers will be kept confidential. Nothing you tell us will be attributed to any individual person. Although we may use quotes, none of the individuals interviewed will be named in the report. However, as this is a group setting, to preserve confidentiality, we ask you not to share anything we discuss here today with anyone outside of this group. Risks/Benefits: There are no significant risks to your participation in this study. You will not receive any direct benefit or compensation for participating in this study. Although this study will not benefit you personally, we hope that our results will help improve potential future institutional capacity building programs for government institutions in Haiti. Voluntary Participation: Participation in this interview is completely voluntary. You do not have to agree to be in this study. You are free to leave the FGD at any time or to decline to answer any question which you do not wish to answer. If you decline to participate, no one will be informed of this. Do you have any questions at this time? [Interviewer should answer any questions] Permission to Proceed I understand the purpose of the interview as outlined above and understand that I can withdraw from the interview at any time and for any reason. I agree to participate in the interview (Evaluator records). USAID.GOV SMASH PERFORMANCE EVALUATION | 62 Yes  No Initials of evaluator to indicate receipt of verbal consent: _____________________ Date _________________________ C. Production and Sales of Sorghum 1. What kind of assistance did you receive from SMASH to grow sorghum? (May choose more than one answer) A. Training B. Seeds C. Pre-financing D. Post-harvest training E. Business and financial management training F. Other ___________________ 2. What was the number of hectares of the plots in which you grew sorghum from March 2018-February 2019? ________ ha [work with respondent to translate plot size to hectares] Plots Size of Plot 1 2 3 Total 3. Harvest and Sales of Sorghum (March 2018-February 2019) Plots How much sorghum did you harvest last year? How much sorghum did you sell last year? What was the sales price? Value of Sales (to be calculated by ET) 1 2 3 63 | SMASH FINAL EVALUATION REPORT USAID.GOV Total 4. How has the volume of sorghum you produce changed, on average, since your participation in SMASH activities? (1) Decreased a lot (more than half) (2) Decreased (up to half) (3) Unchanged (4) Increased (up to 1.5 times) (5) Increased a lot (more than 1.5 times increase) 5. How has the volume of sorghum you sell changed, on average, since your participation in SMASH activities (1) Decreased a lot (more than half) (2) Decreased (up to half) (3) Unchanged (4) Increased (up to 1.5 times) (5) Increased a lot (more than 1.5 times increase) 6. Where/to whom did you sell your sorghum? (May choose more than one response) A. BRANA; B. local Madam Sara C. local market D. others 7. What were the total expenses incurred to produce and market sorghum on those plots? (Fill out table below) Cost item Plot 1 Plot 2 Plot 3 Total Labor (soil prep., planting weeding, harvest) Inputs (seeds, fertilizer and pesticides) Transportation Processing (threshing, bagging) USAID.GOV SMASH PERFORMANCE EVALUATION | 64 Total 8. Farmers’ Annual income What were your main sources of income last year? What was your income from these different sources of income last year? Sorghum (to be filled in by ET) Other crops Non-agricultural Trade Other Total Percentage of income from sorghum (to be calculated by ET) _______ 9. How has the percentage of your income that comes from sorghum sales changed since you joined SMASH? (Read choices aloud) A. Decreased a lot (by more than half) B. Decreased (by up to half) C. Unchanged D. Increased (by 1.5 times) E. Increased a lot (more than 1.5 times) 65 | SMASH FINAL EVALUATION REPORT USAID.GOV Protocol for Direct Observation of SMASH Interventions 1. Date 2. Primary Observer: 3. Description of the Activity intervention being observed: 4. Location: Observation of SMASH Mobile Ask stakeholder to demonstrate how SMASH Mobile is being used. Request to see evidence of features that are used – for how to enter data about potential suppliers, production, logistics, and/or examples of reports produced. Ask for the person/persons in charge of system management, providing user support, or performing data entry to be identified. Ask about training they received. 1. To what extent is the person who was asked to demonstrate SMASH mobile able to demonstrate its various features? (rate: was not able to demonstrate most features applicable to their role, was able to demonstrate most features applicable to their role, appeared to master all features) 2. To what extent are the main features of the SMASH mobile being used based on evidence such as data in data base, reports produced, availability of mobile devices with SMASH access (rate: not at all exploited, partially exploited, fully exploited) 3. To what extent was the software adapted/customized for local needs? Is the interface in a local language? Have forms been customized to match forms in use by entity? (rate: not adapted/partially adapted/fully adapted) 4. Were user manuals provided? Are they in a language that is easily understood? (rate no user manual, user manual not in local language, user manual in local languages) 5. To what extent is the system being properly maintained? Is there evidence of maintenance occurring? (no evidence, some evidence, extensive evidence) Observation collection centers Visit the collection centers set up by SMASH. Visit both a micro collection center and the new model of centralized center. 1. Observe the type of equipment and infrastructure available at the collection center to collect and process famers’ sorghum stock 2. If possible, observe the process of collecting sorghum at the center including interactions with farmers. 3. Observe the location of the center in relation to smallholder farmer fields. Protocol observation in markets Visit the local market (preferably during the market day) and observe transactions on sorghum being conducted. Look also for storage facilities. 1. Observe whether sorghum is available in the market during site visit. 2. Observe who are the suppliers and buyers and where they come from. 3. Observe sorghum transaction prices. Notes by Observer USAID.GOV SMASH PERFORMANCE EVALUATION | 66 ANNEX C: REFERENCES AND ENDNOTES Annual reports Papyrus S A. Smallholder Alliance for Sorghum in Haiti (SMASH) Year 3 annual report, July 1, 2015 to June 30, 2016. Papyrus S A. Smallholder Alliance for Sorghum in Haiti (SMASH) Year 4 annual report, July 1, 2016 to June 30, 2017. Papyrus S A. Smallholder Alliance for Sorghum in Haiti (SMASH) Year 5 annual report, July 1, 2017 to June 30, 2018. Quarterly reports Papyrus S A. SMASH technical report Year 1 Fourth Quarter April-July 2014. August 2014. Papyrus S A. SMASH technical report Year 2 Third Quarter January-March 2015. Papyrus S A. SMASH technical report Year 2 Fourth Quarter April-June 2015. Papyrus S A. Smallholder Alliance for Sorghum in Haiti (SMASH) Year 6 Quarter 1 report, July 1 to September 30, 2018. Papyrus S A. Smallholder Alliance for Sorghum in Haiti (SMASH): Year 6 Quarter 2 report, October 1 to December 31, 2018. Work Plans Papyrus S A. SMASH transitional Work plan 1 January 2015 to 30 June 2016. Papyrus S A. SMASH transitional work plan July 2017 to June 2019. August 2017. Ayiti Nexus. Assessment and revision of SMASH’s M&E System. Consultancy report. July 2014. Publications Agroconsult Haiti S A. Etude des systèmes de production agricole et des associations paysannes dans les bassins versants de la rivière la Quinte et de la rivière Grise. Port-au-Prince. USAID/WINNER. Octobre 2009. Bayard, B. Consultation pour le renforcement de la capacité d’analyse de la Coordination Nationale de la Sécurité Alimentaire (CNSA) dans les processus des achats locaux. Port-au-Prince. MARNDR/DEFI. Janvier 2012. CACG/Agroconsult. 2018. Etude socio-économique du Bassin versant des Trois-Rivières. CHIBAS. Pitimi pa pè pichon : réponse à la prolifération du puceron jaune (Melanaphis) sur le sorgho (Pitimi). Rapport mai 2017. Eckert Elisabeth and Anna Latané. RTI International. The sorghum value chain in Haiti: A mapping and analysis. International Development Working Paper No. 2017-04. September 2017. FEWS NET. Haiti: Staple Food market fundamentals. March 2018. FEWS NET. Haiti Supply and market outlook. October 2018. 67 | SMASH FINAL EVALUATION REPORT USAID.GOV MARNDR. Résultats des enquêtes nationales de la production agricole année 2014. Port-au-Prince, Haïti. Unité de Programmation. 2015. MARNDR. Résultats des enquêtes nationales de la production agricole année 2016. Port-au-Prince, Haïti. Unité de Programmation. 2016. MARNDR. Synthèse Nationale des Résultats du Recensement Général de l’Agriculture 2008/2009. Port￾au-Prince, Haïti. 2012. USAID. Economic Growth and Agricultural Development Fact Sheet. March 2017. Van Vliet, Geert, Gael Pressoir, Marzin Jacques, and Giordano Thierry. Une Étude Exhaustive et Stratégique Du Secteur Agricole/Rural Haïtien et Des Investissements Publics Requis Pour Son Développement. Montpellier, France: CIRAD. 2016. USAID.GOV SMASH PERFORMANCE EVALUATION | 68 ANNEX D: LIST OF KEY INFORMANTS SMASH Implementation Team Luc Hilhorst Alice NKUNZIMANA Michael O' Donelly Jean Mario Bayane Roudy Jeune Hans Rene Beatrice Gerard Patrick Dessources BRANA/USAID/IDB Wieste Mutters Reginald Toussaint Alexious Butler Yolanda Strachan Richard Phillips Other Value Chain Stakeholder Gael Pressoir Ken Michel Oyama Michel Romain Eliasaint Magloire GOH Officials Emmanuel Prophet Gaston Démosthène, Other Donor/International Organization/INGO representatives Annah Latane Robin Padberg Patrick Dion