This publication was produced at the request of the United States Department of Agriculture. It was prepared independently by AUTHOR NAME. Foreign Agricultural Service, United States Department of Agriculture Senegal Food for Progress Project FINAL EVALUATION May 2020 Millet Business Services Project - Senegal Final Evaluation Report Program: Food for Progress Agreement Number: FCC-685-2014/055-00 Funding Year: Fiscal Year 2014 Project Duration: 2014-2019 Implemented by: National Cooperative Business Association Cooperative League of the United States of America (NCBA CLUSA) Evaluation Authored by: IMC Worldwide Jason Wares Christopher Root Alioune Sow FINAL EVALUATION REPORT MILLET BUSINESS SERVICES PROJECT SENEGAL PREPARED FOR: NCBA CLUSA PREPARED BY: IMC WORLDWIDE, INC. Submitted: 6 March 2020 Revised: 15 May 2020 Publication Information This report was prepared by IMC Worldwide under contract to NCBA CLUSA as part of its work with The Millet Business Services Project in Senegal. Project team: Mr. Jason Wares, Team Lead Mr. Christopher Root, Co-evaluator Mr. Alioune Sow, Co-evaluator Disclaimer This report’s authors are solely responsible for its content. It does not reflect the official views of NCBA CLUSA or any other individual or organization mentioned herein. Table of Contents EXECUTIVE SUMMARY............................................................................................................................................... 1 FINAL EVALUATION PURPOSE AND EVALUATION QUESTIONS....................................................................................................... 1 DESIGN AND METHODOLOGY ............................................................................................................................................... 1 SUMMARY OF FINDINGS ...................................................................................................................................................... 2 LESSONS LEARNED, RECOMMENDATIONS, AND CONCLUSION...................................................................................................... 3 1. BACKGROUND AND CONTEXT................................................................................................................................ 6 1.1 MILLET IN SENEGAL ............................................................................................................................................... 6 1.2 PROJECT AREA DESCRIPTION ................................................................................................................................... 9 1.3 PROGRAM DESCRIPTION.......................................................................................................................................10 1.4 EVALUATION TERMS OF REFERENCE (TOR) ..............................................................................................................12 2. METHODOLOGY...................................................................................................................................................14 2.1 DOCUMENT REVIEW ............................................................................................................................................14 2.2 QUALITATIVE RESEARCH .......................................................................................................................................14 2.2.1 Quantitative research ................................................................................................................................15 3. FINDINGS .............................................................................................................................................................18 3.1 PROJECT PERFORMANCE.......................................................................................................................................18 3.1.1 Extent of Goal and Intermediate Results Achievement..............................................................................20 3.1.2 Activities and Targets................................................................................................................................. 31 3.2 RELEVANCE ........................................................................................................................................................37 3.2.1 Alignment with Senegal Government Strategy..........................................................................................37 3.2.1 Responsiveness to Participants/Beneficiaries............................................................................................38 3.2.2 Alignment with USDA/US Government Goals, Objectives, and Strategies ................................................41 3.3 EFFECTIVENESS ...................................................................................................................................................41 3.3.1 Increases in Yields and Reduction in Waste ...............................................................................................41 3.3.2 Quality Adoption........................................................................................................................................41 3.3.3 Training In Quality Standards And Agricultural Techniques And Technologies.........................................44 3.3.4 Post-Harvest Processing, Handling, Marketing and Branding...................................................................46 3.3.5 Extension Officers Technical Support to Farmers.......................................................................................48 3.3.6 Linkages Among Project Participants ........................................................................................................49 3.4 EFFICIENCY ........................................................................................................................................................52 3.4.1 Internal and External Factors That Influenced Expected Results...............................................................52 3.4.2 Staffing, Management, and Resource Allocation ......................................................................................54 3.5 SUSTAINABILITY ..................................................................................................................................................57 3.6 IMPACT .............................................................................................................................................................59 3.6.1 Impact on program participants................................................................................................................59 3.6.2 Millet as a viable crop................................................................................................................................59 3.6.3 Extent of project reach...............................................................................................................................59 3.6.4 Unexpected Outcomes...............................................................................................................................59 3.6.5 Overall impact of increasing trade.............................................................................................................60 3.6.6 Increase in trade ........................................................................................................................................60 3.7 LESSONS LEARNED AND BEST PRACTICES..................................................................................................................60 4. RECOMMENDATIONS AND CONCLUSION.............................................................................................................61 ANNEX A: EVALUATION TERMS OF REFERENCE .......................................................................................................65 ANNEX B: WORKS CITED AND DOCUMENTS REVIEWED...........................................................................................84 ANNEX C: DATA COLLECTION TOOLS........................................................................................................................89 ANNEX D: RESULTS TABLE...................................................................................................................................... 113 ANNEX E: TEAM COMPOSITION.............................................................................................................................133 ANNEX F: EVALUATION MATRIX ............................................................................................................................134 ANNEX G: APPLICATION OF RECOMMENDED MID-TERM ACTIONS........................................................................140 Tables Table 1 Literacy rates........................................................................................................................................ 9 Table 2 Results Framework.............................................................................................................................11 Table 3 Key Informant Interviews...................................................................................................................15 Table 4 Survey Overview ................................................................................................................................17 Table 5 Extent of Achievement.......................................................................................................................20 Table 6 Number of individuals receiving short-term agricultural sector productivity or food security training as a result of USDA assistance by category ....................................................................................................23 Table 7 Increased use of improved agricultural techniques and technologies..............................................25 Table 8 Value of Sales by Respondent Type (2018/2019) ..............................................................................28 Table 9 Products market share and value (USD) ............................................................................................29 Table 10 Demo plots, fairs, and PO extension training ..................................................................................31 Table 11 Input suppliers trained.....................................................................................................................31 Table 12 Financial services activities that met or exceeded targets..............................................................32 Table 13 Financial services activities that missed targets..............................................................................33 Table 14 Post harvest processing activities that missed targets....................................................................33 Table 15 Market access activities...................................................................................................................34 Table 16 Improved agriculture activities that met targets.............................................................................35 Table 17 Improved agriculture activities that met targets (continued) .........................................................35 Table 18 Improved agriculture activities that missed targets........................................................................36 Table 19 Improved marketing and branding activities...................................................................................36 Table 20 Post-harvest processing and handling activities..............................................................................37 Table 21 Top two most useful trainings by organization type .......................................................................45 Table 22 Visited hub status.............................................................................................................................47 Table 23 Attendance at Events by Respondent Type .....................................................................................49 Table 24 Success in Making and Maintaining Linkages..................................................................................50 Table 25 Percent Budgeted and Spent on Activity Components....................................................................56 Figures Figure 1 Food Supply, Grain Type by Year........................................................................................................ 6 Figure 2 Agricultural land utilization in Senegal, 1961-2017 (hectares)........................................................... 7 Figure 3 Grain prices (XOF/KG)......................................................................................................................... 8 Figure 4 Millet products and packaging designed by participants in the MBSP.............................................10 Figure 5 Locations of Survey Respondents........................................................ Error! Bookmark not defined. Figure 6 Change in Yield between Baseline and Final Evaluation ..................................................................21 Figure 7 Reasons for Yield Increases (n=119)................................................................................................. 22 Figure 8 Sellers’ Reasons for Revenue Increase .............................................................................................28 Figure 9 Mean Producer Sales Volume...........................................................................................................29 Figure 10 Hub President Demonstrating a Quality Test to Show that Millet Grain Size and Quality is Consistent throughout a bat...........................................................................................................................42 Figure 11 Processor Perception of Quality Improvements.............................................................................43 Figure 12 An MBSP Coach Demonstrates Packaging Equipment ...................................................................43 Figure 13 One processor on left demonstrates good hygiene practices,.......................................................44 Figure 14 A hub president weighs high quality seeds to go into storage on behalf of a producer................46 Figure 15 The Khelcom Birane Hub president with millet bags including hub's contact information (Photo printed with consent) .....................................................................................................................................47 Figure 16 MBSP improved packaging on left, traditional packaging on right ................................................48 Figure 17 Kaffrine Processor Demonstrating Packaging and Weighing .........................................................50 Figure 18 Financial Arrangement for Input Purchases...................................................................................52 Figure 19 A millet drier procured for a processor through the USDA| MBSP................................................52 Figure 20 The Many Spellings of Producer Organization Names ...................................................................55 Figure 21 The storage facility roof at the Keur Samba Guèye hub requires maintenance ............................58 Abreviations and Terms Abbreviation Definition ANCAR Agence Nationale de Conseil Agricole et Rural (National Agricultural and Rural Advisory Agency) APS Agents Prestataire de Service CMS Crédit Mutuel du Sénégal CRS Catholic Relief Services COP Chief of Party CRS Catholic Relief Services DHS Demographic Health Survey DRDR Directions Régionales De Développement Rural (Directorate of Rural Development) FAIS Food Aid Information System FAO Food and Agriculture Organization FARK Femme Artisans Rurales de Kaffrine FAS Foreign Agriculture Service FFE Food for Education FFPR Food For Progress Result FTF Feed the Future ICS Industries Chimiques Du Senegal ISRA Institut Sénégalais de Recherches Agricoles (Senegalese Agricultural Research Institute) ITA Institut De Technologie Alimentaire MBSP Millet Business Services Project MGD McGovern Dole MVCP Millet Value Chain Project NCBA CLUSA National Cooperative Business Association Cooperative League of the United States of America ODK Open Data Kit SILC Savings and Internal Lending Communities TOR Terms of Reference USDA United States Department of Agriculture USG United States Government 1 Executive Summary Millet is a highly nutritious crop that is resistant to drought weather conditions commonly experienced in Senegal. It contributes to relieving food insecurities brought on by the market fluctuations and climate change created challenges that lead to The World Food Program (WFP) classifying Senegal as a “least￾developed and food-deficit” country (World Food Program). The USDA funded Millet Value Chain project (MVCP), implemented from 2011-2013, demonstrated that millet yields can increase through improved production techniques and quality inputs. In fact, MVCP increased millet yields in targeted areas by as much as 75% (NCBA CLUSA, 24 November 2015). Following the success of the Millet Value Chain project, USDA funded and NCBA CLUSA implemented a second phase of the project; the Millet Business Service Project (MBSP). The MBSP began in October 2014 and ended in December 2019. This second phase of the project focused on post-harvest handling and building the capacity of producer organizations to access markets and add value to their crop. The main goals of the project were: 1. Increase the agricultural productivity of the millet value chain by: • Developing processing systems; • Building producers’ capacity to improve production and quality; • Training producers and processors in improved techniques, post-harvest handling, marketing, and seed production; and • Providing grants and loans for equipment and inputs. 2. Expand trade of millet by: • Developing public-private relationships; • Facilitating trade relationships, researching export opportunities, and building agricultural extension capacity; and • Promoting food safety issues and requirements (JE Austin Inc., 2018). The MBSP sought to work with 22,150 producers, 16,630 trade beneficiaries, government agencies, and the private sector in the specific areas of Kaolack, Kaffrine, Fatick, as well as the capital, Dakar. Final Evaluation Purpose and Evaluation Questions The purpose of this report is to provide an “independent assessment and analysis of the overall effectiveness, impact, sustainability, and the degree to which MBSP has accomplished its objectives and results as outlined in the results framework (NCBA CLUSA, p. 6).” The evaluation’s terms of reference included 28 specific questions grouped in 7 evaluation components; Goal Achievement, Relevance, Effectiveness, Efficiency, Sustainability, Impact, and Best Practices and Lessons Learned. Design and Methodology This MBSP final evaluation used mixed methods design. The methods used include the following: • A review of project documentation including baseline and midterm reports, biannual performance reports, grant agreement documents, annual strategy documents, financial reports, assessments, monitoring documents, and other documents that the project produced. • 57 key informant interviews (KIIs) with a range of actors including producers, producer organizations and hubs, processors, kiosks, government, and project staff. • A survey of 650 MBSP affiliated producers 2 • A survey of 189 producers who were neighbors, but not members, of CLUSA-supported producer organizations. • A survey of 531 organizations and business involved in buying and selling millet. Methodological limitations This evaluation has a number of significant limitations. • The MBSP did not have a complete list of participating producers, which means that the survey’s results are not representative of the entire population of MBSP producers. • Due to record keeping inconsistencies, it was not possible to randomly sample producers. • Baseline study data was not available so the final evaluation is unable to test differences between baseline and endline for statistical difference. • This evaluation includes a group of neighbor producers and makes comparison between them and MBSP producers. However, this neighbor farmers group is not large enough to support statistical tests of differences. • The last millet harvest in the project region was in and around November 2019. Therefore producers still had not sold all of the millet they will sell this season at the time of the survey. Reported sales figures combine what had been sold with what producers expect following the completion of the survey. Summary of Findings In addition to the achievements described below, the evaluation team documented project successes included the innovative creation of trusted purchase points in the form of millet hubs and kiosks, the fulfillment of branding and packaging campaigns to change attitudes about millet products, and the successful creation of relationships between millet actors, especially concerning financing. The project’s shortcomings included pursuing targets as ends in themselves, such as purchasing electrical equipment for project locations with no access to power in order to meet performance indicator targets, and struggling with tracking project activities and data. Here the executive includes a brief summary of major findings by evaluation component. Goal Achievement The MBSP sought to achieve two high level strategic objectives: SO1 – Increased Agricultural Productivity and SO2 – Expanded Trade of Agricultural Products (Domestic, Regional, and International. The evaluation team believes that the MBSP achieved SO1 to the extent that farmer yields are larger than they would have been in the absence of MBSP’s efforts given sparse rainfall that fed the millet crop in 2016 and 2019. The evaluation found that 82 percent of millet farmers saw a decline in millet yield during the 2019 season. Despite these lower yields, the evaluation team presents evidence that suggests that yields are higher than they would have been in the absence of MBSP’s efforts given the sparse rainfall that fed the millet crop in 2016 and 2019. Trade of millet and millet products expanded under the MBSP. The evaluation found that 88.7 percent of sellers that participated in the MBSP reported that sales revenue is higher than at the start of the program. 3 63.8 percent of these sellers reported revenue increasing due to better millet quality. 49 percent pointed to better customers, which are customers that recognize improved quality and are willing to pay for it. . Relevance The evaluation team found that the MBSP’s activities were in line with Government of Senegal strategy and priorities. Government officials that the evaluation team spoke with were knowledgeable about the project’s specific activities and achievement, and had a high opinion of the project. The project developed mechanisms that ensured it was generally responsive to its participants, particularly events and forums and interfacing with coaches. Events and forums were especially useful as they allowed participants to interact with each other, developing new ideas and linkages. Effectiveness The MBSP’s participants adopted quality practices. For producers, improved quality meant delivering grain that was sorted by size and color, and contained almost no foreign materials. For processors, improved quality meant adopting sanitary practices, improved packaging, and improved processing techniques. Over 87 percent of MBSP producers said that quality had improved since the start of the project. Participants attributed quality increases to the trainings that they’d received through the project. Partner organizations, in the form of hubs and producer organizations, are increasing the use of improved post￾harvest processing and handling practices, though the degree to which practices are increasing varies greatly among organizations and hubs. Kiosks and processors improved marketing and branding through two channels, improved packaging and improved customer management. The project established effective linkages among different project participants. Hubs established linkages to processors particularly successfully in establishing linkages, with greater than 50 percent establishing linkages through MBSP events. Linkages were also established among producers, processors, and some financial institution, input providers, and equipment manufacturers. Efficiency Staffing and Management appear to have been structured efficiently, with the exception of the MBSP’s monitoring and evaluation function and its grant management function, which managed the project’s grants to its participants. MBSP was strategically driven and benefited from strong leadership, but it fared poorly in data management. Sustainability The evaluation identified several factors contributing to sustainability, including training continuing after the program, improved agricultural processes, raised awareness in the market, and growing demand. This section of the report also discusses factors hindering sustainability, such as climate change, access to electricity, and uncertain government resources. Lessons Learned, Recommendations, and Conclusion The report ends with a review of lessons learned, a presentation of the evaluation team’s recommendations, and a summary conclusion of the project. The lessons learned section details the following lessons that the evaluation highlights: Lesson 1: A strong knowledge of the sector and its history drove project success: The project’s successes were driven by the project leadership’s strong knowledge of the millet sector, its structure, its history, and its main actors. This knowledge carried over from the first phase of the project. For example, the project’s leadership understood that millet was a grain that was grown in regions that typically had difficulty growing crops, and that millet was seen as a low quality, subsistence crop. As was uncovered during staff 4 interviews and interviews with project participants, the project’s leadership understood that the project would have to raise the profile of millet as a product relied on for food security to a high quality, delicious and nutritious product. The project’s leadership also came into the project understanding lack of trust among value chain actors contributed to high transaction costs. Coming into the project with this knowledge established an early foundation for the project’s successes. Lesson 2: A clear strategy based on creating points of market entry, building trust, and demonstrating value: The project had a clear strategy based on creating points of entry into the market for different value chain actors, building trust, and demonstrating value. Key points of entry included the project-established millet hubs, business forums and events, and millet kiosks. Lesson 3: The project culture was important to its success: A review of project documentation and interviews with project staff reveal that the MBSP leadership instituted an ethic of acting strategically and acting in service to project participants. Lesson 4: Opportunities for participants to provide feedback to the project and to each other pay off: The project’s events and forums provided unique opportunities for the project and its partners to listen to participants and create initiatives that responded to their needs and aspirations. Importantly, the design of those events created opportunities for participants to listen to each other. Lesson 5: Output oriented results frameworks, especially matched with short time frames, may lead to strange incentives: The project’s results framework was nearly entirely output based. The overemphasis on outputs created an incentive that led to rushing to complete tasks, such as building and outfitting the hubs rather than laying a solid foundation for future initiatives. The Evaluation Team’s recommendations include: Future project’s should learn from the MBSP’s leadership in establishing trust among millet sector actors: The MBSP identified early in the project that trust among actors led to high transaction costs. In response the project engaged in trust building actions and investments that were important to the project’s success. Use a market facilitation approach: The MBSP donated and heavily subsidized infrastructure and equipment. These donations and subsidies included millet storage structures and cleaning equipment for hubs and kiosks for millet vendors. In some cases, the evaluation team found this equipment in disuse or not suitable for the user, implying wasted project resources. Instead of giving away the equipment or providing such generous subsidies, the future projects should work to link businesses and hubs to finance, using business and technical capacity building training to buy down risk. Maintain a regular conversation about project goals. The project had two strategic objectives, increase productivity, in the form of increased millet yields, and increase trade. External factors influenced the way these objectives were pursued. However, USDA’s understanding of project strategy did not change despite the change in the project’s strategy. Future projects should keep an open conversation with USDA about the dynamic environments they operate in and how tactics change so that projects receive the support they need and their achievement is measured appropriately. Project’s results frameworks should focus on outcomes rather than mainly outputs: The indicators that measured the MBSP’s achievement were mainly focused on counting outputs. They measured the number of people that went through trainings, the number of producer organization members, the number of agreements signed, and the like. Future projects should include means of tracking actual changes in behavior. These methods would allow project management to adjust activities as needed and allow USDA to have a better understanding of a project’s success. 5 Improve project monitoring: Basic information about the project was unavailable, such as the number of trainings the project supported, the number of people that had attended trainings, the number of project beneficiaries, and the number of producer organizations the project served. The data that the project kept, such as its database of producers and producers’ organizations, was incomplete and contained unusable data. In the future USDA and NCBA CLUSA should provide monitoring oversight. The full evaluation provides specific suggestions for improving oversight. Consolidation and harmonization of learning questions. The final evaluation TOR had 28 evaluation questions, the mid-term 25, and the baseline a total of 28 questions. These are exceptionally high numbers of questions that necessarily limit how well each question can be answered and reduce report readability. Furthermore there was little alignment in evaluation questions between the baseline and the mid-term and endline evaluations. Requiring a smaller number of focused and overlapping evaluation questions across baseline, midterm and endline would enhance the quality of evaluations and the opportunities for learning. The report concludes with a review of the project’s context and ambitious undertaking, and its path to achievement. 6 1. Background and Context In this section, the evaluation report presents the context in which the MBSP operated in. 1.1 Millet in Senegal Since domestication between 3000 and 2000 BC (Jere Brunken, 1977), farmers in the Sahel have depended on millet’s hardy abilities to provide protein, iron, and zinc for their families as well as feed for their livestock (G. Basavaraj, 2010). Millet has played an important role in many regions of present-day Senegal throughout the centuries due to its aptitude for survival in areas with irregular rainfall, and remains a staple food in rural areas of the country today. The cultural dishes presented during traditional Senegalese ceremonies highlight its historical importance. For example, men provide millet porridge to guests at baby naming ceremonies as a symbol of their role in feeding others (Buggenhagen, 2011) . Unfortunately, millet has fallen out of use in many households as a result of opportunity cost. Beginning in the 17th century, farmers in Senegal began to prioritize the production of peanuts, a cash crop, and substituted labor-intensive millet with cheap rice imports from Asia (Brooks, 1975). The use of rice as a primary source for nutrition became even more widespread in the country after urbanization in the 1960’s, with non-grain producing families primarily consuming imported rice (Ross, 1982). Data published by the Food and Agriculture Organization of the United Nations (FAO) records a steep decline in the household consumption of millet since 1975, where it provided 28% of the food supply (kcal/capita/day). In 2013, millet comprised of just 8.6% of the food supply while rice claimed 29.1% (see figure 1, right) (Food and Agriculture Organization of the United Nations (FAO) , 1961 to 2013). Senegal’s dependence on imported rice has had several consequences. First, rice does not contain the micronutrients that millet does – leading to a lower quality diet among the population. Second, Senegalese families are exposed to rice’s price fluctuations as dictated by the foreign food market, as occurred during the world rice market crisis in 2008 (Aker, Block, Ramachandran, & Timmer, 2010). This exposure occurs because most of Senegal’s agricultural land is located in dry areas, and the country does not have the capacity to produce enough rice to satisfy domestic demand. The Senegalese government has long been concerned with the country’s dependence on rice imports, although efforts to promote more millet production and consumption have generally underperformed due to illegal trade and organizational challenges (Ross, 1982). Data on food production indicate that little has changed over the past 50 years in terms of the amount of land utilized for millet production, unaffected by a significant overall increase in the population and urbanization in the country (see figure 2) (Guèye, 2007). This suggests that, despite its cheaper price, millet remains a subsistence crop for rural families to be grown and consumed in localities. While processed millet exports have increased significantly since 2012 (the FAO estimates $2,000 worth of millet exported in 2012 and $191,000 worth of millet exported in 2017), amounts are insignificant when compared to Senegal’s main exports: Prepared foods (not specified), groundnuts, and tobacco products. Figure 1 Food Supply, Grain Type by Year Data Source: Food and Agriculture Organization of the United Nations (FAO) database, Senegal, 1961 to 2013. 7 Figure 2 Agricultural land utilization in Senegal, 1961-2017 (hectares) Data Source: Food and Agriculture Organization of the United Nations (FAO) Millet productivity constraints and marketing in Senegal Millet is a relatively hardy crop that thrives well in the Senegalese environment, yet challenges associated with normal crop production in dry climates combined with socioeconomic factors like transport infrastructure, changing market demand, and government policy within the country have inhibited its productivity and marketing. Millet, like other grains, is susceptible to destructive disease and pests; and historical bouts of biotic restraints in Senegal have interrupted yields. For example, in 2004, incident rates of downy mildew disease in millet reached 65% causing historically low yields (J.P. Wilson, 2008). Millet head miner, an insect pest common in West Africa, can inflict devastating losses between 40-80% (Feed the Future Innovation Lab for Collaborative Research on Sorghum and Millet , 2018). Other biotic stresses such as birds, insects and viral disease have caused interrupted harvests. Studies show that Senegalese farmers lack access to appropriate improved varieties of millet that are more resistant to biotic stresses, contributing to consistently low yields (Waddington, 2010). Further barriers to millet production and marketing have been documented in recent studies. Researchers indicate that transportation costs account for 20-40% of price differentials1 , the main source of inefficiency for millet markets in Senegal." Transportation costs are estimated to account for 1%-9% of price differentials (Jacques, 2018). Poor infrastructure, especially in rural areas of the country, increase the cost of transporting millet to market centers for consumption in urban areas. This is compounded with its low value-to-weight ratio, which limits the extent of long-distance transport and international trade (Masters, 2007). Finally, compared to other flour types, the storage life of prepared millet flour is limited due to relatively high fat content. Untreated flour will develop a bitter taste after only a few days of storage (Yadav, 2012). Urban customers, concerned with storage abilities and relative ease of preparation, have consistently shown preference to imported rice. This preference has not changed even though imported rice has been more expensive than domestically produced millet for at least the last decade.2 1 Differentials in this case refers to the difference in cost between millet sold in different locations. 2 This conclusion was reached based on data trends observed for food consumption and production in Senegal. Data was retrieved from the World Food Program’s Global Food Prices Database and aggregated. Database accessible here: 0 200000 400000 600000 800000 1000000 1200000 1400000 1600000 1961 1964 1967 1970 1973 1976 1979 1982 1985 1988 1991 1994 1997 2000 2003 2006 2009 2012 2015 Groundnuts, with shell Maize Millet Rice, paddy Sugar cane 8 Figure 3 Grain prices (XOF/KG) Data Source: Food and Agriculture Organization of the United Nations (FAO) Historical government policies have also affected millet’s place in Senegal’s agricultural system. Prior to independence in 1960, French colonial policy focused on the production of groundnuts, to be converted into peanut oil for export and use in Europe. Millet thus became the natural source of nutrition for rural farmers, while urban populations depended on cheaper imported rice. As the urban population grew, agriculture policy continued to focus on the production of export products, largely neglecting domestic food production and processing (John, 2015). Focus on peanuts encouraged monocropping – which degraded the sandy Senegalese farming soils, led to reliance on imported grain, and resulted in exposure to market shocks that contributed toward high malnutrition rates (Masters, 2007). More recent agricultural policies in Senegal have been heralded as positive, as increased government spending on agriculture has attempted to increase domestic food production (including efforts to expand production of grains like millet and rice). In fact, government spending on agriculture in 2017 reached $60.60 per capita, the fourth-highest in sub-Saharan Africa (Ibrahima Hathie, 2017). Yet, overcoming the problems associated with overspecification and climate have proven very difficult, and Senegalese farmers remain some of the most inefficient in the region (Ibrahima Hathie, 2017). Overall, the gap in agricultural productivity can be explained in large part by the low level of access to inputs like infrastructure and technology. 0 50 100 150 200 250 300 350 400 450 500 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 Maize (imported) - Retail Maize (local) - Retail Millet - Retail Rice (imported) - Retail Rice (local) - Retail 9 1.2 Project Area Description The Millet Business Services Project (MBSP)’s focus area was a continuation of the project’s previous work under MVCP. Like MVCP, the MBSP project activities took place in millet producing/processing regions of Senegal; specifically, the Kaolack, Kaffrine, and Fatick regions. Differently from MVCP, MBSP also included activities in the capital, Dakar. The three regions outside of the capital lie in Senegal’s groundnut basin, an area of the country that is dominated by the production of groundnuts and millet through rainfed agriculture (Famine Early Warning System Network, 2015). The economy in these regions revolves almost entirely on agriculture; comprising of 46% of the country’s cultivated surface and 51% of the national agricultural production (Direction de l’agriculture de la Prevision et des Statistiques agricoles, 2015) . According to data published by the Demographic and Health Survey (DHS) program in 2017, approximately 46.6% of adult males in these regions are engaged in agriculture as their primary form of employment (ANSD/Senegal and ICF, 2017). Child malnutrition rates, which are linked heavily to food access and quality in Senegal and thus relevant to this program, vary substantially between administrative regions. Data published by the most recent Standardized Monitoring and Assessment of Relief and Transitions (SMART) survey in 2015 show that child malnutrition in the peanut basin regions are significantly higher than in the capital (Dakar), although much lower than the drought-prone desert areas in the north of the country. Capacity: Education and Infrastructure Education, literacy, and internet access rates in the project region are some of the lowest in the country. In Kaffrine, 70% of men and 80% females cannot read. In the project region as a whole, 53% of the adult population has no education at all (Agence Nationale de la Statistique et de la Démographie (ANSD), DHS Program, 2017). Internet usage is also notably low, only 5.5% of women and 22.3% of men in Kaffrine region report having ever used the internet (Ibid.). Table 1 Literacy rates Region Fatick Kaolack Kaffrine Gender Male Female Male Female Male Female % of adults (15-49) who are illiterate 40.6 47.9 39.4 52 70.6 80.7 % Adults who have never completed any education level 32 45.9 39 53.7 70 78.1 Data Source: Enquete Demographique et de Sante Continue (EDS-Continue)”, 2017 Low education levels and poor infrastructure in the project regions pose barriers to the adoption of new agricultural technologies or more complicated farming techniques. Additional constraints to millet Figure 4: Child Malnutrition (% children under 5). Data source: 2015 Standardized Monitoring and Assessment of Relief and Transitions (SMART) 10 production in the project area identified through literature review include disease and pests (including locusts), poor access to quality inputs, low levels of training, low quality soil, and poor transport infrastructure. 1.3 Program Description The Millet Business Services Project (MBSP) was implemented from October 2014 through December 20193 . The MBSP was originally slated to launch in 2009. However, a global crisis in the supply of rice that year delayed project activities (Dong, 2011). Upon relaunch, MBSP was slated to run from 2014 through 2017, and was finally extended through December 2019. The MBSP’s final budget was USD $11,181,193. The MBSP followed up on the activities and successes achieved by the Millet Value Chain Project (MVCP), which USDA funded and NCBA CLUSA implemented from 2011 – 2013.4 MVCP focused on “training producers, artisans and processors; structuring and formalization of various categories of stakeholders (producers, seed producers, artisans); the establishment of customer/provider relationships (millet producers/processing businesses); the establishment of economic actor/support structure relationships (seed producers/DRDR for seed certification); the connection of economic actors with the credit institutions and the establishment of a guarantee fund; in-kind grants to processors (Soterco Group, 2016).“ These activities led to increases in millet farm productivity and increases in millet sales in the Kaolack, Fatick, and Kaffrine regions (JE Austin Inc., 2018). The MBSP continued the training it delivered to millet producers in the MVCP regions. MBSP extended its reach to include actors involved in millet trade, processing, input supplies, financial service providers, equipment manufacturers, government agencies, and other private sector actors. The project focused more on increasing trade and growing the Senegalese and international market for millet and millet products than increasing agricultural productivity. The USDA|MBSP’s goals and objectives were also described in its results framework. The results framework was focused on the achievement of two strategic objectives; SO1 - Increased Agricultural Productivity and SO2 -Expanded Trade of Agricultural Products (Domestic, Regional, and International). The project aimed to achieve each of these strategic objectives through producing specific intermediate and foundational results, as shown in Table 2. 3 The project was initially scheduled to run from 2014 to 2017. The timeframe of the project was extended through modifications to the grant agreement. In 2017 the project was extended until April 2019 through modification 1 to the grant agreement. In 2018 the project was extended until December 2019 through modification 4. 4 The project was initially slated to start in 2009, but the 2008 world food crisis delayed project launch (Dong, 2011). Figure 4 Millet products and packaging designed by participants in the MBSP 11 Table 2 Results Framework Strategic Objective IR Level 1 IR level 2 FFPr SO1 - Increased Agricultural Productivity FFPR 1.2 Increased Use of Improved Agricultural Techniques and Technologies FFPR 1.2.1 Increased Availability of Improved Inputs FFPr 1.2.3 Increased Use of Financial Services FFPr 1.2.4 Increased use of Improved Agricultural Techniques and Technologies FOUNDATIONAL RESULTS FFPr 1.4.1 Increased Capacity of Governmental Institutions FFPr 1.4.4 Improved Capacity of Key Groups in the Agriculture Production Sector FFPr SO2 – Expanded Trade of Agricultural Products (Domestic, Regional, and International) FFPr 2.1 Increased Value Added to Post-Production Agricultural Products FFPr 2.1.2.2 Improved Post￾Harvest Infrastructure FFPr 2.2 Increased Access to Markets to See Agricultural Products FFPr 2.2.2 Improved Linkages Between Buyers and Sellers FOUNDATIONAL RESULTS FFPr 2.4.4 Improved Capacity of Key Organizations in the Trade Sector USDA|MBSP aimed to achieve these strategic objectives and intermediate results through the implementation of nine activity components. These were: 1. Capacity building of agricultural extension agent services, producers, and the private sector - building capacity of government extension agents, also known as Direction Régionalede Développement Rurale (DRDR) agents, and working with millet unions to build internal extension services. 2. Financial services: facilitate agricultural lending – Increasing access to credit so farmers can purchase inputs, machinery, and other equipment. 3. Training: Improved agricultural production techniques – linking seed suppliers with farmers to provide inforomation on correct seed planting and fertilizer use. Also supporting conservation agriculture practices. 4. Inputs: Develop Agri-dealers and input suppliers – Using grants and other means to strengthen the capacities of seed providers, fertilizer providers, conservation agriculture machinery providers and conservation agriculture plowing services. 5. Training: Post-harvest handing and processing – Assisting producers to meet processors millet quality standards, improving storage techniques, building warehouse management capacity, connecting producers and processors, and building other post-harvest capacities. 6. Infrastructure: Post harvest handling and storage – Facilitating producer’s purchase of post￾harvest equipment and storage facility construction. 7. Infrastructure: Post harvest processing – Facilitating the purchase of processing equipment by processing firms. 12 8. Training: Improved marketing and branding – Providing marketing training to producer community members. 9. Market Access: Facilitate buyer-seller relationships – Facilitating linkages between producer organizations and processors, processors and retailers, and machinery service providers and processors. USDA|MBSP kicked off with a baseline study, completed in 2016, that CLUSA used to monitor the project’s progress and achievements. The baseline study was used as the basis of comparison to indicator measurements taken in the August 2018 MBSP mid-term evaluation. This final evaluation used the results of both the prior studies that were available to the final evaluation team to understand the project’s achievement within the framework of the evaluation’s terms of reference. 1.4 Evaluation Terms of Reference (ToR) National Cooperative Business Association, Cooperative League of the USA (NCBA CLUSA) contracted IMC Worldwide to conduct a final evaluation of its Millet Business Services Project (MBSP) in Senegal. The purpose of this evaluation “is to provide an independent assessment and analysis of the overall effectiveness, impact, sustainability, and the degree to which MBSP has accomplished its objectives and results as outlined in the results framework.”5 The evaluation had seven objectives, which were: 1. Assess project performance in meeting the end of the project activity and result targets; 2. Assess the relevance of the project; 3. Assess project effectiveness; 4. Assess efficiency of systems and processes; 5. Assess the degree of monetary and socio-economic impact of the project on the project beneficiaries in comparison with the baseline and mid-term evaluation results; 6. Assess the sustainability of the project by reviewing the capacity of producers, processors, and related value chain actors to continue providing support to MBSP millet production systems for potential expansion; 7. Document best practices and lessons learned and assess the application of recommended actions from the mid-term evaluation6 ; The evaluation’s terms of reference included questions grouped under the following criteria; Goal Achievement, Relevance, Effectiveness, Efficiency, Sustainability, Impact, and Best practices and lessons learned. These evaluation questions were: Goal Achievement • To what extent have project goals been achieved? • Which activities have been successful in meeting their targets? • Which activities failed to reach their targets and why? Include analysis of factors that delay/prohibit adequate progress towards results. Relevance • Are the project activities in alignment with Senegal Government priorities and strategies? • Do the project activities respond to the aspirations and needs of the project beneficiaries/participants? 5 Evaluation TOR, page 6 6 The Final Evaluation Report discusses the application of recommended actions from the mid-term throughout the text of the report where relevant. It also includes a description of the MBSP’s response to each of the 38 mid-term recommendations in Annex F. 13 • Are the project activities in alignment with USDA/US Government development goals, objectives, and strategies? • Are the project activities in alignment with award agreement with USDA? Effectiveness • To what extent have different actors in the value chain been able to increase quantity e.g., yield/ha and minimize waste since the inception of the project? • To what extent have quality standards been adopted by all actors in the value chain? • To what extent value chain actors have been trained in quality standards? • Are partner organizations providing training to beneficiaries that are tailored to the local context and achieving increased use of agricultural techniques and technologies? • Are partner organizations increasing the use of improved post-harvest processing and handling practices, and improved marketing and branding? • Are local government extension officers offering technical support to farmers in new techniques and technologies? • Have effective linkages been established between different project participants e.g., between producers and processors? Efficiency • What internal and external factors have influenced the ability of the project to meet expected results and targets? • Do project staffing, training activities and management structures efficiently support the achievement of expected project outcomes? • Does the project resource allocation to activities and to the project’s geographic areas efficiently support the achievement of expected project outcomes? Sustainability • What factors are likely to contribute towards the sustainability of the following factors in relation to the MBSP: policy environment, financial incentives (i.e., profit motive), access to inputs and information, market access, etc. • What factors are likely to hinder sustainability of the following factors in relation to the MBSP: policy environment, financial incentives (i.e., profit motive), access to inputs and information, market access, etc.? • What project activities are likely to contribute towards sustainability and is there evidence that project activities or benefits will be sustained after the project ends? Impact • What impacts are the project activities having on the program participants, both positive and negative, especially in relation to the expected results and strategic objectives? • To what extent have target groups (farmers, processors, vendors, consumers, etc) adopted millet as a viable crop that has economic and nutritional benefits? • What is the extent of the project reach beyond direct and targeted beneficiaries? • What are the unexpected outcomes that could have been due to the project? • What is the overall impact on increasing trade of millet? (marketing strategies, creation of new partnerships, market information systems, certifications, etc.) • How has the trade increased as a result of USDA MBSP on local, regional, national, and international level? 14 Best Practices and Lessons Learned • What lessons learned can be documented at this point to inform implementation of the program until completion? • What best practices can be shared with stakeholders and donors? 2. Methodology This MBSP final evaluation was a mixed methods evaluation. Fieldwork was carried out from December 10, 2019 to January 2, 2020. The methods used include the following: • A review of project documentation including baseline and midterm reports, biannual performance reports, grant agreement documents, annual strategy documents, financial reports, assessments, monitoring documents, and other documents that the project produced. • 55 key informant interviews (KIIs) with a range of actors including producers, producer organizations and hubs, processors, kiosks, government, and project staff. • A survey of 650 MBSP affiliated producers • A survey of 189 producers who were neighbors, but not members of CLUSA-supported producer organizations. • A survey of 531 organizations and business involved in buying and selling millet. The evaluation matrix, which links evaluation questions to the data tools that were used to answer them, is included as Annex F. This section describes each of these methods as well as the limitations encountered in carrying out the evaluation. 2.1 Document review The evaluation team reviewed a range of project documents to inform the evaluation design, including the design of data collection tools. The evaluation team reviewed the data collection tools used at midterm and baseline and adapted some of the questions appropriate for answering the evaluation questions. The evaluation team also reviewed and used data from performance reports to assess the project’s self￾reported progress against targets. Additionally, the evaluation team reviewed journal articles and public data on millet production in Senegal and the socioeconomic conditions in the project area to provide context for this evaluation. A full list of documents used can be found in Annex B. 2.2 Qualitative research The evaluation team featured three qualitative interviewers who conducted 55 key informant interviews. These interviews were conducted in all three project regions and Dakar. CLUSA provided lists of project beneficiaries and stakeholders, made recommendations on potential respondents who had played important roles in the project, and helped to arrange interviews. Ultimately however, the evaluation team chose which respondents to interview. The evaluation team aimed to include a representative selection of different actor types and locations, while also weighing the importance of their role in the project and logistical considerations. Table 3 below shows a summary of KIIs. Key informant interviews were informed by question guides. The guides ensured that interviewers asked key questions across the same respondent types. However, interviews were largely conversational and only semi-structured to allow interviews to explore respondents’ different, unique, and often unanticipated perspectives on the project. 15 The evaluation team used MaxQDA software to analyze the evaluation’s 55 interviews. The first round of codes were created at the same time as the creation of the interview guides to match the evaluation criteria and evaluation questions. The evaluation team first coded the interviews using this code-set. After the first review a second code-set was created to match themes and trends discovered that were discovered during the first review. The interviews were then recoded according to this second set. In addition to codes relating to the evaluation questions, the evaluation team utilized the second code-set to identify key quotes, stories, and recommendations. Table 3 Key Informant Interviews Respondent Type Fatick Kaffrine Kaolack Dakar Total CLUSA producers 9 1 3 13 Producer organizations 4 1 3 8 Hubs 3 2 3 8 Processors 1 2 3 6 Kiosk 3 3 Artisans 1 1 2 4 Advisors/APS 1 1 2 Input business 2 2 Finance 2 2 Transporter 1 1 Government 3 1 2 6 Staff 2 2 Total 19 5 21 10 57 2.2.1 Quantitative research The quantitative portion of the research consisted of two types of surveys: the producer survey and buyer￾seller surveys (and census). These surveys were carried out by a team of 20 interviewers using tablets and the data collection software Open Data Kit (ODK). The surveys were conducted between December 18, 2019 and January 2, 2020. The two questionnaires used can be found in the Annex C. Producer survey The producer survey targeted farmers who were members of producer organizations that participated in MBSP. The survey sample utilized a two-stage cluster design, with producer organizations (the cluster) selected at the first stage, and producers that were members of those organizations at the second stage. Initially the design sought to randomly select 220 producer organizations, which represents a simple random sample from a population of 500 producer organizations, to achieve a 95 percent confidence level and 5 percent margin of error. In the second stage, the evaluation team would randomly select three producers in each organization from a list provided by the project for a total of 660 producers. This sample 16 size of 660 results from taking a simple random sample size for a population of 30,000 producers of 3807 . To account for clustering, we multiply 380 by the design effect. The formula for the design effect is: DEFF = 1 + Intracluster correlation coefficient (cluster size – 1) Using a conservative intracluster correlation coefficient of 0.3, we get a design effect of 1.6 which leads to a sample size of 608. However since 220 is the necessary sample size for producer organizations to achieve a 95 percent confidence level and 5 percent margin of error, the team selected a larger than necessary producer sample with 3 producers at 220 producer organizations. The MBSP provided the evaluation team with a list of producers who were program participants. However, the list received from the MBSP was incomplete. It comprised 23,321 producers out of the 28,666 that the MBSP reported had participated in the program. Additionally, the list contained numerous spelling inconsistencies in organization, village and commune names, as well as duplicate entrees. The evaluation team cleaned the data as best it could but nevertheless, these inconsistencies persisted. These two errors in the list created difficulties in producing a complete list of individual organizations to serve as the first stage of the cluster design, and a complete list from which the evaluation team could select project participants. As a result, while the evaluation team was ultimately able to randomly select 220 producer organizations, it decided that it was too risky to randomly select producers given the high likelihood that a preselected list would include many producers that were not organization members, were listed as living in the wrong locality, or were not otherwise available. Instead, the survey team contacted producer organization presidents and asked them to select three members to serve as respondents. The consequence of working with a partial sample frame and non-randomly selected respondents is likely biased results. Producer organizations with less engagement with the project may have been more likely to have been omitted from the incomplete list used for selecting producer organizations. Then, producer organization presidents may have been more likely to select more engaged and successful members to be interviewed. The likely result is a bias towards more engaged and successful producers in more engaged and successful producer organizations. However this bias cannot be confirmed without knowing the true population values of our estimates. The producer survey also included neighbor producers. These were defined as millet producers who lived near the MBSP producer organizations, but who were not a member of those organizations. The survey team attempted to interview one neighbor farmer for each of the 220 sampled producer organizations. This was not always possible, and as a result the actual number of neighbor farmer interviews is lower. Neighbor producers were identified by producer organization presidents. Note that the evaluation did not have the resources for a control group. The neighbor farmer sample size was not intended to produce statistically representative results or statistically significant comparisons of differences with MBSP producers. Nevertheless, the results allow for some tentative comparisons between changes realized by MBSP members did compared to changes realized by non-members. 7 The formulas used to calculate sample size were: x =Z(c/100)2r(100-r), n = N x/((N-1)E2 + x), E = Sqrt[(N - n)x/n(N-1)]; where N = population size, r=fraction of responses, Z(c/100) is the critical value for the confidence level c. (Source: http://www.raosoft.com/samplesize.html) 17 Table 4 Survey Overview Fatick Kaffrine Kaolack Dakar Total MBSP producers 206 245 199 0 650 Neighbor producers 68 60 61 0 189 Producer organizations 70 82 60 0 212 Hubs 5 6 7 0 18 Processors 8 7 10 18 43 Women’s group processors 6 13 17 18 54 Kiosks 5 9 10 11 35 Restaurants/hotels 2 1 1 7 11 Other buyers and sellers 23 20 54 61 158 Buyer and seller surveys and census The second type of survey (and census) targeted millet buyers and sellers, including those at producer organizations, hubs, processors, kiosks and others. All these different types of respondents were issued the same questionnaire, with questionnaire logic designed to customize the questionnaire for each respondent type. Producer organizations and hubs– The producer organization survey randomly sampled 220 of the approximately 500 producer organizations supported by MBSP. This sample size is consistent with a 95 percent confidence level and 5 percent margin of error. The list of producer organizations was created from the list of MBSP producers, which, as discussed previously, was incompletely and inconsistently labelled. Even when furnished with 27 replacement producer organizations (a total of 247 producer organizations), the survey team was only able to find and interview 212 producer organization presidents. This corresponds to a slightly higher margin of error of 5.1 percent. Hubs - Hubs are an MBSP supported organization one level above producer organization. They serve as an organization of organizations. Presidents of all 18 MBSP supported hubs were interviewed. Processors – The evaluation team conducted a census of the two types of MBSP supported millet processors. All 43 project supported unit processors were interviewed and 54 out of the 64 women’s group processors listed by the project were interviewed. The survey team found that the other 10 were no longer in business or not contactable. Women’s group processors are generally smaller and less commercially oriented than unit processors. Other buyers and sellers – Other buyers and sellers include MBSP supported kiosks (small retail structures), restaurants and hotels, and other buyers and sellers (e.g. retail outlets, wholesalers, exporters and others). Only kiosks were directly supported by MBSP. Other buyers and sellers’ types of respondents were identified on a referral basis. At each hub, unit processor, and women’s processor group, the survey team asked who their buyers were and then contacted them to arrange interviews. In total, 35 kiosks, 11 restaurants and hotels and 158 other buyers and seller respondents were interviewed. Since we do not know the population of these types of business, the results from this group of respondents cannot be considered statistically representative. 18 Analysis and baseline data Quantitative data was analyzed using Stata. Unfortunately, data from the baseline study was not available. As a result, the evaluation team was not able to make before and after comparisons beyond the relevant indicator results included in the baseline report. Anticipating this potential issue, the evaluation team included questions to assess respondent status at program inception five years ago. Recall over such a long period is obviously fraught with error and should be interpreted with caution. Methodological limitations This evaluation has a number of significant limitations. The previous sections of the methodology discussed all except one. We summarize the limitations here. 1. The MBSP did not have a complete list of participating producers. Therefore, as the surveys sample was taken from an incomplete list, the survey’s results are not representative of the entire population of MBSP producers. 2. Due to record keeping inconsistencies, it was not possible to randomly sample producers. As a consequence, data on MBSP producers is not based on a statistically representative sample. 3. The final evaluation is unable to test differences between baseline and endline for statistical difference. As a result, this evaluation only uses the few relevant figures from the baseline study report as points of comparison. This limitation is because baseline study’s data was not available. The evaluation team attempted to compensate for this limitation by asking respondents to remember back five years ago to before MBSP started but memory over such a long period is prone to large errors. As a result of this limited quality baseline data, this evaluation has little to say empirically about the differences between MBSP beneficiaries before and after MBSP. 4. This evaluation does include a group of neighbor producers and makes comparison between them and MBSP producers. However, this neighbor farmers group is not large enough to support statistical tests of differences. Additionally, since they are neighbors, they likely benefited from MBSP through knowledge of new production practices and improved market access. 5. The last millet harvest in the project region was in and around November 2019. Therefore producers still had not sold all of the millet they will sell this season at the time of the survey. Sales figures reported here combine what has been sold so far with what producers expect to sell from this latest harvest. This projection is unlikely to be accurate. On the other hand, the buyers and sellers’ surveys ask respondents about their sales in the past season (2018/2019). While these sales are complete, there may be some error in recall from a year ago. 3. Findings In this section, we describe the MBSP’s achievement compared to its targeted results and key indicators. We also answer the evaluation questions related to the relevance, effectiveness, efficiency, sustainability, and impact criteria. 3.1 Project Performance This evaluation’s scope of work requires that the evaluation “assess project performance in meeting the end of the project activity and result targets.” In this section we review the extent to which the project achieved its goals and met its targets in reference to its results framework and indicators. 19 However, we begin the section with the MBSP had four innovative achievements that are not measured in the project’s standard or custom indicators. These achievements were identified through key informant interviews and the final evaluation’s questionnaire, and support for them is provided in sections indicated parenthetically below and throughout the report. These achievements are: 1. The establishment of 18 millet hubs – The MBSP established 18 millet hubs. These hubs serve as local, rural centers of the millet trade. For producers, the hubs provide access to quality seeds and fertilizer at a reasonable price, secure storage facilities, access to new and often large customers, and millet cleaning services. The hubs also regulate the quality of millet they store (See Section 3.3.2 Quality Adoption). High quality millet has grains that are of consistent color, sorted by grain size, and is free of foreign materials such as stones. The hubs’ monitoring of quality serves as the equivalent of a producer certification process. As one president of a hub put it, “When customers visit, they see what’s in storage. The quality is better here, meaning that processors get more flour per kilogram.” For buyers, such as processors, the hubs are a reputable seller that can provide millet in large quantities and at a guaranteed level of quality. Quality millet provides more value per kilogram and saves buyers money in terms of labor required to clean and process millet themselves. As this report discusses, the hubs are not all entirely operational. Still, in many instances they provide crucial services to millet producers, and serve as an important example to government and local businesses of how proper management leads to higher yields per hectare for farmers and greater volumes of flour per kilogram for processors (See Section 3.2.1 Alignment with Government of Senegal Strategy among others). 2. The implementation of quality and branding standards – The MBSP introduced standards and trust throughout the value chain. Fair grading methods and higher prices for quality allowed the program to build trust between producers and hubs and hubs and processors. Clean packaging with the seller’s contact information printed on the label built trust between hubs and processors and processors and consumers. Importantly, the MBSP introduced a creative and effective means of manufacturing packaging on a small scale. This small-scale form of packaging allows for local business people to express their creativity, design packaging for their particular products, and frees them from having to work with large corporations. The introduction of these standards and branding practices aimed to convert millet from a porridge eaten during times of scarcity to a nutritious substitute for rice and a desired product (See section 3.3.4). 3. The creation of a new model for financing producer inputs – The project worked with banks, processors, producers, and input providers to come up with new models for financing inputs. Certainly, producers and hubs continue to struggle to obtain sufficient financing. Still, the MBSP introduced a model, discussed in section 3.3.6 Linkages Among Project Participants that will continue to grow and finance future agricultural endeavors. 4. The distribution of sales kiosks – The MBSP raised awareness about the benefits of eating millet and matched that new awareness with new, visible locations to buy ready-to-eat and processed millet. The kiosks the project introduced provide new sales outlets for small processors, and an easy channel for consumers to access products (See the Expanded Trade section under SO2 – Expanded Trade of Agricultural Products (Domestic, Regional, and International). The evaluation team believes that these four innovations are among the project’s most important achievements, though they aren’t captured in the results or indicator framework. They are discussed in greater detail throughout the finding’s section of this report. 20 3.1.1 Extent of Goal and Intermediate Results Achievement Evaluation Question 1: To what extent have project goals been achieved? In this section, the report describes the extent to which the project achieved strategic objectives and intermediate results. We rate the extent of achievement as described in Table 5 below, followed by an explanation of the rating and the details of specific final results. Table 5 Extent of Achievement Status Definition Achieved/Exceeded Met or exceeded targets Mostly achieved Nearly met targets, and those missed were because of clear extenuating circumstances Somewhat achieved Achieved 40 - 70 percent of the target Unachieved Project resulted in less than 40 percent of the target SO1 - Increased Agricultural Productivity Degree of achievement: Achieved The evaluation team measured achievement under this strategic objective by:  Seeking to understand if MBSP increased farmers’ millet yields (or mitigated losses) and improved quality.  Calculating whether or not MBSP met its results’ targets under the strategic objective. Increased Yields The evaluation team believes that farmer yields are larger than they would have been in the absence of MBSP’s efforts given sparse rainfall that fed the millet crop in 2016 and 2019. The evaluation team makes this judgement based on three sources:  Producers’ survey responses about yield: While producers reported lower yields than were recorded at baseline (Figure 6), farmer’s who participated in MBSP reported higher yields than those that did not participate in MBSP.  Producers reported overall yield increased in interviews: Producers that the evaluation team interviewed reported that overall their yields had increased because of MBSP.  Producers report low rainfall: Rainfall, a variable that MBSP couldn’t control, was responsible for yield declines in 2019.  Results targets were mostly met: The MBSP met 7 of the 12 results indicator targets that measured achievement under this SO. The evaluation team believes that the five targets related to yield that were not achieved were a result of this phase’s focus on increasing trade, whereas the MVCP, the preceding project, focused on agricultural production. In response to the final evaluation’s survey, the MBSP participating millet farmers reported that they produced a mean yield of 528 kg/ha in 2019 compared to the 657 kg/ha estimated at baseline (Soterco Group, 2016, p. 48). The reader should note that the baseline study used yield averages calculated by the Ministry of Agriculture’s Division of Analysis, Forecasting and Agricultural Statistics (DAPSA) rather than a survey of project participants. 21 At final evaluation, yield was highest in Kaolack that produced an average of 577 kg/ha, and lowest in Kaffrine, which produced 498 kg/ha on average (Figure 6). To understand if the MBSP mitigated farmer losses from climate conditions, the evaluation also surveyed neighbor farmers who did not participate in a producer organization. The evaluations results indicate that neighbor farmers produced an average of 475 kg/ha, 53 kg less than farmers that participated in the MBSP. While the sample of neighbor farmers was too small to be statistically significant, neighbor farmers’ responses provide an interesting point of comparison to the MBSP’s direct participants’ experience. Figure 5 Change in Yield between Baseline and Final Evaluation However, the decrease in yield between baseline and final evaluation may have more to do with lack of rainfall then any variable that MBSP targeted. The evaluation team asked 14 randomly selected farmers through key informant interviews if farmers believed that their yields per hectare were higher than before the program. All 14 answered that yields were higher in general. When asked how they knew that their yields were higher, farmers mentioned that they had achieved yields during the program that weren’t possible before the program. One producer that is a member of a producer organization in Kaffrine said, “before CLUSA, my yields per hectare never exceeded 350 kg but with the innovations brought by the project, I was able to reach 650 kg per hectare.” Another farmer with a producer organization concurred, stating “before the project, I harvested 500 kg per hectare and now I'm at one ton per hectare.” Several farmers also mentioned reducing the number of cultivated hectares dedicated to millet because of yield increases. A farmer from the a producer organization in Fatick noted, “I cultivated four hectares before the project, but since we started working with CLUSA, I cultivate three hectares with more yield than before.” “Before CLUSA, my yields per hectare never exceeded 350 kg, but with the innovations brought by the project, I was able to reach 650 kg per hectare.” ~ MBSP Producer in Kaffrine 22 The final evaluation’s producer questionnaire asked farmers “Has your millet harvest per hectare changed since 2014?” Of the 650 farmers that responded to the questionnaire, 332 answered this question. Of those 119 (44% of those that answered the question) indicated that their yield had increased since 2014. Farmers who indicated that their yield had increased were then asked “Which of the following reasons do you believe describe why your millet harvest has increased?” The options and proportion of farmers that chose each option8 is shown in figure 7. The most common reasons farmers reported for the increase in yield were; improvements in their skill at preparing the soil (74%), better access to seed (seeds are more affordable and easily available) (65%) higher quality seeds (seeds are from varieties that are resistant to drought or pests and/or are more productive) (32%), and better harvesting practices (25%). Figure 6 Reasons for Yield Increases (n=119) Another reason that farmers perceived an increase in yield is reduced post-harvest loss. Thirty-nine percent (39%) of MBSP producers that responded to the final evaluation survey said that their post￾harvest loss had decreased since 2014. The mean loss reported by survey respondents was 5.3 percent. The data above describes farmers perception of changes in their millet yield since 2014. In KIIs, the evaluation team asked farmers how 2019’s harvest compared to 2018’s harvest. Of the farmers that the team interviewed, 10 reported the difference in the volume of their harvest between 2019 and 2018. On average those farmers reported that this year’s harvest was around 50 percent of last year’s harvest. Farmers reported that insufficient rainfall was the reason for 2019’s poor harvest. 8 Note that the possible responses were the same used in the mid-term evaluation. 23 SO1 Indicators and Intermediate Results The project measured increases in agricultural productivity using three indicators; the volume of commodities sold by project beneficiaries, the value of those sales, and the number of individuals receiving short-term agricultural productivity or food security training. The project reports through its semi-annual Food Aid Information System (FAIS) reports that it exceeded targets in terms of volume of commodities sold by beneficiaries and value of those commodities sold over the life of the program. Volume of commodities sold exceeded the project’s target by 9 percent, and value exceeded the target by 7 percent. The project also appears to have exceeded its target for individuals trained in agricultural productivity or food security, though it is difficult to know exactly how many received training because of the way the project’s data is disaggregated. The training data under this result is disaggregated according to the following categories9 :  Continuing  New  Female  Male  People in Civil Society  People in Firms  People in Government  Producers As shown in Table 6 below, the project’s targets for the sum of male and female and the sum of continuing and new are equal, while the sum of the other categories is slightly higher. In the project’s record of achievements under this indicator, the other disaggregation categories do not similarly sum up to the total. This report discusses issues with the project’s data management and record keeping under the section 3.4.2 Staffing, Management, and Resource Allocation. Under a few disaggregation categories for the training indicator the project missed its target. For example, the target for training people in Government was 1,000 individuals, and only 143 were achieved. The project notes in its October 2018 FAIS report, “This indicator is overestimated and the project will not be able to reach it.” As another example, people receiving continuing training fell about 17 percent short of the intended target. Overall though, the target was exceeded. Table 6 Number of individuals receiving short-term agricultural sector productivity or food security training as a result of USDA assistance by category Category Baseline Target Achieved Difference Continuing and New 3,851 35,945 40,687 + 4,742 Female and Male 3,851 35,945 42,398 + 6,453 Producers, Government agents, people at firms, civil society workers, and producers 3,905 36,707 43,883 +7,176 9 Complete results tables with disaggregated data are included in this report as Annex D. 24 While the project data regarding the number of people trained is somewhat unreliable, project participants interviewed during the final evaluation indicated that they had received training, and it was often what they appreciated most about the project. In 18 of 21 producer and producer organization president interviews, respondents emphasized that composting training as having increased their yield significantly. Other trainings were also highly valued. One president of a producer organization put it this way, “Composting and good practices (training) was really appreciated by all in this area. Previously we left 50 to 60 cm between plants, but now we know to leave 90 cm10.” This finding also indicates that the MBSP took action in response The mid-term evaluation’s recommendation that “NCBA CLUSA reinforces its training in composting, encouraging and supporting the use of compost to fertilize the soil (JE Austin Inc., 2018, p. Recommendation 10).” FFPr 1.2 - Increased Use of Improved Agricultural Techniques and Technologies Degree of achievement: Somewhat achieved Evaluating success under this result is difficult because the targets and accounting for results achievement is inconsistent. This result was measured using four indicators:  Number of hectares under improved techniques or technologies as a result of USDA assistance. Under this indicator the project reports in its semi-annual FAIS performance reports that it missed its target for the number of total hectares with at least one improved technique or technology, achieving 29,962 hectares against a target of 43,488, a shortfall of 31 percent11. A portion of this shortfall could be an initial underestimation of farm size, as the project exceeded its target for number of people who adopt new techniques or technologies. The project was most successful reaching farmers with crop genetics improvements (exceeding the target by 14%), in the form of new seed varieties and climate mitigation techniques (exceeding the target by 13%). The project was least successful in meeting soil fertility targets (under target by 56%), an odd result considering, as mentioned above, that in qualitative interviews, training on soil fertility was by far rated the most successful by respondents.  Number of individuals who have applied improved farm management practices The project improved farm management practices of 14,350 individuals against a target of 23,268, a shortfall of 38 percent. The project also sought to improve practices for an equal number of men and women under this indicator (11,634 of each). The project reports in its FAIS reports that it served 6,254 men and only 2,664 women. The disaggregation shows that more men were served. That the total of women plus men does not equal total individuals means that the disaggregation may not represent the true imbalance between men and women.  Number of individuals who have applied new techniques or technologies as a result of USDA assistance The project was most successful under this indicator. It brought new techniques and technologies to 16,149 people, exceeding the project’s target of 15,266. However, it still served many more men than women. The project exceeded its target for men in this area by 42 percent, while underserving women by 36 percent. MBSP was most successful in bringing new pest management and soil conservation and fertility techniques, which as mentioned tracks commentary by producers in interviews. 10 Producer organization interview 20 December 2019. 11 The number of targeted hectares with at least one new technique or technology is 1,009 hectares less than the targeted number of new and continuing hectares. Those numbers should match. 25  Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community-based organizations (CBOs) that applied new technologies or management practices as a result of USDA assistance. MBSP was mostly successful in serving private enterprises with new technologies and management. As captured by the MBSP’s FAIS reports, the project reached 703 organizations against a target of 717 organizations. It was most successful reaching private enterprises, exceeding its target by 16 percent. However, it only reached 40 percent of the trade and business associations and 50 percent of the women’s groups that the project intended to reach. This result may reflect an over estimation of the numbers of women’s groups and trade associations available rather than a failure of effort. Table 7 Increased use of improved agricultural techniques and technologies Indicator Life of Project Target Achieved Percentage of Target Achieved Number of hectares under improved techniques or technologies as a result of USDA assistance. 43,488 29,962 69% Number of individuals who have applied improved farm management practices 23,268 14,350 62% Number of individuals who have applied improved techniques and technologies 15,266 16,149 105% Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community￾based organizations (CBOs) that applied new technologies or management practices as a result of USDA assistance 717 703 98% FFPr 1.2.1 - Increased Availability of Improved Inputs Degree of Achievement: Somewhat achieved The indicator measuring this result is “Value of inputs and support services sold by agricultural input suppliers to farmers in US dollars.” MBSP estimated the value achieved at $1,754,024 against a target of $3,000,000. Difficulties bringing input suppliers into the project may have contributed to missing this target. In its 2017 program report, MBSP notes that “(input) suppliers are very private and difficult to mobilize.” The evaluation team interviewed two major input suppliers as part of the evaluation, ICS and OCP. ICS noted 26 during an evaluation interview that it had engaged with the project early on, but after much deliberation decided not to participate. ICS chose not to participate because it could not serve the project and earn a profit. OCP did not officially sign a partnership agreement with the project until July 2018 following a period of conducting its own due diligence and negotiating with the project’s leadership to arrange for the logistics of understanding farmers needs and delivering inputs. 12. FFPr 1.2.3 - Increased Use of Financial Services Degree of achievement: Mostly Achieved The project reported through its FAIS reports that it saw 40,238 people receive financial services as a result of its efforts against a target of 40,174. An important achievement that the evaluation discusses in more detail in 3.3.6 Linkages Among Project Participants. Also, importantly, men and women were nearly as likely to receive financial services because of the project, with 20,671 women and 19, 567 men benefiting. However, MBSP fell short in the number of loans distributed, seeing 10,256 disbursed against a target of 13,129. FFPr 1.4.1 - Increased Capacity of Government Institutions Degree of achievement: Exceeded The project exceeded its target under this result. Increase in capacity of government institutions was measured through results under one indicator, the number of millet producers trained by government extension agents, who were supported by MBSP. The MBSP estimated that it trained 17,513 agents against a target of 12,285, exceeding the target by 41 percent. FFPr 1.4.4 - Increased Capacity of Key Groups in the Agriculture Production Sector (Coops and Small Shareholder Farmers) Degree of achievement: Uncertain. The indicator that measured success under this result was “number of producer organization or cooperatives that had certified extension officers or agents by the end of the project.” The target under this indicator was 199 organizations. The project reported that it had served 99 organizations under this result as of its October 2019 report. The project’s leadership told the evaluation team that the remaining organizations were trained by the end of 2019 and the results will appear in the final FAIS report. However, the MBSP narrative reports include descriptions from the 2017 report that say, “A total 362 PO representatives were trained on extension techniques; 292 from Kaolack and 70 from Fatick. In addition, 61 PO members became certified extension agents for their POs. The training of PO extension agents through DRDR is a continuous activity,” and from FY 2018 “A total of 162 extension workers (111 and 50 from Fatick and Kaffrine respectively) from producer organizations (POs) are preparing for a training series on advanced extension work techniques.” The evaluation team believes that there is discrepancy in achievement as reported through FAIS reports that may relate to the project’s struggles managing data. 12 The “CONVENTION DE PARTENARIAT EntreUSDA|Projet des Services des Entreprises du Mil (USDA|PSEM)Et Office Chérifien des Phosphates (OCP Sénégal SA)” Was signed in July 2018. 27 SO2 – Expanded Trade of Agricultural Products (Domestic, Regional, and International) Degree of achievement: Achieved Trade of millet and millet products has expanded under the MBSP. The second phase of the project was most focused on this strategic objective. The evaluation team measured achievement under this strategic objective by:  Seeking to understand how trade volume and value has changed for the MBSP participants.  Calculating whether or not the MBSP met its results’ targets under the strategic objective. Expanded Trade The MBSP expanded trade for actors in the millet sector. The evaluation team makes this judgement based on four sources:  Value of sales by project beneficiaries: from 2015 to 2019 as reported by the project.  Private enterprise and organizations responses to survey questions about sales values: The evaluation reports the value of sales produced by producer organizations, hubs, processor firms (including women’s groups processing firms), and kiosks.  Producer survey responses about sales, including volume and value: The evaluation asked millet producers if they sold a portion of their harvest, the volume they sold, and the value of their sales. The report compares these values to data available from the project’s mid-term evaluation to show an increase in trade.  Interviews with millet sector sellers: The evaluation team interviewed millet product sellers in producer organizations, hubs, processors, and kiosks.  Results targets were met: The MBSP met all 6 of its results indicator targets under SO2. When asked about sales history, 88.7 percent of sellers that participated in the MBSP reported that sales revenue is higher than at the start of the program. When asked why revenue increased, 63.8 percent reported that sales increased because of the improvement in millet quality. 49 percent pointed to better customers, meaning customers that were more discerning about the product and were willing to pay a higher price for quality. These answers point directly to two of MBSP’s initiatives; increasing millet quality and raising market awareness. Millet sellers also pointed increases in the quality of millet, in the form of bigger grains and less debris or fewer bad grains in millet purchases leading to higher yields per kg (see quality discussion in the Effectiveness section of this report). Other significant reasons for revenue increases included 43.3 percent indicating that higher revenues were a product of millet market growth. included the increase in higher value millet products and the increase in millet supply. 28 Figure 7 Sellers’ Reasons for Revenue Increase Table 8 below shows the total value of trade in 2018/2019 for MBSP supported organizations and businesses based on the final evaluation survey. The columns also show the percentage of total volume of sales across all supported actor types and the mean value per actor. The last column shows the percent of producer organizations and hubs that reported selling millet in the past year. The value creation at the hub and processor level is notable, with processors capturing nearly 60 percent of market value. Table 8 Value of Sales by Respondent Type (2018/2019) Total % of total value Mean % selling By Seller type (USD) Producer organizations 731,097 10.71 4,697 31.1 Hubs 484,931 16.76 37,265 72.2 Processors 1,706,949 58.98 37,949 Womens processing groups 475,129 13.55 7,843 Kiosks NA 0.00 13,658 Total 3,398,106 100.00 29 Table 9 shows the value various important millet products contribute to the total sales value. Table 9 Products market share and value (USD) Products Sample Total % of value Mean % selling Thiakry 486,951 8.51 2,308 39.7 Thiere 474,862 8.30 2,683 33.3 Sanxal 478,680 8.37 1,986 45.4 Araw 325,103 5.68 1,617 37.9 Ngourbane 88,465 1.55 2,328 7.2 Farine fortifee 442,087 7.73 6,501 12.8 Farine Simple 51,141 0.89 1,826 5.3 Unprocessed 3,126,432 54.65 32,308 18.3 Other 247,190 4.32 41,197 1.1 Total 5,720,911 100.00 The final evaluation found that 42.7 percent of the MBSP producers sold at least a portion of their harvest, compared to 34 percent of their neighbors that didn’t participate in the project. The mean volume sold by MBSP producers was 354 kg, realizing a mean value of $326 (Figure 9). The neighbor producers that the evaluation surveyed reported that they sold an average of 223 kgs and realized a mean value of $236. Figure 8 Mean Producer Sales Volume The MBSP participants that the evaluation team interviewed agreed that their revenue rose as a result of the project. A respondent that co-managed a processing firm said that the firm produced 400 kg a month, and now they produce over 1000 kgs/month. The increased production and reputation that the firm has developed allows FARK to sell to universities and restaurants and to reach customers from Dakar to the Casamance. The manager of a cooperative said that training in recognizing millet quality allowed the 30 cooperative to increase its quality, which led to increases in revenue. A large distributor in Dakar that reaches export markets in Europe said that after one year with the MBSP, revenue increased from CFA 120 million to CFA 150 million. SO2 Indicators and Intermediate Results SO2 is “Expanded Trade of Agricultural Products (Domestic, Regional, International).” The project’s indicator directly connected to SO2 measured the “total number of individuals benefiting directly as a result of USDA assistance.” It is difficult to know how many individuals the project benefited with expanded trade. In its results framework, the project targets do not match. Targets include 63,426 new and continuing individuals, 69,376 female and male individuals, and 437,277 individuals in general. The project’s achievement totals 212,799 new and continuing individuals, 82,401 male and female individuals, and 282,768 individuals in general. Improved Capacity of Key Organizations in the Trade Sector (Processing Organizations and Trade Associations) Degree of achievement: Achieved The indicator measuring this result was “Number of producer organizations/groups with marketing specialists.” The project sought to build the capacity of 110 producer organizations with marketing specialists and the project reports 108 organizations had specialists as of September 2019. Increased Value Added to Post Production Agricultural Products Degree of achievement: Mostly Achieved The project sought to create 714 new jobs as a result of USDA assistance, and it reports that it created 545 new jobs. More than 60 percent of these new jobs went to women. Improved post-harvest infrastructure Degree of achievement: Mostly Achieved Two indicators measured this result:  The number of agriculture producers in target regions that have access to effective off-farm storage for their product.  Total increase installed storage capacity as a result of USDA assistance The project ensured that 10,894 people had access to installed storage compared with a target of 12,404, and increased new and refurbished storage to 16,091 square meters against a target of 16,982. Increased Access to Markets to Sell Agricultural Products Degree of achievement: Exceeded The project records in its FAIS performance reports that it created 783 linkages to markets against a target of 599. Improved Linkages Between Buyers and Sellers Degree of achievement: Mostly Achieved The project saw 281 agreements signed between buyers and sellers against a target of 304. 31 3.1.2 Activities and Targets Evaluation Question 2: Which activities have been successful in meeting their targets? Evaluation Question 3: Which activities failed to reach their targets and why? MBSP implemented activities under nine components. Each component’s success was measured comparing output under each indicator to the target set for that indicator in the modification 4 to USDA’s award agreement with NCBA CLUSA. In this section, this report discusses activities performance against targets and the reason for that performance. The data used to inform this section comes from the project’s semi-annual FAIS performance reports. This section also utilizes other project documentation and the results of interviews to put the data in context. Capacity Building: Agricultural Extension Agent Services, Producers, and the Private Sector Activities that met or exceeded targets Under this project component, the project exceeded its target in the number of demonstration of plots created and the number of producer organizations members trained in extension skills. MBSP met its target in the number of input demonstration fairs held. Table 10 Demo plots, fairs, and PO extension training Indicator Life of Project Target Achieved Proportion of Target Achieved Number of demonstration plots created 360 427 1.19 Number of input demonstration fairs 12 12 1.00 Number of Producer Organization (PO) members trained in extension skills 360 567 1.58 Activities that missed targets MBSP trained 199 input suppliers, failing to meet its target of 310. Table 11 Input suppliers trained Indicator Life of Project Target Achieved Proportion of Target Achieved Number of input suppliers trained in extension skills by input firms 310 199 0.64 The project missed this target because the project’s first two years were spent identifying and building relationships with input suppliers. The project did not train input suppliers in 2016 and 2017 though it had targets of 50 and 100 firms trained in those years. By 2018, MBSP had identified and built relationships with input suppliers, particularly OCP, and trained 199 individuals in extension services in that year, exceeding its 2018 target of 100 suppliers trained in extension services. 32 Financial Services: Facilitate Agricultural Lending MBSP had four indicators under this activity component; number of individuals receiving financial services, number of loans disbursed, value of agricultural and rural grants provided, and value of loans provided. The project exceeded its targets under each indicator. Importantly, the project also met its gender equality targets, ensuring even distribution of financial services among men and women. Activities that met or exceeded targets Table 12 Financial services activities that met or exceeded targets Indicator Life of Project Target Achieved Proportion of Target Achieved Number of individuals receiving financial services as a result of USDA assistance (female) 5300 17,981 3.39 Number of individuals receiving financial services as a result of USDA assistance (male) 2500 15,193 6.08 Number of loans disbursed as a result of USDA assistance 800 10,129 12.66 Value of agricultural and rural grants provided as a result of USDA assistance (USD) $834,500 $931,609 1.12 Activities that missed targets Financial services coming from lenders was less successful than financial services in general. While the numbers of loans exceeded targets, the value of those loans was smaller than expected. The total value of loans disbursed was 78% of the project’s target. The value of loans went disproportionately to men. Women received approximately $51 per loan, whereas men received about $75 per loan. In project reporting, the MBSP noted that high participation in Savings and Internal Lending Communities (SILCs) was not producing as much loan value as expected. In 2017, the project noted “Despite this high participation in the SILCs, the number and value of loans are low.” In its Fiscal Year 2017: October 2016 – March 2017 Narrative Report, the project notes one reason for higher numbers of loans going to men. The report states that “there (were) various protocols in preparation with particular MFIs in regards to processors (most of who are women). Banks and MFIs are hesitant to lend to processors because it is not a well-known sector,” and that “the number of loans for men who are mostly producers, is going to greatly increase, given the agricultural credit campaign (which occurs in the months before the rains come- the second half of the FY17 reporting period).” The reporting of this target underplays some MBSP financial services innovations, which are discussed in section 3.3.6 Linkages Among Project Participants. 33 Table 13 Financial services activities that missed targets Indicator Life of Project Target Achieved Proportion of Target Achieved Value of loans provided as a result of USDA assistance (total) $2,638,192 $2,055,730 0.78 Value of loans provided as a result of USDA assistance (female) $1,392,129 $921,511 0.66 Value of loans provided as a result of USDA assistance (male) $1,246,063 $1,134,219 0.91 Infrastructure Post-harvest Handling and Storage Activities that met or exceeded targets The post-harvest handling and storage activity component was monitored using two indicators, the total increase in new installed storage capacity and the total increase in refurbished installed storage capacity The project exceeded its target overall in creating storage capacity. The project created 13,672 square meters in total storage capacity against a target of 12,500 square meters. The project was more successful at creating new storage capacity than refurbished. Infrastructure: Post- Harvest Processing Activities that missed targets The project measured its results in this activity using one indicator, the number of agricultural and rural grants provided as a result of USDA assistance – Only post-harvest processing infrastructure. MBSP only provided 40 percent of the targeted number of grants, missing its target each year. Table 14 Post harvest processing activities that missed targets Indicator Life of Project Target Achieved Proportion of Target Achieved Number of agricultural and rural grants provided as a result of USDA assistance (USD)- ONLY Post harvest processing infrastructure 324 131 0.4 34 Inputs: Develop Agro-dealers and Input Suppliers Activities that missed targets MBSP measured achievement under this activity component through the number of agro-dealers developed into networks. The project recorded that it developed 221 agro-dealers into networks against a target of 250. As reported previously, delays in bringing input businesses into the project may have contributed to missing this target, though clearly much work was done to nearly meet it. Market Access: Facilitate Buyer-Seller Relationships MBSP tracked its success creating buyer-seller relationships by counting the volume of commodities sold by beneficiaries and the number of jobs the project created. Activities that missed targets The project nearly met its target in terms of volume of commodities sold, delivering 91 percent of the project target. However, it only created about 71 percent of the jobs the project sought to create. The project did however meet its targeted number of created jobs for women. Table 15 Market access activities Indicator Life of Project Target Achieved Proportion of Target Achieved Number of jobs attributed to USDA assistance (female) 310 337 1.09 Number of jobs attributed to USDA assistance (male) 425 191 0.45 Volume of commodities (metric tons) sold by project beneficiaries 21,454 19,596 0.91 Training: Improved Agricultural Production Techniques The indicators under this section are the same as those reported under FFPR 1.2 Increased Use of Improved Agricultural Techniques and Technologies in the previous section. However, the indicators differ somewhat. For example, when MBSP’s semi-annual reports are totaled for the length of the project, the reports show under the results section 29,962 hectares under improved technologies against a target of 43,488. However, under the activities section, the reports show 14,658 improved against a target of 51,283. Whichever accounting one accepts, the target was missed by a considerable margin. Similarly, under the results indicator MBSP reported that it reached 14,350 individuals with improved farm management practices against a target of 23,268. Under the activity component the project reports reaching 13,500 individuals against a target of 25,218. Also, for the indicator “Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and CBOs that applied new technologies or management practices,” MBSP reported reaching 703 organizations against a target of 713 under the result indicator. Under the activity component, the project reports reaching 1,550 organizations against a target of 350. It is unclear which is correct. We 35 report targets achieved and missed as reported under the activity component below. The full disaggregated tables are included in Annex D. Activities that met or exceeded targets The project reports that it reached 42,063 individuals with agricultural sector productivity training. Table 16 Improved agriculture activities that met targets Indicator Life of Project Target Achieved Proportion of Target Achieved Number of individuals receiving short- term agricultural sector productivity or food security training as a result of USDA assistance (Male and Female) 36,200 42,063 1.17 The project reports that it served 1523 private enterprises, producer organizations, and other types of organizations. The evaluation team does not believe this figure is accurate. The project did maintain a list of producer organizations. It did maintain a partial database of project beneficiaries that included the name of the organization that the beneficiary belonged to. However, this list was riddled with errors (e.g. the same organization spelled different ways, the same organization with different names, the same organization with accent marks in the name slanted in different directions). The evaluation team counted 1,779 distinct organization names in the database. After an initial attempt at cleaning the names by applying consistent spelling, the list was reduced to 604 organizations. The actual achievement is closer to the target than reported by the program, so likely successful. Table 17 Improved agriculture activities that met targets (continued) Indicator Life of Project Target Achieved Proportion of Target Achieved Number of private enterprises, producers’ organizations, water users associations, women's groups, trade and business associations, and CBOs that applied new technologies or management practices 450 1523 3.38 Activities that missed targets Activities that missed their mark include number of hectares under improved techniques and number of individuals who have applied new techniques and technologies. The evaluation team believes this result may reflect poor data management rather than underserving individuals or missing targets related to hectares. This reported result implies that over 42,000 individuals received training, as reported above, 36 but only around 25 percent of them applied what they learned. Producers that the evaluation team interviewed were universal and specific in their praise for the MBSP’s trainings and demonstrations. Missing this target, is likely related to the project’s data management short comings rather than the low success rate that the measured achievement implies. Table 18 Improved agriculture activities that missed targets Indicator Life of Project Target Achieved Proportion of Target Achieved Number of hectares under improved techniques or technologies as a result of USDA assistance (continuing and new) 51,283 14,658 0.29 Number of individuals who have applied new techniques or technologies as a result of USDA assistance 15,000 10,760 0.72 Training: Improved Marketing and Branding The project measured success in this activity component with one indicator “Number of service provider/ input supplier employees trained in marketing techniques.” Activities that missed targets The project achieved less than 20 percent of its target in this area. The result is likely associated with the amount of time the project spent building relationships over its first two years, a press to train input suppliers in 2017, and its intense efforts to build buyer-seller networks in 2018, as described in the MBSP Semi-Annual Narrative Reports (NCBA CLUSA, 2017) (NCBA CLUSA, 2018). Table 19 Improved marketing and branding activities Indicator Life of Project Target Achieved Proportion of Target Achieved Number of service provider/ input supplier employees trained in marketing techniques 525 95 0.18 37 Training: Post-Harvest Handling and Processing This activity component’s achievement was measured through two indicators:  Number of individuals who have applied improved farm management practices  Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community- based organizations (CBOs) that applied new farm management practices Activities that met or exceeded targets The MBSP exceeded both indicators as shown in the table below. Table 20 Post-harvest processing and handling activities Indicator Life of Project Target Achieved Proportion of Target Achieved Number of individuals who have applied improved farm management practices 2375 5226 2.2 Number of private enterprises, producers’ organizations, water users associations, women's groups, trade and business associations, and community￾based organizations (CBOs) that applied new farm management practices 350 405 1.16 3.2 Relevance The relevance section of the evaluation discusses the alignment of project activities with the government of Senegal’s priorities and strategy, the extent to which the project was responsive to the needs of participants, the project’s alignment with USDA and US Government goals, objectives, and strategies, and the MBSP’s alignment with the award agreement. 3.2.1 Alignment with Senegal Government Strategy Evaluation Question 4: Are the project activities in alignment with Senegal Government priorities and strategies? The MBSP is in close coordination with government officials at the national, regional, and local level. The evaluation team interviewed officials in the ministry of agriculture, the governors of Kaffrine and Kaolack, a governmental research agency, and local community leaders. The Government of Senegal’s ministries and national agencies struggle with setting priorities and creating a unified strategy13. While the government has produced the National Strategy for Food Security and 13 See McGovern-Dole International Food for Education and Child Nutrition Project, Mid-Term Evaluation (2017) pg. 38 for an example of the Ministry of Education’s struggles in this area. 38 Resilience 2015 – 2035, which describes a plan for reinforcing millet’s role in strengthening national food security, the government lacks a detailed and comprehensive strategy such as the one provided by MBSP. A government employee agreed, stating “The government may have some programs but likely won’t have the same resources for millet as CLUSA project.” Additionally, the government struggles with managing many international and domestic partners that work in overlapping areas. The official in the Ministry of Agriculture that the team spoke with mentioned that the government manages partners from the US, Europe, Japan and China. He noted that “One weakness is that the government has many partnerships in Senegal that are not in harmony in their way of working. The ministry needs a platform to follow all partners efforts.” The MBSP demonstrated an understanding of the government’s struggle with tracking partners’ work. The project communicated to government officials about progress and successes, and also about areas in which the project was struggling. The same official reported that the project communicated with him monthly, was clear about difficulties, and “is a good collaborator.” At the regional levels, regional governmental employees demonstrated a clear understanding of the project and were up to date on its progress. One employee credited the project with improving food security in the region, stating, “the project taught people how to cook a larger variety of dishes with millet, including a type of pizza. Now they do more than millet and fish. The producers have been organized and people eat millet for their health.” Likewise, another regional government employee had clear opinions on the importance of the project and was up to date on its progress. The employee told the evaluation team, “the project builds self-reliance for producers. It encourages consumption of millet, improved cultivation techniques, and new varieties of millet. It has built linkages between customers and producers. It promoted transformation of millet into new product and encouraged consumption through supermarkets. Consumption is important because the nutritional content of millet is different from rice.” 3.2.1 Responsiveness to Participants/Beneficiaries Evaluation Question 5: Do the project activities respond to the aspirations and needs of the project beneficiaries/participants? The MBSP established mechanisms that ensured that project activities were designed with participant input and were responsive to participant needs and aspirations. These mechanisms operated from the project’s inception through its close in December 2019. In executing some activities, particularly those having to do with construction and equipment, the project struggled to remain as responsive as it was during the design phase. These difficulties were primarily due to the project’s struggles to communicate effectively with project participants and manage their expectations. This section will review the mechanisms that the project relied on to ensure responsiveness to participants’ needs and discuss challenges the mechanisms faced. Events and Forums One mechanism for ensuring project relevance and responsiveness was the hosting of events and forums. These events and forums were effective because they instituted discussions among millet value chain actors, including government officials and researchers, to share ideas and understand how best to work with each other. At the start of project activities, The MBSP held four forums in December to raise awareness of the program, and to facilitate discussion among stakeholder groups. People working in the millet business, national and local government officers, training institutes, and civil society organizations attended the forums. The forums were used to raise awareness, and provide for exchanges among actors in the value chain and the MBSP. For example, millet value chain actors discussed their needs with government 39 officials, and civil society institutions discussed coordinating promotions to tout millet’s nutritional benefits. The project reports that it collected attendee’s opinions at the end of each forum and incorporated those opinions into project design (NCBA CLUSA, 2016). Importantly, events and forums led millet value chain actors to better understand each other’s needs. Examples include workshops on input markets attended by millet producers and input providers, including large providers such as OCP Senegal, a large agricultural input firm. At the project’s inception, OCP Senegal, a subsidiary of OCP Africa, was a recently established firm. A representative for OCP said, “Before the project in 2016, OCP didn’t know a lot about the market and producers.” OCP started its involvement at a workshop in Mabo attended by producers and seed and fertilizer providers. The OCP representative said that the initial workshop led to other information gathering meetings with producers, and allowed OCP to establish delivery channels directly to producers while providing improved inputs at a lower price (OCP Sales Representative, 2019). The workshops also provided information that led some input providers to exit the project. For example, one input supplier indicated in an interview that the company attended MBSP events and decided not to participate in the program upon realizing that the supplier could not meet the prices and delivery needs of the market. Another example of events’ contribution to relevance is their role in increasing access to finance. For example, a director at Crédit Mutuel du Sénégal (CMS) reported that at the start of the MBSP, he was invited to a workshop where farmers expressed needs for inputs. Value chain needs and challenges were also discussed at the workshop. After seeing how effective the MBSP organized workshop was at identifying needs and solutions, CMS organized a support workshop to collect further details about farmer needs. The workshops led to new loans that the director attributes to the links that MBSP through its events (Director, Crédit Mutuel du Sénégal, 2019). It merits noting that the mid-term evaluation recommended that the MBSP work with partners to find credit for farmers to purchase seeds (JE Austin Inc., 2018, p. Recommendation 5). The MBSP took action on this recommendation, ultimately finding its own creative strategy for financing seed purchases, as described in the preceding paragraph. Assessments The MBSP benefited from the information carried over from the first phase of the project. The MBSP staff reported in interviews with the evaluation team that, when the project started in 2016, staff reviewed reports that CLUSA had retained from the first phase which provided the number of producers in each of the project’s zones, the particular issues each zone faces, and the specific people who needed particular assistance. Staff took these reports to the field in the early stages of the project to confirm that the information was accurate and still relevant. In the field, coaches used a common diagnostic tool to understand their status and aspirations. Staff used the information they had reviewed and gathered to create the first annual workplan (MBSP Staff Interview, 2019). ScopeInsight Reports ScopeInsight is a Dutch organization that works to develop farmer organizations and strengthen agricultural supply chains. The MBSP commissioned ScopeInsight to produce around 30 reports on approximately 24 processor units and producer organizations that the project worked with. The reports rated each organization across a number of factors, such as quality of governance, operations, market position, financial status and others. The project used these reports to gain an objective understanding of some project participant’s needs and aspirations 40 Project Learning and Response The project staff that managed project participant’s engagement with the MBSP were referred to as “coaches.” The project had nine coaches in total. Eight of the coaches had geographic regions they were responsible for managing, and one was responsible for all of the project’s commercialization initiatives.. A central component of the work that coaches performed was listening to beneficiaries to understand their needs. Coaches then worked with their coordinator to understand how the project should design or adjust activities to better meet participants needs. Producer Benefits of Participating in the MBSP and Project Weaknesses Interviews with producers, producer organizations, and processors revealed the degree to which the MBSP was responsive to their needs. Producers most often pointed to access to new and better inputs, and the new agricultural techniques they’d learned as making a significant difference in their productivity. When we asked producers if the project met their expectations, 21 out of 22 producers and producer organization presidents replied that it had. The most important benefits that they cited were improving yield due to, mostly commonly, learning how to compost. Producers also mentioned better access to quality seeds, new varieties, and new skills. A farmer in Fatick summed up the common experience succinctly, stating, “Before joining (the MBSP), I had trouble finding fertilizer or getting funds to buy it. Now I no longer have this problem. Before, my yield per hectare was 500 kg, now I am a ton per hectare.” The evaluation team also asked producers what they liked best about the project. Producers replied most commonly that they liked the improvements brought about by using compost, the capacity building activities, the organizational development, and the training on seed selection. When the evaluation team asked what producers thought was the weakest part of the project, responses seemed to suggest that the MBSP had difficulty managing expectations. Many respondents could cite no specific weakness of the project. For those that could point to specific weaknesses, responses commonly cited perceived broken promises or non-delivery of facilities or equipment. Examples include a farmer in Kaolack who said the weak point of the project for him was “The non-realization of our storage.” A producer in Fatick said he thought the weakest point was “the broken promises … because the project had promised to find us buyers in order to avoid (our having to sell) our crops in the weekly markets.” One final representative remark comes from a producer in Fatick who commented, “The lack of rigor and follow-up vis-à-vis the contractor responsible for the construction of the works because he accused of a lot of delay in the delivery of the keys.” The evaluation team understands that complications arise during program implementation. The weaknesses that participants point to reflect a difficulty managing expectations. In the case of construction and equipment delivery issues, the weakness also points to the project’s struggles collecting activity and participant information, which is also reflected in the project’s overall poor data management. Processor Benefits of Participating in the MBSP and Project Weaknesses Processors that the evaluation team interviewed pointed to three major benefits of participating in the project; the training they received, the improvement in the quality of millet, and the equipment that they received. Trainings provided different benefits for different processors. Some processors pointed to hygiene and sanitation training as having made a big improvement in their operations. Larger processors tended to find organizational development training more important. Processors valued the improvement in millet quality because it saves them money. Bags of millet that are more than 5% debris can take hours of staff time to clean. Bags with little grains, or grains of various size, 41 can produce an unpredictable amount of final product. Having access to a regular supply of high quality, guaranteed product increases profits and makes management easier. The project weaknesses that processors described had to do mainly with the irregular supply and price of millet. They described the hubs as only having a supply of millet for a few months after harvest. Some processors have to close their doors when millet is no longer available. Processors also mentioned price fluctuations as an issue, quoting prices as low as $0.30 per kilogram right after harvest to $0.67 per kilogram in the summer months, and increase of over 120%. 3.2.2 Alignment with USDA/US Government Goals, Objectives, and Strategies Evaluation Question 6: Are the project activities in alignment with USDA/US Government development goals, objectives, and strategies? Evaluation Question 7: Are the project activities in alignment with award agreement with USDA? The project activities are in alignment with USDA development goals, objectives, and strategies, and in alignment with the award agreement with USDA. USDA established Food for Progress to improve agricultural productivity and expand trade in developing nations. These objectives were the central goals of MBSP, and both were achieved. The MBSP planned and executed activities as defined in its award agreement, and strove to produce results under the two objectives and nine activities categories defined in the agreement. 3.3 Effectiveness 3.3.1 Increases in Yields and Reduction in Waste Evaluation Question 8: To what extent have different actors in the value chain been able to increase quantity e.g., yield/ha and minimize waste since the inception of the project? Producers Please see section 3.1.1 Goal Achievement/SO1 - Increased Agricultural Productivity/Increased Yields for analysis of the change in producer’s yields and reduction in post-harvest loss. Sellers Please see section 3.1.1 Goal Achievement /SO2 – Expanded Trade of Agricultural Products (Domestic, Regional, and International) for analysis of the change in seller yields. 3.3.2 Quality Adoption Evaluation Question 9: To what extent have quality standards been adopted by all actors in the value chain? Actors at all levels of the value chain have adopted quality standards to some degree. The introduction of quality standards to producers, hubs, processors, and kiosks is one of MBSP’s most important achievements. 42 Producers The final evaluation found that 75.7 percent of producers that participated in MBSP said that they’d received training since the MBSP’s launch. When asked which types of training they’d received, the most common related directly to quality were seed quality identification (62.7%), post-harvest handling (53.9%), and business planning and marketing (34%). Though it is not statistically significant, it is useful to note that only 16.9 percent of neighbor farmers that responded to the survey said that they had received a training in millet cultivation since the beginning of MBSP. Regarding quality adoption, 87 percent of MBSP producers said that quality had improved since the start of the project. Figure 11 shows producers’ responses when asked what had most improved. Better variety (48.2%) and bigger grains (32.9%) were the most common responses. In interviews, producers demonstrated that they’d adopted quality standards. In 17 interviews with farmers, producer organization presidents, and hub managers during which quality adoption was discussed, all but one said that the quality of the millet they produced had improved. When asked why it had improved, the two most common answers were improvement in the quality of seeds that farmers used, and changes in the variety of millet that they were planting. A producer in Fatick summed up a common thought when he said, “We received quality seeds that produced quality.” Referring to the change to the Thialague 2 variety of millet, a farmer in Kaffrine said, “With the arrival of CLUSA, our yields have improved due to the introduction of millet with long stems.” Figure 9 Hub President Demonstrating a Quality Test to Show that Millet Grain Size and Quality is Consistent throughout a bat Figure 11 Producer perceived quality improvements 43 Hub presidents reported that hubs contributed to quality improvements. Hubs perform services like providing inputs on credit and assisting with sorting millet quality. A Hub president said that the hubs ability to regulate inputs contributed to millet quality. He said that the hub provides certified seeds and five bags of fertilizer to each farmer. Of the 13 farmers that the evaluation team interviewed, 11 were asked how many kilograms of seeds they applied and how many hectares they cultivated. Each responded that she/he applied 4 kilograms per hectare exactly, demonstrating that the training had been adopted. The evaluation survey asked 531 millet buyers (producer organizations, hubs, processors, kiosks and other retailers) if they had seen an improvement in the quality of millet they’d purchased since the beginning of the MBSP. Ninety-three-point-six (93.6) percent of all millet buyers surveyed said that millet quality had improved.14 When asked how it had improved, the top choices were fewer debris and bad grains (69.3%), bigger size grains (49.2%), and better taste (38.4%). Processors and Sellers The MBSP recorded in its strategic planning documents that demonstration of prioritizing quality management was a condition for processors to participate in the MBSP. Accordingly, processors were trained in sanitation and hygiene, processing, and packaging and branding. The evaluation team interviewed processors to understand the degree to which they’d adopted these practices. Packaging was the most widely adopted quality improvement. As mentioned at the introduction to the finding section of the report, processors of all sizes, from those processing one ton or less a month to Maria Distribution, which processes more than 20 tons a month, have valued the MBSP’s introduction of small scale packaging. Each processor that the evaluation team visited could demonstrate packaging, how the packages all contained the same volume of product, and the benefits of the different elements of packaging such as the window that allowed product visibility, and the contact information that built customer confidence. Sanitation and hygiene practices were not as universally adopted. Some processors, particularly those that target large clients or institutions such as hospitals, saw hygienic processing as a source of competitive advantage. Others seemed to understand the idea but have more trouble implementing it, as shown in Figure 13. 14 Only 388 reported buying and selling last year as many producer organizations and several hubs did not buy or sell. Figure 10 Processor Perception of Quality Improvements Figure 11 An MBSP Coach Demonstrates Packaging Equipment 44 Figure 12 One processor on left demonstrates good hygiene practices, 3.3.3 Training In Quality Standards And Agricultural Techniques And Technologies Evaluation Question 10: To what extent value chain actors have been trained in quality standards? Evaluation Question 11: Are partner organizations providing training to beneficiaries that are tailored to the local context and achieving increased use of agricultural techniques and technologies? Training on Agricultural Techniques and Technologies 75 percent of MBSP producers said that they’d received training since the beginning of the program. Of those 75 percent, 84.5 percent said they had received training in soil management, which was the most common and valued type of training, followed by crop monitoring and seed quality identification. Trainings and technologies appear to have been tailored to the local context and achieved increased use. There is evidence that those who received the training applied it as soil management and seed quality identification training were identified as the most important in the evaluation’s survey, qualitative interviews with 22 producers produced 23 mentions of composting as a form of soil management and 14 mentions of seed quality identification, and evidence throughout the projects results that quality and yields were stronger than without the training. Additionally, 76.4 percent of producer organizations and 17 of the 18 hubs continue to provide training for their members. Training in Quality and Related Skills The final evaluation asked producer organizations presidents, hubs, processors, women’s group processors, and kiosk managers to choose the two most useful MBSP activities. Their choices are listed in Table 21 below. While composting was not an option that was read to respondents, over 28 percent mentioned it under “other,” which reflects how popular the training was. 45 Table 21 Top two most useful trainings by organization type Respondent Type and Choice Percentage Producer organizations Training producers 54 Composting 28.2 Hubs Training producers 35.3 Business planning and marketing 35.3 Processors Business planning and marketing 55.3 Hygiene and quality 16.3 Women's group processors Business planning and marketing 40.8 Processing (including Farine) 18.4 Kiosks Business planning and marketing 64.3 Linkages to customers 21.4 Business planning and marketing training was popular among organizations because it led to better organizational management and higher profits. A processor in a women’s group said, “Before CLUSA the organization was not well organized. It was selling in random places. After CLUSA came, the organization is better organized and now saves money. The training increased income about 75%.” Hygiene training was also an important contributor to quality. Hygiene training built confidence among customer groups and improved the general quality of the overall product. A processor in Fatick started with a small operation. Following training in packaging and hygiene, she was able to secure sales to a local hospital, which built her reputation in her neighborhood and town. 46 3.3.4 Post-Harvest Processing, Handling, Marketing and Branding Evaluation Question 12: Are partner organizations increasing the use of improved post-harvest processing and handling practices, and improved marketing and branding? Post-Harvest Processing and Handling Practices Partner organizations, in the form of hubs and producer organizations, are increasing the use of improved post-harvest processing and handling practices, though the degree to which practices are increasing varies greatly among organizations and hubs. The millet hubs that the project established increase the use of post-harvest processing and handling practices through two mechanisms. The first is demonstration and incentives. The hubs that are functioning well have shown their members that producing higher quality millet that is sorted and graded commands a better price. The millet hubs are centrally located and were created to provide a variety of services to hub members, including providing storage and post-harvest processing facilities available to producers. The degree to which hubs provide these assets varies greatly. The evaluation team visited 9 of the 18 hubs established by the MBSP. Of the 9 hubs visited, 2 had partnered with CLUSA to purchase post-harvest processing equipment, acquired an electrical connection, and were providing post-harvest processing services. Three of the 9 had acquired equipment, but the equipment was not functioning. In one case it was because of a lack of electricity. The other two hubs had electricity, but the equipment was not functional for some other reason. The remaining four hubs did not have post-harvest handling equipment and were either providing cleaning and sifting services through manual labor or not providing those services at all. Eight of the 9 hubs visited had storage facilities that was used to store millet purchased from producers and producer organizations. One of the storage facilities had issues with the quality of its construction and another had not yet had its storage facilities built. Figure 13 A hub president weighs high quality seeds to go into storage on behalf of a producer 47 Table 22 Visited hub status Hub Equipment Functioning Storage Functioning Keur Saloum Diané Yes No Yes Yes Keur Samba Gueye Yes No Yes No15 Taiba Niassene No No No No Taiba Mbayene Yes Yes Yes Yes Nioro Alassane Tall No No No No Khelcom Birane No No Yes Yes Godaguene Sorokh No No Yes Yes Mabo Yes Yes Yes Yes Boulele Yes No Yes Yes Total 5 2 7 6 Improved Marketing and Branding The project improved producer marketing and branding through producer organizations and hubs. Hubs built customer confidence by grading and storing millet and guaranteeing quality. As part of their quality guarantee, hubs print their contact information on storage bags so that processors can contact the hubs directly if any issues with quality are discovered after delivery. Processors reported that they preferred to buy millet from the hubs, rather than farmers individually or in markets. A buyer at a distributor said that she “prefers to buy millet from the hubs because they have the best quality and less debris.” She said she buys 70 percent of her millet from the hubs, with the rest coming from a local wholesale market when the hubs run out. Producers reported that the strategy of marketing through the hubs was effective. One producer in Fatick said, “Before, I sold in the market where (millet) was transformed, and I’d sell it in any condition. Now, I sell straight to the hub. It needs to be higher quality and more organized, and it pays a higher price.” 15 Interviews revealed that The MBSP built a storage facility for the Keur Samba Gueye hub. However, it was constructed poorly and the roof had entirely blown off of it, rendering it non-functional. Figure 14 The Khelcom Birane Hub president with millet bags including hub's contact information (Photo printed with consent) 48 Kiosks and processors improved marketing and branding through two channels, improved packaging and improved customer management. As mentioned previously in this report, the packaging that MBSP trained participants to use has important features that translated into increased sales (Figure 17). First, the packaging is made mostly of paper, which processors feel conveys a message of higher quality than the plastic disposable bags that many used prior to the program. Second, the packaging contained a window that showed the quality and consistency of the product. Finally, the packaging contained the contact information of the seller, allowing customers to voice any complaints and exchange or receive a refund if any quality issues were discovered. The packaging was widely popular with processors and sellers that the evaluation team interviewed. Processors demonstrated for the evaluation team how the packaging was filled, ensuring that it contained the same quantity of product in each, and how the packaging was assembled. Some interview respondents’ one issue with the packaging was that the glue that sealed the package was weak. One processor used additional glue to ensure the package would stay sealed. An MBSP trained APS who was a packaging specialist acknowledged the problem, stating that the closure of the paper bags is an ongoing problem and that she is working on a solution with other APSs. Nonetheless, improved packaging led to improved branding and increased sales. The MBSP customer service training was important to processors and led to more repeat customers and, for several processors, allowed for greater reach. A women’s processor group credited MBSP’s customer service training with helping them sell to a local university, restaurants, and customers at great distances such as the Casamance and Dakar. The group’s treasurer said, “The most important thing was training in how to maintain customers and build relationships. Sometimes now, customers call just to talk!” Kiosks have also improved branding and marketing. Kiosks provide sales points for processed products and ready-to-eat products. Processors who run kiosks that the MBSP delivered report that kiosks “attract customers.” However, interview respondents reported difficulties in utilizing the kiosks and questioned their sustainability. Issues include the kiosk’s lack of electricity, the excessive heat inside kiosks on high temperature days, the unpredictability of kiosk’s open hours, and the location of some kiosks. This report discusses this issue in more detail in section 3.5 Sustainability. 3.3.5 Extension Officers Technical Support to Farmers Evaluation Question 13: Are local government extension officers offering technical support to farmers in new techniques and technologies? Extension officers provided technical support to farmers over the course of the project with financial and technical support from MBSP. Throughout 2017 and 2018, the project worked with DRDR and linked it with producer organizations to train farmers in agricultural techniques. The evaluation team cannot assess the extent to which extension agents are currently or will continue to offer support to farmers. The mid￾term evaluation recommended that “That NCBA CLUSA and the Agricultural Extension Agents investigate why certain practices are not being followed; that retraining and/or support is provided to farmers where necessary (JE Austin Inc., 2018, p. Reccomendation 8).” The evaluation did not find evidence that this Figure 15 MBSP improved packaging on left, traditional packaging on right 49 recommendation was implemented. The evaluation team interviewed an extension agent that currently provides services to approximately 6,000 farmers in 3 communes. He has conducted trainings through the MBSP since 2017. He provided trainings in agricultural practices, post-harvest handling and storage. The extension agent reported that the MBSP did not develop standardized content for farmer trainings, but included that farmers have such little knowledge that providing basic information is what is most required. The extension agent also pointed out that the MBSP made an important contribution to extension services as the project was the first to work with extension agents in the area. 3.3.6 Linkages Among Project Participants Evaluation Question 14: Have effective linkages been established between different project participants e.g., between producers and processors? The project established effective linkages among different project participants. Hubs and processors were particularly successful in establishing linkages, with greater than 50 percent establishing linkages through MBSP events. Other types of participants also had a high rate of successfully establishing relationships. In this section the report describes the degree to which different project participants were engaged in the MBSP’s networking activities, and then reports interview respondents’ descriptions of the linkages they have established. The MBSP mainly linked project participants through three mechanisms, events such as business forums and workshops, through coaches making introductions, and special arrangements to solve specific problems. Events The MBSP events served several purposes. They raised awareness about the potential of millet and the millet sector, they served as a means of group discussion and problem solving, and they connected various actors in the millet value chain. An example of a MBSP event is the Touba business forum. Touba is a major hub of the millet trade in Senegal and is where approximately a third of national millet sales takes place (NCBA CLUSA, 2017). At business events such as these, contracts would be signed between buyers and sellers, and contacts made between producers and financial service providers as described in section 4.2.1 of this report. The evaluation survey sought to understand the degree to which project participants benefited from MBSP events and whether they made and maintained linkages at those events. The survey found that hubs (94%), processors (82%) ,and women’s groups (76%) attended events in the highest proportions, as seen in Table 23. Table 23 Attendance at Events by Respondent Type Respondent Type Percentage Hubs 94.4 Processors 82.2 Women’s processing groups 76.9 Producer Organizations 48.6 Restaurants/Hotels 45.5 Kiosks 20 All types 41.2 50 The survey then asked those that had attended events if they had made and maintained linkages. As shown in Table 24, hubs and processors that attended events reported that they were particularly successful at maintaining linkages made through the project. Table 24 Success in Making and Maintaining Linkages Respondent Type Percentage Hubs 70.6 Processors 70.3 Restaurants/Hotels 60 Producer Organizations 47.6 Women’s processing groups 47.5 Kiosks 42.9 All types 54.8 As mentioned in the Relevance section of this report, banks such as CMS and input providers such as OCP attended workshops to gather information on the market, establish trust with potential new customers, and design new products. These organizations report that they established linkages and maintained relationships through MBSP events. The OCP sales representative that the project spoke to said “the project gave us a point of penetration into the market.” Other enterprises grew revenue through event participation. A kiosk owner in Kaolack said her relationship with the project started at a fair, where she was invited by MBSP to sell her products. She sold products for 15 days at the fair, and then was awarded a kiosk. A small processor in Kaffrine provides a case study that represents a common experience among small businesses that made linkages and grew revenue through the MBSP. This processor, a member of women’s cooperative, had a small business processing and selling millet products in her neighborhood. She originally heard about MBSP through word of mouth. She found an MBSP coach, who provided her with information about the project and invited her to attend a workshop. At the workshop she was trained in packaging. At later workshops she learned how to produce new millet-based products. She also learned to maintain sanitation and hygiene standards, which she reports gave her customers more confidence in her product. She reports that the combination of improved packaging with her contact information on the label, her improved hygiene standards, and Figure 16 Kaffrine Processor Demonstrating Packaging and Weighing 51 her improved products has allowed her to sell products to a local hospital. Parents who have children in the hospital particularly seek her out, because her fortified millet flour helps fight malnutrition. She now aspires to serve bigger customers. However, as she sees it, two obstacles are holding her back. The first is that she has not gone through the processes that would legally formalize her business. Formalization is a common obstacle that holds back many MBSP enterprises, and the evaluation will return to it in the Sustainability section. Establishing herself as a formal business would allow her to access supermarkets and exporters, which she can’t current sell to because of her lack of legal papers. Second, she reports that MBSP coaches identified and brought her current major customers to her. She says, “We need to learn to make the connections ourselves.” Coach Coordination At the producer level, coaches first established the hubs and then brought processors to negotiate sales with the hubs. Coaches initially recruited producer organizations that were most viable to establish the hubs. After hubs were established, hub members were encouraged to raise awareness among remaining villages and organizations to join the hub. Once a sufficient number of producer organizations had joined and quality millet was assured, coaches brought buyers. The Kaffine processor in the case study above was brought to a hub and now buys from the hub when millet is available. Processors prefer to buy from hubs because of the guarantees in millet quality, though supply is not always available. Special Arrangements A final mechanism that MBSP used to link participants was working out special arrangements to solve particular problems. Particularly, two types of special arrangements were made, access to finance to purchase inputs, and access to specialized equipment. The MBSP identified producers lack of funds to purchase quality inputs as a major constraint to achieving the project’s objectives. Obstacles along the value chain made providing access to finance for the purpose of purchasing inputs difficult. For example, individual producers did not qualify for loans of adequate size, input providers were hesitant to provide products on credit, and producers were wary of engaging with formal lenders. The MBSP, Banque Agricole, OCP, MBSP hubs, and Processors worked out a solution: First the hubs would calculate their fertilizer and seed needs for all of their members, so the bank could deal with significant volume and would not have to manage the risk of individual producers. The hubs bring their order for seeds and fertilizer to the bank, who orders it from OCP. OCP delivers the inputs to the hubs and is paid by the bank. At the end of the millet harvest, the hubs provide millet valued at a set price to processors who then pay back the bank. This arrangement provides the quality inputs that producers require, ensures millet for processors at a stable price, reduces Banque Agricole’s risk, and grows a market for quality inputs in the millet sector. 52 Figure 17 Financial Arrangement for Input Purchases A second type of arrangement provided specialized equipment for producers and hubs. This arrangement was more traditionally done, with the MBSP advertising the need for equipment and evaluating manufacturers proposals for providing the equipment. However, manufacturers were brought into the project to design for a specific need and occasionally an innovation would arise out of the relationship. 3.4 Efficiency 3.4.1 Internal and External Factors That Influenced Expected Results Evaluation Question 15: What internal and external factors have influenced the ability of the project to meet expected results and targets? In this section the report summarizes factors that contributed to and hindered success. Included after each factor is the section of this report that discusses it in more detail. Bank Agricole Producer Processor Millet Figure 18 A millet drier procured for a processor through the USDA| MBSP 53 Internal Factors That Contributed to Success Internal factors that contributed to success were:  The project’s strong knowledge of the millet sector, its structure, and the main actors that carried over from the first phase of the project (3.7 Lessons Learned, Lesson 1).  Building on that knowledge, the project created a clear strategy based on creating points of entry into the market for different value chain actors, building trust, and demonstrating value (3.7 Lessons Learned, Lesson 2 and Lesson 4).  The project team’s culture (3.7 Lessons Learned, Lesson 3). Internal factors that hindered success were:  The project’s difficulty managing participant expectations, especially in the construction of facilities and delivery of functioning equipment (3.2.1 Responsiveness to Participants/Beneficiaries - Producer Benefits of Participating in the MBSP and Project Weaknesses, page 37)  The program’s struggles to collect and manage data, which partially led to difficulties in managing expectations (3.4.2 Staffing, Management, and Resource Allocation – Staffing and management, page 52)  The project’s internal incentives, which may have led to delivering equipment, buildings, and canteens in order to meet project output targets without an incentive or time to evaluate if those deliveries were likely to produce intended outcomes (3.7 Lessons Learned, Lesson 5).  Managing the need to recruite hub members to pay fees before hubs had implemented services that would attract members (3.7 Lessons Learned, Lesson 5). External Factors External factors that contributed to the project’s success were:  Positive media coverage facilitating market entry approval (NCBA CLUSA, 2017).  Growing demand for millet (3.5 Sustainability, Factors contributing to sustainability).  Government campaigns aimed at overcoming malnutrition. External factors that hindered success were:  Insufficient rainfall (3.5 Sustainability, Factors hindering sustainability).  Lack of professional management capacity among hubs, processors, and others while facing a short deadline to have hubs operating independently and processors mastering marketing and quality practices (3.5 Sustainability, Factors hindering sustainability).  The private nature of suppliers and difficulties the project had mobilizng them (NCBA CLUSA, 2017).  Bureaucratic delays, such as those in signing purchase orders with DRDR (Project progress report, October 2017 – March 2018).  The difficulty faced obtaining accurate data from partners, such as member lists from producer organizations and hubs, financial data from sellers, and data on branding activities (3.4.2 Staffing, Management, and Resource Allocation – Staffing and management, page 52).  Balancing the need for increased millet yields with maintaining a reasonable price, especially immediately following the millet harvest (3.2.1 Responsiveness to Participants/Beneficiaries - Producer Benefits of Participating in the MBSP and Project Weaknesses, page 37.  Senegal’s expensive and difficult to access electricity market (3.5 Sustainability, Factors hindering sustainability).  Senegal’s expensive process of business formalization (3.5 Sustainability, Factors hindering sustainability). 54 3.4.2 Staffing, Management, and Resource Allocation Evaluation Question 16: Do project staffing, training activities and management structures efficiently support the achievement of expected project outcomes? Evaluation Question 17: Does the project resource allocation to activities and to the project’s geographic areas efficiently support the achievement of expected project outcomes? Staffing and Management Staffing and Management appear to have been structured efficiently, with the exception of the MBSP’s monitoring and evaluation function and its grant management function, which oversaw grants to participant organizations and the use of those grants for their intended purposes. The MBSP was structured to provide a presence in the communities it worked in while maintaining relationships with millet value chain actors in other parts of the country. Project activities were mainly implemented through 9 coaches. Eight of these coaches were assigned geographical responsibilities, and the 9th was tasked with management commercialization in the form of packaging, marketing, and branding. Coaches raised awareness and trained hub and producer organization members to raise awareness, arranged and delivered trainings, conducted follow up activities. One coach described his role thusly, “First we help people with good practices in the field, like composting and conservation farming to have good yield. Then, we link farmers and processors. After that we helped processors to have a good brand in the market where they can sell. We train processors to influence people to explain that millet is important. We train processors about commercialization.” Coaches themselves received training in agriculture practices, organizational development, marketing, packaging, branding, post-harvest handling, and storage. Coaches were managed by a coordinator. The coaches created monthly plans for activity implementation. The coordinator and coaches would review each plan at the end of the month to understand what went well and which activities were challenged, and then created a new plan for the coming month to move the project forward. In addition to this core management structure, the MBSP maintained specialists in technical areas on staff to ensure proper implementation of agricultural, construction, and logistical activities. The project also maintained an administrative staff. The benefit of this structure is that it kept staff strategy and mission focused. Coaches that the evaluation team interviewed had a clear understanding of the project’s strategy, which one coach summarized as “Build trust through physical investments, then change minds.” As the coach described the strategy in action, “ The first step was soil fertility. We explained that a good harvest could happen even with climate change. At first (producers) said it was nonsense, they were jaded. But when they saw the demonstration farms, they said ‘wow!’ CLUSA showed them it was cheap to implement and locally available. This built independence and trust in the project.” According to the coaches, the physical investment in hub infrastructure had a similar effect.” 55 One area where the management structure performed poorly was monitoring and data management. The project lost its monitoring and evaluation staff member early on in the project, which may have gotten the project off to a bad start. The project did not maintain a list of organizations it managed, accurate information on the beneficiaries it served, information on event participants, lists of trainings delivered, or other data required to report on the project and make management decisions regarding its progress. Currently the project does not have an accurate count of the number of producer organizations or participants that have benefited from the project. Project staff mentioned that poor data management was caused by the project’s reliance on volunteers to manage trainings and other activities and report on those trainings and activities. However, this reason does not explain errors in the data on hand. For example, the project’s dataset which included 23,000 producer organization members was also the dataset that the project relied on to count participating organizations. The dataset contained many easily identifiable errors in the org names, such as the same name appearing in all caps, all lower case, sometimes first word capitalized, sometimes second word capitalized, sometimes village name included, sometimes not, sometimes village and region reversed, and other errors. These errors lead to confusion about which organizations are receiving which project interventions, and mistargeting of some interventions. They also indicate that the project’s reports regarding its indicators may not be accurate. The second shortcoming relates to the management of constructing and equipping the hubs. The project’s hubs were managed by local volunteers that had greatly varying managerial abilities. Some, such as the hub in Mabo, have outstanding leadership that have created awareness in the surrounding area to the degree that they have to rent space in the town of Mabo because the demand for services and storage is much greater than their ability to serve that demand. The leadership in Mabo worked with the MBSP to acquire equipment that will serve hub members, and themselves acquired electricity connections to operate the equipment. Mabo also has become a formal business, allowing it to sell to large formal businesses and institutions. Other hubs operated in a different context. As mentioned in section 4.3.4, several of the hubs have equipment that is not functioning either because hub managers do not know how to operate the equipment, the equipment is faulty, or the hub has no access to electricity, which is expensive in Senegal and difficult to acquire. As also mentioned in 4.3.4, some hubs saw delays in construction without information forthcoming from the project. Many hubs remain informal which constrains their ability to do business with large customers. The mid￾term evaluation recommended that the MBSP “continues to support producer organizations in Fatick in acquiring formal legal status (JE Austin Inc., 2018).” In fact, the hubs, processors, and other participating businesses would benefit from achieving legal status. The MBSP was aware of this issue throughout the project, recognizing in its 2019 Narrative Report, which covered the period of April – September 2019, that the cost and time required to formalize the hubs and other businesses remained a challenge (CLUSA, 2019). The hubs are an effective innovation that deserves recognition. The evaluation recognizes the great benefits they have brought to the populations they serve. They provide an example to the government in serving people, and built trust among the population the MBSP served. This population is necessarily in Figure 19 The Many Spellings of Producer Organization Names 56 rural, hard to reach areas where they will benefit the rural population. However, their implementation could have been managed more carefully, which would have ensured that funds would only have purchased equipment where there was a likelihood of managing installing electrical connections and developing the expertise to operate it, and expectations regarding construction could have been managed more closely. Allocation of Resources Table 25 compares the budget for MBSP activity components to the amount actually spent. It shows the proportion that the project planned to spend on each component, and the proportion it actually spent. As the table shows, the project spent almost exactly as it had planned. The project spent a bit more on capacity building related to extension services than planned, a slightly less on building buyer/seller relationships. Table 25 Percent Budgeted and Spent on Activity Components Activity Component Percent of Activity Budget Percent of Activity Expenditure Capacity Building: Ag Extension Agent Services 33 37 Financial Services: Ag Lending 19 18 Infrastructure: Post-Harvest Handling and Storage 03 03 Infrastructure: Post-Harvest Processing 14 15 Inputs: Develop Agro-dealers and Input Suppliers 08 08 Market Access: Facilitate Buyer Seller Relationships 07 05 Training: Improved Agricultural Production Techniques 03 02 Training: Improved Marketing and Branding 05 04 Training: Post-Harvest Handling and Processing 08 08 57 3.5 Sustainability Evaluation Question 18: What factors are likely to contribute towards the sustainability of the following factors in relation to the MBSP: policy environment, financial incentives (i.e., profit motive), access to inputs and information, market access, etc. Evaluation Question 19: What factors are likely to hinder sustainability of the following factors in relation to the MBSP: policy environment, financial incentives (i.e., profit motive), access to inputs and information, market access, etc.? Evaluation Question 20: What project activities are likely to contribute towards sustainability and is there evidence that project activities or benefits will be sustained after the project ends? The evaluation team understands the ToRs Sustainability questions to ask for a description of the factors that are likely to promote the sustainability of the MBSP’s results, the factors that are likely to hinder the sustainability of those results, and the likelihood that project activities will continue. This section addresses those sustainability issues. Factors Contributing Towards Sustainability  The governments positive view and confidence in the project – Government officials are aware of the project and have seen the benefits it has brought to their constituents. Physical infrastructure that the project has built could become monuments to the governments leadership if local officials work to sustain them, or memorials to what might have been. The enthusiasm that government officials showed for the project and its benefits in interviews suggest that there’s a strong possibilty the government will work to sustain the MBSP’s efforts.  Organizations having the means to continue training – In interviews the evaluation team asked respondents if their organization would continue training members after the project ends. Several said that they would. Organizations that value the training and have developed internal capacity to deliver it will sustain the MBSP’s capacity building momentum.  Improved Agricutural Processes – Better agricultural practices, particularly the use of compost, quality seeds, new varieties, and stiga identification, are now embedded in the agricultural zones the project has worked in. Those knowledge and practices will remain to enhance millet yields after the project is completed.  The relationships established among particpants – The project provided a point of entry for input suppliers looking to serve the producer market, banks eager to provide financial services, processors that wanted higher quality millet and greater supplies, and producers looking for new markets that would value quality product. The MBSP established those relationships and interviews with project participants indicate that they will persist. As demonstrated throughout this report, processors have indicated that they prefer to purchase millet from hubs as the price and quality is more trustworthy; banks have indicated that they will continue working with hubs, processors, and input suppliers; and input suppliers that had success through the project have indicated that they will continue to serve producers through the hubs.  Raised Awareness – The MBSP raised awareness that millet is an imporant source of nutrition.  Hub Fees – Hubs that are successful charge a small fee, approximately CFA 25 per kg, to support ongoing expenses. These fees provide sufficient resources to maintain hubs that are operating successfully. 58  Growing demand – Producers and processors report that they sell everything they produce and that demand domestically and internationally easily oustrips supply. The incentives that these markets provide should continue investment and innovation in the sector.  Possibility of additional efficiency gains – The are still many opportunities in the sector to increase profits through efficiency gains. For example, there is still a lack of sufficient storage that would allow most processors to store millet right after the harvest when the price is low and sell it in June when the price is highest. Investments in energy sources would allow the mechanization of the sector. These opportunities for new profit could drive investment after the project is complete. Factors Hindering Sustainability  Climate – As a producer in Kaolack said “When rain is weak, all of the training doesn’t produce much.” Over the course of the project, rainfall has been the greatest variable determining yearly yields. As the climate changes, if rainfall declines producers will not maintain high yields.  The government’s lack of resources to sustain intiatives – As noted by the ministry of agriculture and the DRDR agent that the evaluation team spoke with, the Government of Senegal does not have the resources or organizatonal capability to support the project at the same level that NCBA CLUSA has.  Contracting – Because of the project’s short time frame and need to produce results quickly, the project had a heavy hand in arranging contracts between producers and processors, and processors and customers. These contracts also appear to have been documented arrangements rather than legally binding, official contracts. Value chain actors may not have the skills or knowledge required to make these arrangements in the project’s absence. That said, respondents to interviews said they would maintain contracts and seek new ones even after the project closes.  Formalization – The expense and arduousness of becoming a formal enterprise threatens to halt the progress of hubs and processors. Hubs and processors that have become formal have grown rapidly, because the legalization of the business allows access to large customers such as supermarkets, universities, hospitals and exporters. However those enterprises are few in number. The remaining will see progress halt without assistance in the formalization process.  Electricity and Maintenance – Cleaning services are in demand and the hub’s potential to offer cleaning equipment appeals to potential members. An MBSP coach pointed out, “People will carry millet far if they can clean it.” However, producers may lose interest in hubs that have equipment but no electricity, rendering the cleaning equipment useless. Likewise, secure storage facilities are important to millet producers. The MBSP built excellent storage facilities at many of the hubs. However, those facilities require maintenance and there is already signs that storage facilities are not adequately maintained. (see Figure 22)  Labor Shortage – One challenge confronting processors particularly is that manual laborers consist mainly of older women, and as they exit the workforce young people are not taking their place. Four processors reported that the biggest constraint on their production was lack of workers. One gave the opinion that the hubs should work to recruit young people. Figure 20 The storage facility roof at the Keur Samba Guèye hub requires maintenance 59 3.6 Impact 3.6.1 Impact on program participants Evaluation Question 21: What impacts are the project activities having on the program participants, both positive and negative, especially in relation to the expected results and strategic objectives? Please see 3.1 Project Performance for the evaluation’s assessment of the impact that the project has had on participants in relation to results and strategic objectives. 3.6.2 Millet as a viable crop Evaluation Question 22: To what extent have target groups (farmers, processors, vendors, consumers, etc) adopted millet as a viable crop that has economic and nutritional benefits? In the absence of baseline data, it is difficult to know how target groups consideration of millet as a viable crop has changed. As discussed previously in this report, processors believe there is more demand than supply and continue to invest in millet-based products, producers claim they sell everything they care to sell without issue, and banks say that they will continue to lend to millet producers. These signs indicate that millet will continue as a viable crop. 3.6.3 Extent of project reach Evaluation Question 23: What is the extent of the project reach beyond direct and targeted beneficiaries? The evaluation team could not assess this question. The extent of reach within direct and targeted beneficiaries is also uncertain. 3.6.4 Unexpected Outcomes Evaluation Question 24: What are the unexpected outcomes that could have been due to the project? There are two possible unexpected outcomes, though this evaluation does not have sufficient data to say with any certainty if they are attributable to the project. The first is that the project may have contributed to a higher volume of millet exports. Exporting processors that the evaluation team spoke with said that they’d increased exports considerably recently, and that access to higher quality millet through the hubs helped in their new endeavors. Second, the project may contribute to future increased agricultural lending. The banks that participated in the project indicated that they would hold MBSP-style events in the future to better understand producer needs and craft new projects. If this were to happen, MBSP will have served as a model. 60 3.6.5 Overall impact of increasing trade Evaluation Question 25: What is the overall impact on increasing trade of millet? (marketing strategies, creation of new partnerships, market information systems, certifications, etc.) The evaluation team understands this question to ask if increasing the trade of millet led to any broader systems or partnerships. Please see section 3.7 Lessons Learned and Best Practices for a discussion of the project’s impact in this regard. 3.6.6 Increase in trade Evaluation Question 26: How has the trade increased as a result of USDA MBSP on local, regional, national, and international level? Please see 3.1 Project Performance / SO2 – Expanded Trade of Agricultural Products (Domestic, Regional, and International) for a discussion on how trade has changed as a result of the project. 3.7 Lessons Learned and Best Practices Evaluation Question 27: What lessons learned can be documented at this point to inform implementation of the program until completion? Evaluation Question 28: What best practices can be shared with stakeholders and donors? Lesson 1: A strong knowledge of the sector and its history drove project success The project’s successes were driven by the implementer’s strong knowledge of the millet sector, its structure, its history, and its main actors. This knowledge carried over from the first phase of the project. As described in the context section of this report, the MBSP leadership understood the constraints to millet production in Senegal, namely low-quality inputs, poor rural infrastructure, high transaction costs, and high farmer inefficiency. The leadership also understood the historical associations with millet as a subsistence food rather than high quality product, and the opportunity to rebrand millet as a substitute for rice. The project’s leadership identified lack of trust among value chain actors as a contributor too high transaction cost. Lesson 2: A clear strategy based on creating points of market entry, building trust, and demonstrating value led to project success Building on its knowledge, the project created a clear strategy. The strategy was based on creating points of entry into the market for different value chain actors, building trust, and demonstrating value. After working with producers to build millet supply during the first phase of the project, the MBSP created the millet hubs. The millet hubs represented a tangible commitment from the project and thus built trust with producers that were skeptical about investing in high quality inputs. It also built trust with processors as the hubs guaranteed supply and quality. Importantly, the hubs served as a point of entry into the market for processors looking for high quality millet, input providers looking a critical mass of producers to serve and negotiate with, and financial service providers, who wanted a central organization to negotiate with to lower transaction costs. The project’s business forums and events served as another point of market entry. They provided a central location where value chain actors would gather for the purpose of transacting goods and ideas. Having many actors together at predictable intervals reduced the cost of having to visit each individually and the 61 cost of testing new ideas across a large area. As discussed earlier, processors, input providers, financial service providers, producers, and hubs all utilized events as a point of entry. For customers, the MBSP’s kiosks also served as point of market entry and trust development. They allowed customers easy access to new products, and sellers visibility to new customers. Lesson 3: The project culture was important to its success A review of project documentation and interviews with project staff reveal that the MBSP leadership instituted an ethic of acting strategically and acting in service to project participants. Interviews with project staff demonstrated that project staff from the Chief of Party to the coaches and administrative staff understood that the project was about building trust, focusing on quality improvements, and creating points of market entry. Project staff were enthusiastic about their work. Project participants, such as farmers and small processors, could quickly produce for evaluators during KIIs the names and phone numbers of coaches and felt that coaches could be called with issues or ideas. Lesson 4: Opportunities for participants to provide feedback to the project and to each other pay off One innovative mechanism that the project created was its events and forums. The MBSP’s events and forums created opportunities for the project to listen to participants and create initiatives that responded to their needs and aspirations. Importantly, the design of those events created opportunities for participants to listen to each other. Large organizations such as OCP, Banque Agricole, and Crédit Mutuel du Sénégal, to their credit, took advantage of these events and walked away with new points of market entry and new products to offer millet value chain actors. Lesson 5:The Indicators used to measure MBSP’s success were oriented toward producing outputs rather than higher level results, which may have lead to unintended incentives The project’s results framework was nearly entirely output based. Even indicators associated with strategic objectives counted numbers of individuals applying techniques or numbers of loans being made without any check on the results of those applications or loans. In an effort to meet the targets associated with the project’s indicator framework, the MBSP rushed to complete some tasks, such as building and outfitting the hubs. However, building hubs in a way that is beneficial to producers and sustainable is complicated and takes time. Unfortunately, rushing to meet targets results in hubs with equipment but no electricity, or hubs that aren’t maintained properly. Project staff recognized that trying to complete 18 hubs was overshooting. The project rightfully acknowledged that building hubs in cities would be easier than rural areas, but rural areas are where the targeted population lived and most needed assistance. However, as staff pointed out, rural areas have fewer resources and building a hub within the project’s timeframe might not be the area’s highest priority. Staff recognized that completing four hubs completely would have served as an excellent model and provided a core for developing the next project. Perhaps a less output oriented results framework would have provided flexibility for that type of reorientation. 4. Recommendations and Conclusion The Evaluation Team has the following recommendations for NCBA CLUSA and USDA to consider when designing or managing future programs that are similar to the MBSP: Future project’s should learn from the MBSP’s leadership in establishing trust among millet sector actors: The MBSP identified early in the project that lack of trust among actors led to high transaction costs. In response the project took the lead in trust building by committing to a number of visible actions, such as coordinating the establishment of the hubs. Taking a leadership role in this manner persuaded 62 other actors to invest in trust. For example, producer organizations engaged with hubs to raise quality standards and work together, and processors coming to believe in the hubs’ higher quality standards and ability to deliver. Use a market facilitation approach: The MBSP donated and heavily subsidized infrastructure and equipment. These donations and subsidies included millet storage structures and cleaning equipment for hubs and kiosks for millet vendors. In some cases, the evaluation team found this equipment in disuse or not suitable for the user, implying wasted project resources. Instead of giving away the equipment or providing such generous subsidies, the future projects should work to link businesses and hubs to finance, using business and technical capacity building training to buy down risk. This market facilitation approach would identify actors willing to invest in equipment on the basis of its usefulness, built their capacity to utilize it more profitably and stretched project resources further. Maintain a regular conversation about project goals. The project had two strategic objectives, increase productivity, in the form of increased millet yields, and increase trade. External factors influenced the way these objectives were pursued. Poor rainfall made the achievement of increased yields impractical. Also, the project saw that there was an opportunity to increase trade even in the context of poor yields. However, the MBSP’s indicators, and USDA’s understanding of project strategy did not change despite the change in the project’s strategy. Future projects should keep an open conversation with USDA about the dynamic environments they operate in and how tactics change so that projects receive the support they need and their achievement is measured appropriately. Project’s results frameworks should focus on outcomes rather than mainly outputs: The indicators that measured the MBSP’s achievement were mainly focused on counting outputs. They measured the number of people that went through trainings, the number of producer organization members, the number of agreements signed, and the like. Many indicators appeared to report behavior change, such as “Number of hectares under improved techniques or technologies as a result of USDA assistance.” However, the actual measurement of this indicator and similar indicators involved estimating the number of people that had a attended a training and assuming that they were applying what they’ve learned. Future projects should include spot checks of farms, follow up interviews with trainees, and other methods to estimate the actual changes in behavior. These methods would allow project management to adjust activities that were not leading to intended outcomes, reinforce activities that were leading to intended outcomes, and allow USDA to have a better understanding of a project’s success. Improve project monitoring: Project monitoring provides many resources to project’s and organizations. Clearly, it is a means for a project to check its progress and fulfill the information needs of its donors. Also though, it can serve as a useful means of communicating with stakeholders, providing value to local government, and assist with the efficient management of resources. As detailed in the evaluation, the project’s monitoring and evaluation information quality was poor. Basic information about the project was unavailable, such as the number of trainings the project supported, the number of people that had attended trainings, the number of project beneficiaries, and the number of producer organizations the project served. The data that the project kept, such as its database of producers and producers’ organizations, was incomplete and contained unusable data. In the future USDA and NCBA CLUSA should provide monitoring oversight. USDA could provide this oversight by reviewing FAIS reports to make sure the data is consistent. For example, asking if project-to-date totals make sense; if the disaggregation of variables adds up to the total, as in the case of the disaggregation of women and men equaling the total of all individuals; and assisting implementers with the assumptions that underlie indicator measurements. NCBA CLUSA could assist projects by conducting annual evaluability assessments of its projects. An evaluability assessment would involve a staff member outside of the project reviewing project 63 documentation to make sure anything required to conduct an evaluation was present, and reviewing the project’s data to make sure it is clean, relevant, and up-to-date. Consolidation and harmonization of learning questions. The final evaluation TOR had 28 evaluation questions, the mid-term 25, and the baseline a total of 28 questions. These are exceptionally high numbers of questions that necessarily limit how well each question can be answered and reduce report readability. Furthermore there was little alignment in evaluation questions between the baseline and the mid-term and endline evaluations. Requiring a smaller number of focused and overlapping evaluation questions across baseline, midterm and endline would enhance the quality of evaluations and the opportunities for learning. Conclusion The MBSP was a highly ambitious project. Slated to start in 2014, the project did not begin in earnest until late 2016 because of the a global rice supply crisis and the late arrival of monetization funds. Thus, the project only had 3.5 years to put its strategy in place and realize results. Two of those years, 2016 and 2019, saw unusually low levels of rainfall which produced scant millet yields on which to base a project that hoped to increase millet yields and increase trade. The MBSP made a difference in the lives of its participants. The project conveyed new agriculture practices and improved access to inputs leading to increased yields for millet producers that participated in the program. It also taught those producers to improve the quality of their millet and increased the incomes of those that sold a portion of their harvest. Processors that participated in the program learned to improve their quality, branding, and packaging and gained access to a new and more diverse customer base. Other participants, such as financial service and input providers, found new points of entry into the millet value chain, and designed new products and services that resulted in new revenue. The project had some weaknesses. Its success relied on three achievements, creating hubs that served producers and were trusted by buyers, encouraging producers to improve quality in products and branding, and change customers perception and demand for products. Still, it took on other tasks that it did not have the time or resources to complete. For example, facilitating access to loans from financial service providers, building and equipping 18 hubs completely, or developing the input supplier market as thoroughly as it would have liked. The project’s output-oriented results framework did not help ameliorate these weaknesses. Measuring project success by counting outputs without and reference to the quality or effect of what is done created strange incentives for the project, as reviewed in the evaluation. Perhaps the weakest part of the project was its data management. The evaluation team understands from its interviews with CLUSA headquarters and project staff that there is a strong desire to statistically attribute the improvement in project participant’s results to the project’s activities. Unfortunately, without a complete sample frame, baseline data, control group data, or some amount of project collected data, attribution is extremely difficult. Nonetheless, the evaluation team believes that, based on data collected through the evaluation, and the project’s narrative as derived from project documentation, it is reasonable to believe that the project produced the results documented in this evaluation. Specifically, the project very likely increased millet yields, or mitigated millet losses, for farmers that participated in the project. Evidence that yields increased are seen through interview respondents’ consistent descriptions of their improved results and what led to those results. It is also captured in the expert responses captured at ITA, ANCAR, the DRDR interview, and government agencies. 64 Trade also very likely expanded as a result of the program. Respondents to the evaluation survey reported that their volume of millet sold has increased, that their income from millet increased, and respondent that they’d attended events hosted by the project and those events led to linkages that they’ve maintained. Interview respondents spoke specifically about how their customer base has broadened, and the factors that have led to increased sales. The evaluation team notes that the project’s results framework and performance indicators may not capture the project’s most important and lasting achievements. These are the establishment of the millet hubs, which serve as locally driven models for building trust and providing services in the business and farmer communities; the implementation of quality and branding standards, which respond to customer demands and build confidence in new products; the establishment of responsive events, which led to new financial products and input availability to farmers; and the development of points of market entry, such as the kiosks that brought new products to customers and provided a new sales channel for processors. 65 Annex A: Evaluation Terms of Reference TERMS OF REFERENCE FINAL EVALUATION Millet Business Services Project (MBSP) Award Number: FCC-685-2014/055-00 August 25, 2020 I. Terms of Reference The Terms of Reference is aimed at soliciting proposals from qualified individuals/organizations interested in implementing the final evaluation of the “Millet Business Services Project” (MBSP), funded by the United States Department of Agriculture (USDA) and implemented by the National Cooperative Business Association, Cooperative League of the USA (NCBA CLUSA ) in Senegal. The terms of reference calls for a team of short-term evaluators who must have relevant prior experience in West Africa, strong familiarity with millet and/or related agricultural value chains, and demonstrated prior experience in undertaking reviews, assessments, and evaluations. Activity Title Millet Final Performance Evaluation Project Name USDA Millet Business Services Project (MBSP) Agreement Number FCC-685-2014/055-00 Donor United States Department of Agriculture (USDA) Project Duration October 2014- December 2019 Funding USD $11,181,193.11 Implementing Organization National Cooperative Business Association’s (NCBA) Cooperative League of the USA (CLUSA) Partner Organization Catholic Relief Services (CRS) Geographic Coverage Kaolack, Fatick, Kaffrine, and Dakar regions in Senegal II. Project Background NCBA CLUSA is implementing a four-and-a-half-year agricultural program hereunder known as the MBSP, funded by the United States Department of Agriculture (USDA). The project began in October 2014 and is scheduled to end in December 2019. It seeks to work with 22,150 producers, 16,630 trade beneficiaries, government agencies, and the private sector particularly in the millet producing/processing regions of Senegal including the specific areas of Kaolack, Kaffrine, Fatick, as well as the capital, Dakar. MBSP is a follow-on project to the Millet Value Chain Project (MVCP) implemented by NCBA CLUSA from May 2009 to January 2014. MVCP focused on setting up millet producer organizations and working heavily with the millet farmers. The current project still works with millet farmers but has an added focus of working with input suppliers and millet processors. 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 67 The objectives of the program are: 1) Increase agricultural productivity of the millet value chain by: developing processing systems; building capacity of producers to improve production and quality; training producers and processors in improved techniques, post-harvest handling, marketing, and seed production; and providing grants and loans for equipment and inputs; and 2) Expand trade of millet by developing public-private relationships, facilitating trade relationships, researching export opportunities, building agricultural extension capacity, and promoting food safety issues and requirements. MBSP activities are categorized into the follow major components: a) Capacity Building: Agricultural Extension Agent Services, Producers, and the Private Sector NCBA CLUSA supports government extension services by working with the Senegal Institute for Agriculture Research (ISRA, in French) to provide refresher trainings in the latest millet production research to Direction Régionale de Développement Rurale (DRDR) agents. NCBA CLUSA also strengthens the extension capacities of DRDR by facilitating contact with millet unions and associations as well as strengthening the capacity of existing millet unions and other high-producing millet groups to develop internal extension services. In addition to providing support to public sector extension services, NCBA CLUSA works with local firms such as the seed and the conservation agriculture equipment unions, to provide technical information to farmers on how to use their products. NCBA CLUSA provides grants and also assists input suppliers to train in the necessary extension services associated with their products. b) Financial Services: Facilitate Agricultural Lending 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 68 NCBA CLUSA works to increase access to credit so that farmers can purchase needed inputs. NCBA CLUSA works to identify micro finance institutions (MFIs) that are interested in establishing credit services and helps them develop commission-based credit intermediaries, then supports these MFIs by providing grants to buy down risk. Communities select village-based credit intermediary service providers (CBSPs) based on financial skills. Where necessary, CISPs support POs to revise or develop business plans, which is the basis of agreement with MFIs. NCBA CLUSA works to increase access to credit for processors and field based post-harvest businesses so that they can purchase post-harvest machinery and other equipment, or secure operating loans to buy appropriate quantities of raw millet. NCBA CLUSA increases access to credit by strengthening and operationalizing the business plans of selected processing firms, threshing providers, and warehouse teams. NCBA CLUSA hosts regional meetings to build linkages between millet processing firms and MFIs. c) Training: Improved Agricultural Production Techniques NCBA CLUSA links private sector seed suppliers with farmers to provide information on correct seed planting and fertilizer use. NCBA CLUSA arranges for DRDR and Producer Organizations (PO) extension agents to train farmers in spacing, rotation practices, striga control and improved weeding schedules. NCBA CLUSA continues to support conservation agriculture practices such as minimum tillage agriculture using ripper tines instead of disc ploughs, water harvesting, composting, intercropping for nitrogen fixation, crop rotation, retention of crop residue, precise spacing of crops and precise application of inputs, and usage of basins with millet production in order to retain soil organic matter, reduce erosion, and increase fertilizer efficiency to improve the quality of land resources. d) Inputs: Develop Agri dealers and Input Suppliers NCBA CLUSA employs a market facilitation approach to input provision, reinforcing the efficiency of millet production business services so that they, in conjunction with public and PO extension services rather than NCBA CLUSA, support millet farmers’ increase in production capacity. NCBA CLUSA strengthens the capacities of four business services (seed providers, fertilizer providers, Conservation Agriculture (CA) machinery providers and CA plowing services) to increase coverage in the targeted area. NCBA CLUSA assists targeted firms to buy down risks with the provision of grants. e) Training: Post-Harvest Handling and Processing NCBA CLUSA facilitates discussion between millet processors and producers in regard to the quality of the millet they require. Processors formalize the standards required in contracts. NCBA CLUSA also facilitates discussion between millet processors and warehouse managers on proper storage techniques and train warehouse managers in financial, logistic and administrative management. NCBA CLUSA trains processing businesses in four types of activity; 1) Value added production; 2) Diversification of production; 3) Packaging; and 4) Food Safety and Hazard Analysis and Critical Control Points (HACCP). As millet firms purchase new processing technologies, NCBA CLUSA ensures that processors are appropriately trained. Where market analysis shows potential, NCBA CLUSA trains millet processing businesses in product diversification, with a particular focus on micronutrient enrichment. f) Infrastructure: Post-harvest Handling and Storage NCBA CLUSA facilitates the purchase of threshers through small grants or loans with MFIs by producer unions and third-party providers interested in investing in threshing provision, in order to reduce post-harvest loss experienced through traditional threshing methodologies. Additionally, NCBA CLUSA assists local construction service providers to refurbish the warehouses of millet unions. NCBA CLUSA requires PO members to contribute 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 69 labor to carry out the refurbishing work, while a combination of savings, credit and grants are used to purchase materials and to pay local masons and other specialized services. g) Infrastructure: Post-Harvest Processing NCBA CLUSA facilitates the purchase of processing equipment by millet processing firms to improve quality and reduce spoilage of the product and increase diversification of the product line. NCBA CLUSA ensures processing firms finance the purchase of threshing infrastructures, mostly through credit. NCBA CLUSA also uses grant funds to cover up to 45% of the cost as necessary, until this technology has been tested, adopted and proved. NCBA CLUSA engages with millet firms to promote three types of processing infrastructure: grading and processing equipment, packaging technologies that reduce spoilage, and equipment for micronutrient fortification. h) Training: Improved Marketing and Branding NCBA CLUSA builds upon the establishment of marketing specialists in producer unions and associations to provide marketing training to relevant producer committee members on how to better market their products through improved communication and networking, contract negotiation skills, and increasing price through bulking and quality control. NCBA CLUSA also works with processing firms to register their products through national registration procedures in order to sell their products in supermarkets and other larger shops. i) Market Access: Facilitate Buyer-Seller Relationships NCBA CLUSA facilitates linkages at three levels: 1) between producer organizations and processors, 2) between processors and retailers; and 3) between machinery service providers and processors. NCBA CLUSA ensures that these entities have requisite business planning and negotiations skills in order to create the linkages that will lead to the expansion of contracts outlining quality demanded and price paid. NCBA CLUSA has interventions along the entire millet value chain in order to reach activity objectives with optimum impact. The following actors are currently part of the project operations: 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 70 Table-1: Type of MBSP Value Chain actors by intervention Value Chain Partner Intervention Producers NCBA CLUSA’s interventions with producers include: training 28,666 millet producers in agriculture technologies via demonstration plots, training in extension techniques (general and advance), training in administration and finance tools as well as basic agricultural marketing. Processors NCBA CLUSA’s interventions with processors include: training approximately 440 millet processing units on processing and preservation techniques and training shopkeepers and managers of pre-processing units, renovating and building processing units and demonstration units, equipping units with processing equipment, training processors and millet kiosk managers on marketing, and training on CEFE (Economic Competence through Entrepreneurial Training). Coops & POs NCBA CLUSA’s interventions with Co-ops and POs involve millet producers and processors who are part of different Cooperatives, Producer Organizations, and Producer Associations. NCBA CLUSA trains millet farmers and processors in new techniques and technologies, and financial planning and management, organizes assessments to determine PO and co-op level of professionalism, renovates existing warehouses and builds new warehouses with a total capacity of 16,172 square meters, and teaches POs the application of new practices. Input Suppliers NCBA CLUSA’s interventions with input suppliers include: organizing agricultural suppliers into 6 networks, hosting dynamic organization training for the newly created input supplier networks, training input suppliers on marketing, training input suppliers on extension, organizing exchange visits for project partner networks and CBSPs, supporting the professionalization of 35 commercial CBSPs, and training in CEFE (Economic Competence through Entrepreneurial Training). Buyers - Sellers NCBA CLUSA’s interventions to connect buyers and sellers include: facilitating the implementation of “Sunu fondé” kiosks, organizing National Millet Day, participating in trade shows, supporting processing companies for export, improving processor products brand image by supporting the production of logos, small bags, and labels, facilitating the signing of contracts with commercial CBSPs and businesses, providing support to businesses with new ventures, particularly large chain stores and export support. Financial Service Institutions NCBA CLUSA’s interventions for Financial Services Institutions include: organizing 5 intermediation workshops between input suppliers, MFIs, and producers, establishing Savings and Internal Lending Communities (SILC) groups, training SILC methodologies for 17 CBSP-IF (financial intermediaries), collecting data concerning “scattered” loans, creating or updating business plans for producer organizations (21 POs) and processing units (39 units), hosting 4 financial workshops to connect stakeholders from the millet value chain, collaborating with the Professional Association for Decentralized Financial Systems (APSFD). Government Agents NCBA CLUSA’s interventions for government agents include: developing a training program for government agents on millet and its importance via greater dissemination of this crop among the various targets in the project intervention zone. Government agents include those primarily involved in the millet value chain activities (DRDR, ANCAR, 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 71 ITA, ISRA, Chamber of Commerce, Chamber of Trades, etc.). This program essentially revolves around training sessions being held and led by a task force of researchers from the Senegalese Agricultural Research Institute (ISRA). III. Purpose and Scope of the Final Evaluation The purpose of this Final Evaluation is to provide an independent assessment and analysis of the overall effectiveness, impact, sustainability, and the degree to which MBSP has accomplished its objectives and results as outlined in the results framework. The evaluation has the following specific objectives: 8. Assess project performance in meeting the end of the project activity and result targets; 9. Assess the relevance of the project; 10. Assess project effectiveness; 11. Assess efficiency of systems and processes; 12. Assess the degree of monetary and socio-economic impact of the project on the project beneficiaries in comparison with the baseline and mid-term evaluation results; 13. Assess the sustainability of the project by reviewing the capacity of producers, processors, and related value chain actors to continue providing support to MBSP millet production systems for potential expansion; 14. Document best practices and lessons learned and assess the application of recommended actions from the mid-term evaluation; This evaluation will capture data from all four project regions i.e., Kaolack, Fatick, Kaffrine, and Dakar using an appropriate mix of documents review, field data collection, and interviews. A baseline study was conducted in August 2015 and provided substantive data on project indicators in addition to project start-up and the initial phase of implementation. The project has also undergone an extensive Mid-Term Evaluation completed by J.E. Austin Consulting between October 2018 and January 2019. The final report documents substantial information on project background with a full assessment of project accomplishments, implementation, and operations. It is expected that the Final Evaluation will fully utilize and build on these two previous project evaluation studies to avoid repetition and expand on assessments where necessary. Access to MBSP Evaluation reports and corresponding data collection tools will be made available to the successful Evaluation team. Specific evaluation questions are organized by USDA’s recommended framework that includes components such as relevance, effectiveness, efficiency, and sustainability in addition to measuring goal achievement, impact, best practices and lessons learned (table 1) Table-2: Evaluation questions Evaluation Component Evaluation Question Goal Achievement  To what extent have project goals been achieved?  Which activities have been successful in meeting their targets?  Which activities failed to reach their targets and why? Include analysis of factors that delay/prohibit adequate progress towards results. Relevance  Are the project activities in alignment with Senegal Government priorities and strategies?  Do the project activities respond to the aspirations and needs of the project beneficiaries/participants?  Are the project activities in alignment with USDA/US Government development goals, objectives, and strategies? 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 72 Evaluation Component Evaluation Question  Are the project activities in alignment with award agreement with USDA? Effectiveness  To what extent have different actors in the value chain been able to increase quantity e.g., yield/ha and minimize waste since the inception of the project?  To what extent have quality standards been adopted by all actors in the value chain?  To what extent value chain actors have been trained in quality standards?  Are partner organizations providing training to beneficiaries that are tailored to the local context and achieving increased use of agricultural techniques and technologies?  Are partner organizations increasing the use of improved post-harvest processing and handling practices, and improved marketing and branding?  Are local government extension officers offering technical support to farmers in new techniques and technologies?  Have effective linkages been established between different project participants e.g., between producers and processors? Efficiency  What internal and external factors have influenced the ability of the project to meet expected results and targets?  Do project staffing, training activities and management structures efficiently support the achievement of expected project outcomes?  Does the project resource allocation to activities and to the project’s geographic areas efficiently support the achievement of expected project outcomes? Sustainability  What factors are likely to contribute towards the sustainability of the following factors in relation to the MBSP: policy environment, financial incentives (i.e., profit motive), access to inputs and information, market access, etc.  What factors are likely to hinder sustainability of the following factors in relation to the MBSP: policy environment, financial incentives (i.e., profit motive), access to inputs and information, market access, etc.?  What project activities are likely to contribute towards sustainability and is there evidence that project activities or benefits will be sustained after the project ends? Impact  What impacts are the project activities having on the program participants, both positive and negative, especially in relation to the expected results and strategic objectives?  To what extent have target groups (farmers, processors, vendors, consumers, etc) adopted millet as a viable crop that has economic and nutritional benefits?  What is the extent of the project reach beyond direct and targeted beneficiaries?  What are the unexpected outcomes that could have been due to the project? 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 73 Evaluation Component Evaluation Question  What is the overall impact on increasing trade of millet? (marketing strategies, creation of new partnerships, market information systems, certifications, etc.)  How has the trade increased as a result of USDA MBSP on local, regional, national, and international level? Best Practices and Lessons Learned  What lessons learned can be documented at this point to inform implementation of the program until completion?  What best practices can be shared with stakeholders and donors? IV. Evaluation Approach and Methodology The successful evaluator shall collaborate with NCBA CLUSA staff to develop and document the most rigorous and appropriate approach and methodology for the evaluation that offers relevant answers to the evaluation’s objectives and key questions. The methodology should include but not be limited to: document review; key stakeholder interviews, survey questionnaires, expert consultations, case studies, and direct observations. The evaluator will develop and pilot test appropriate field data collection methods and tools, consulting the baseline and midterm survey questionnaires to ensure comparability. The Evaluator will conduct a review of relevant project records and data from NCBA’s field activities. The evaluator may consult other documentation from NBCA CLUSA’s Washington and/or NCBA CLUSA’s MBSP office in Kaolack, Senegal. The Evaluation is expected to collect data in all four regions within the project to represent a wide area of implementation as well as taking gender into consideration when choosing potential participants to sample. All MBSP project supported facilities including demo plots, processing centers, kiosks, hubs etc. will also be available for observation as relevant to the MBSP activities. The Evaluator will identify any data gaps for further investigation. There will be time spent in Kaolack where MBSP’s main office is located and in the project implementation Districts. The Evaluation team and NCBA CLUSA staff will develop a more detailed travel schedule based on the evaluation team’s requirements prior to arrival in country. The USDA Monitoring and Evaluation Policy supports a participatory evaluation process. This helps to ensure the quality, validity, utility and mutual ownership of the evaluation findings and recommendations. The evaluation methodology is to be approved by USDA and NCBA CLUSA prior to commencing the in-country phase of the evaluation. It is expected that the evaluation team will liaise closely with relevant NCBA CLUSA staff during the development of the evaluation methodology and implementation. The selected evaluator will implement the following activities during the evaluation process: a) Undertake a literature review of relevant documents from country reports, national policies and regulations, special studies carried out by other agencies, most recent zone of influence level population-based survey conducted by FTF, and project documentation in this area; b) Gain a full understanding of the project operations, implementation schedule and project indicators; c) Using project documentation, calculate the economic impact of project interventions by the project including potential economic impact 5-year post project for project beneficiaries; d) Develop study design and methods that will be mixed designed involving quantitative and qualitative. This will include review of secondary data sources i.e data from project M&E system, project activity and beneficiary record systems. Design appropriate survey tool(s) for data collection from project farms and 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 74 value chain partners including farmer households, farmer groups, agro-input small farmer and cooperative nursery businesses, and other partners which considers but is not limited to the following items: o Identify a representative sample of value chain partners to be interviewed / surveyed that includes all four districts targeted by the project; o Pre-testing, editing, translation, finalization, and reproduction of interview/ survey instruments (final survey tools to be approved by NCBA and USDA prior to use); o Integrate qualitative data collection methodology and survey tools; o Apply statistical techniques for designing, analyzing and reporting quantitative data; e) Develop data collection guide specifying data collection and management structure, field schedules and data quality assurance methodology. o Coordinate, train, orient and supervise field interviewers and enumerators as necessary; o Coordinate field data gathering using tested survey tools to address objectives of this study and apply quality assurance; o Process and analyze collected survey data; o Perform quality assurance during design, testing, fieldwork, data entry, data analysis and reporting; o Organize all data collected into a database. This database will ensure anonymity of survey participants by using identification numbers and separating names from survey participants’ data; o Using statistical methods where possible, undertake a comparative analysis of the evaluation findings against the project baseline and mid-term study results; o Report the study results in appropriate tables, charts and narrative; o Present findings, conclusions and recommendations (in both written report and PowerPoint formats). Table-3: illustrative Evaluation Methods Value Chain Actor/Partner Selection Method Data Collection Method Direct and indirect beneficiaries Use project’s beneficiary listing as a sampling frame Treat each region as a stratum Determine the sample size for each region utilizing appropriate sample size estimation formula. Select sample randomly from each stratum. Household survey to capture quantitative information M&E report and other project document review Cooperatives and producer organizations (PO) Quantitative Data Use project’s cooperatives and POs listing as a sample frame Treat each region as a stratum Determine the sample size utilizing appropriate sample size estimation formula. Document reviews Structured survey to capture qualitative data Focus group discussions and Key Informant 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 75 Apply stratified random sampling methods to select participants from each region. Qualitative Data Use of availability and snow-ball sampling method to select focus group participants. Interviews to capture qualitative information. Agro-dealers; input suppliers; buyers and sellers; private sector companies Quantitative Data Census instead of sample as the number is manageable. Qualitative Data Use of availability and snow-ball sampling method to select focus group participants. Document reviews Structured survey to capture qualitative data Focus group discussions and Key Informant Interviews to capture qualitative information. Government officials (local and national) Purposeful and availability sampling method Document reviews Key informant interviews Storage, handling, processing units Quantitative data Census instead of sample as the number is manageable. Qualitative data Availability and snow-ball sampling method to select focus group participants Document review Structured survey to collect quantitative data Focus group discussions and Key Informant Interviews to capture qualitative information V. Data Analysis Data will be analyzed at the regional and project level. The regional level analysis will include exploration as well as explaining of the emerging themes and concepts within each region of Kaolack, Fatick, Kaffrine, and Dakar (if appropriate). Project level analysis will involve putting the emerging themes and concepts and explanatory factors from all the regions together. Quantitative data will be analyzed using SPSS or similar statistical solutions. The analysis will involve descriptive analysis that will be presented in tabular/chart formats. Within region and across region analysis will be conducted for each indicator and participant category. The evaluation team will develop a code book that will include codes for location, participant type, questions and response categories. Analysis of qualitative data will follow the framework of data reduction, data visualization, and conclusion drawing and verification. Data reduction phase will include selecting, focusing, simplifying, abstracting, and transforming the data that appear in written-up field notes or transcripts. All the field notes will be transcribed first in the local language and then will be translated in English. Findings from the document review will be noted in the document summary forms. Data will be presented in visual format that will help conclusion drawing 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 76 and verifications. Finally, the team will draw conclusions based on regularities, patterns, explanations, causal flows, and propositions VI. Addressing Limitations Due to the complexity of the different components of the project and the spatial location of its activities, opportunities for errors exist. The evaluation team is expected to be aware of such complexity and develop strategies to minimize opportunities for the errors. The evaluation team will develop quality control systems, including developing detailed data collection guides and overseeing field data collection. This evaluation will heavily rely on secondary performance information reported in bi-annual and annual reports. The quality of performance data will affect the accuracy of projected trends. The evaluation team should check the validity and the reliability of performance data before analyzing it. Primary data collected from beneficiaries may reflect the opinions of the most dominant groups without capturing the perceptions of less vocal groups. The evaluation team should take this into account and make sure that all parties are freely expressing their views. The evaluation team will mitigate this potential oversight by organizing focus groups based on the need to ensure that participants can speak freely during discussion. VII. Deliverables Deliverables under this assignment include the following: 1. Initial presentation: A PowerPoint presentation on methodology, sampling design and sample size calculation. The presentation should not be more than 15 slides; English and French 2. Weekly Progress Reports: A written and/or electronic report of the evaluation’s progress made in the field covering key scheduled activities, completion status and found constrains in regard to data collection; English or French 3. Inception report that will include methodology, sampling approach, data analysis, field procedures, and update on timeline 4. Data collection and analysis tools: Set of questionnaires, formats, and Excel spreadsheets used to collect and analyze data, and their implementation guidelines; French 5. Data files: raw field notes, transcribed notes, tabulated data 6. Submission of Draft Evaluation Report: The team will submit a draft report to the NCBA CLUSA key staff (Project COP and M&E Specialist; and Program Manager at HQ) and USDA Analyst, who will provide comments for revision; English 7. Final Report: A written and electronic document that includes the following: a. Executive Summary; b. Highlights of the evaluation c. An introduction containing the objectives of the Terms of Reference (ToR) and a brief description of the program with relevant references to the MBSP baseline and mid-term evaluation reports; d. Background of physical, geographic, agricultural, and socio-economic conditions surrounding the project; e. Evaluation methodology; f. Data limitations; g. Results and findings by summary and by project locations (including narrative, charts, and tables); h. Reponses to Study Objectives with reference to survey data and analysis; i. Table of responses to Key Questions with references to supporting survey data and analysis; 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 77 j. Lessons Learned; k. Conclusion with Recommendations; l. Annexes: o Evaluation TOR o Composition of the team o List of sites visited o List of key informants o References o Indicator Performance Tracking Tables (IPTT) o Survey tools o List of acronyms 8. Final presentation: A PowerPoint presentation on results and conclusion of the evaluation. The presentation should not be more than 15 slides; English and French 9. Some specific impact quotes from the various stakeholders from interviews and focus group discussions. Quotes should contain identifying details about the speaker (i.e., farmer v. kiosk owner, types of training, storage, etc.); 10. A selection of at least 5 project success stories that can be shared externally; 11. Electronic copies of 20-25 high quality photographs (minimum 300 dpi resolution photos) of some program evaluation data collection activities i.e. surveys, interviews, and focus group discussions among others. 12. 2-4-page nontechnical summary brief. The brief should summarize the main findings of the report and include any charts, graphs, etc that help visualize the data in a clear, easy format that is accessible to all intended audiences of the report. 1. VIII. Audience and Intended Use This evaluation will help measure project performance and measure the effectiveness, efficiency, and relevance of the activity’s approach in achieving its stated goals. Performance results, lessons learned, and the success of the supported activity are to be shared with USDA to inform future activity design. The audience will include all key stakeholders e.g., USDA, NCBA CLUSA, partner organizations e.g., program participants, local government departments including ANCAR and DRDR, public institutions, including ITA, private sector partners, and other non-governmental organizations working in synergy with the MBSP. Other audience may include individuals and groups who are not necessarily engaged in the project activities but may benefit from the findings that may be published and disseminated through CLUSA website and/or presentation in national and international conferences. As recommended in the USDA Monitoring & Evaluation Policy, all USDA final evaluation reports will be made publicly available on the FAS website. In addition, USDA may use this report to regularly publish information on project and program level results and accomplishments. This will ensure that the widest audiences as possible are reached and that other organizations learn from FAS’s experiences. Principled exceptions may be made where classified, personal or proprietary information is concerned. In a situation where the evaluation report includes 1) proprietary information owned by third parties and/or 2) information that could put individual safety at risk or personally identifiable information, the team will ensure that this information will be removed before the evaluation is released. 2. IX. Evaluation Management and Coordination 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 78 Per the requirement of USDA policy this evaluation will be conducted by independent third-party evaluators. NCBA-CLUSA will provide logistic support that will also include inviting stakeholders for the meetings. Evaluator will present methodology and findings to USDA official and CLUSA team. Draft report will be shared with USDA for their review and comments. The report will be finalized after the approval from USDA. Table-4: Approximate Timelines of Events Event Timeline Develop a scope of work for the evaluation and list of required qualifications for the external evaluation team, post a call for proposals Three months prior to evaluation (August September 2019) Review applications Two months prior to evaluation (September 2019) Select external evaluation team/consultants One/Two months prior to evaluation (September / October 2019) Work with the external evaluation team to finalize project evaluation TOR and agree on evaluation methodology One months prior to evaluation/ Approximately beginning of (October 2019) Pilot test of the methodology and training of the evaluation team Approximately mid November 2019 Field work – data collection (interview, focus groups) 3-4 weeks in November/ December 2019 External evaluators prepare analysis documents and reports One month after the evaluation activities (Janvier 2020) Program team (COP, senior leadership in HQ, M&E Specialist, technical teams where appropriate) review evaluation report and analysis, as well as lessons learned and other documentation One months after evaluation (Janvier 2020) Submit final project evaluation report to USDA Within 15 days following end of review process (January 2020) Presentation of findings to stakeholders Two to three weeks weeks after evaluation (January 2020) Report on implementation of follow-up actions January 2020 X. Team Composition An outside firm/external evaluator will be selected for the final evaluation. Through this competitive process NCBA CLUSA will select an evaluation firm who have local expertise and experience evaluating agricultural and trade initiatives with similar target audiences. Major requirements for the evaluator will include: • Demonstrated knowledge, analytical capability, language skills and experience in conducting evaluations of development programs involving agriculture, trade, and marketing in Senegal. • Technical skills and capacity in the application of analytical frameworks such as comparison with non￾project areas, surveys, involvement of stakeholders in the evaluation, and statistical analyses. • Have knowledge of US Government projects including either USDA or USAID. 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 79 • Have experience managing or evaluating agricultural value chain projects. • Preferably have experience in sub-Saharan Africa development projects. • Ability to travel to rural areas of Senegal on difficult roads to meet with stakeholder partners. • Budget Competitiveness • Competency to write detailed, concise and coherent final evaluation report. • Gender balance and cultural sensitivity 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 80 XI. Conditions of application Consultants and companies wishing to carry out this mission must send the following items (12-point font) in the order specified below to be taken into account during selection: A. Technical proposal The technical proposal (12 pages max.) Must reflect how the offeror will carry out the tasks included in the work. Candidate companies will provide a detailed plan of the specific activities, the timetable for carrying out the mission, as well as the data collection and analysis activities. In addition, it will include a proposal for the statistical approach. B. Financial application The candidate companies/consultants will propose a realistic estimate of the cost of this mission, including a detailed budget and a justification of the expenditure. The budget will only contain costs that can be directly attributed to the proposed activities, with an explanation of the line items. All training costs, such as rental of premises, advance transport, etc. must be clearly articulated for each training. Applicants must present adequate administrative and financial systems to manage the funds covered by this agreement. If the candidate companies charge overhead, they will need to provide their NICRA. The maximum amount will not exceed $150,000, but competitive budgets will receive a higher score. C. Organization The candidate companies/consultants must briefly list and describe their history, vision, objectives, legal status, ownership and management structure, current projects / services, current clients / assignors, current geographic scope and experience. D. Personnel The candidate companies/consultants must briefly list and describe the names and qualifications of the key personnel assigned to the assignment (the CVs of the proposed staff must be included in the Annex). The proposed team should preferably be multidisciplinary and include at least (A) a team leader for monitoring and evaluation, with extensive experience designing and implementing evaluations and analyzing the results (ex: a statistician and / or analyst familiar with the analysis of quantitative and qualitative data) Experience in the target regions is desirable; (B) an agricultural economist, a rural development specialist or a person with an equivalent level of competence, specializing in economic development and data collection activities.; (C) a database and information system specialist; (d) a data entry team; and (e) enumerators. E. Proof of Experience Applicant companies/consultants will have to prove their experience in similar missions by providing a list of all contracts and / or cooperative agreements involving similar or related work during the last two years. Additionally, the applicants must submit at least three reference letters from previous consultancies. F. Legal registration Candidate companies/consultants’ engagement is subject to the consultant obtaining necessary visas and work authorization if needed. XII. Criteria for the evaluation of the proposal The following criteria are those under which all proposals will be judged: Quality of technical approach and methodology (40%) 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 81 Past experience of individual and / or company with USDA and / or USAID or other International Development Organizations (10%) Demonstrated experience and technical skills of the team/reference letters (15%) Completeness of proposal (including schedules, total budget, employee CVs, etc. (15%) Cost realism, budget justification and effectiveness. Given it meets these standards, competitive budgets will receive a higher score. (20%) XIII. Application Interested parties must submit their CVs, references, and requested documents under the submission requirements in the timeframe provided. • Submission mush be in English or in French and typed single-spaced. • The pages must be numbered, and each page must contain the name of the company. • Proposals must be sent by e-mail to NCBA CLUSA at the following addresses: lakanvou@ncba.coop and lgomis@ncbaclusa.net • Subject of the email should be: USDA|MBSP Final Evaluation • Applications must reach the project by September 16, 2019 at 5:00 pm EST (Washington D.C. time) at the latest. Please include the name of the person in your organization sending the application, as well as the phone number, fax number and e-mail address. If you need more details, please contact NCBA CLUSA offices by email at lakanvou@ncba.coop. NCBA CLUSA reserves the right to subsidize all or none of the applications submitted and / or to modify the terms of reference / geographical areas before the project begins. Theory of Change Improved Farm Management Operations (Operations, Financial) (FFPr 1.3) Improved Knowledge Regarding Farm Management (FFPr 1.3.1) Activity: Training: Improved Agricultural production techniques Increased Agricultural Productivity (FFPr SO1) Improved Quality of Land and Water Resources (FFPr 1.1) Increased Availability of Improved Inputs (FFPr 1.2.1) Activities: Inputs: Develop agro-dealers and/or other input Increased Use of Improved Agricultural Techniques and Technologies (FFPr 1.2) Increased Use of Financial Services (FFPr 1.2.3) Activities: Financial services: Facilitate agricultural lending Increased Knowledge by Farmers of Improved Agricultural Techniques and Technologies (FFPr 1.2.4) Activity: Training: Improved Agricultural production techniques Foundational Results Project Level Results Framework #1 of 2 Improved Capacity of Key Groups in the Agriculture Production Sector (FFPr 1.4.4) Activity: Capacity building: Agricultural extension agents/services Increased Capacity of Government Institutions (FFPr 1.4.1) Activity: Capacity building: Agricultural extension agents/services 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 83 Project Level Results Framework #2 of 2 Foundational Results Expanded Trade of Agricultural Products (Domestic, Regional, and International) (FFPr SO2) Increased Efficiency of Post-Production Processes (FFPr 2.1.2) Improved Post-Harvest Infrastructure (FFPr 2.1.2.2) Activity: Infrastructure: Post harvest handling and storage; Infrastructure: Post harvest Increased Access to Markets to Sell Agricultural Products (FFPr 2.2) Increased Value Added to Post￾Production Agricultural Products (FFPr 2.1) Improved Marketing of Agricultural Products (FFPr 2.2.1) Activity: Training: Improved marketing and branding Increased Use of Improved Post￾Production Processing and Handling Practices (FFPr 2.1.2.1) Activity: Training: Post harvest handling and processing Improved Linkages Between Buyers and Sellers (FFPr 2.2.2) Activity: Market access: Facilitate buyer-seller relationships Improved Capacity of Key Organizations in the Trade Sector (FFPr 2.4.4) Activity: Capacity Building: Agriculture extension Increased Use of Financial Services (FFPr 2.2.3.1, 2.3.1.1) Activities: Financial services: Facilitate agriculture lending Annex B: Works Cited and Documents Reviewed Works Cited Agence Nationale de la Statistique et de la Démographie (ANSD), DHS Program. (2017). Enquete Demographique et de Sante Continue (EDS-Continue). Aker, J. C., Block, S., Ramachandran, V., & Timmer, P. (2010). West African Experience with the World Rice Crisis, 2007–2008. FAO. ANSD/Senegal and ICF. (2017). DHS (Demographic Health Survey). Brooks, G. E. (1975). Peanuts and Colonialism: Consequences of the Commercialization of Peanuts in West Africa, 1830-70. The Journal of African History, 16(1), 29-54. Brooks, G. E. (1975). Peanuts and Colonialism: Consequences of the Commercialization of Peanuts in West Africa, 1830-70. Journal of African History, 16(1), 29-54. Buggenhagen, B. (2011). Are Births Just ‘Women's Business’?: Gift Exchange, Value, and Global Volatility in Muslim Senegal. American Ethnologist, 38(4), 714–732. BUGGENHAGEN, B. (2011). Are Births Just ‘Women's Business’?: Gift Exchange, Value, and Global Volatility in Muslim Senegal. American Ethnologist, 38(4), 714-732. (2019, December 10). Buyer, Maria Distribution. CLUSA. (2019). April - September 2019 Progress REport. Direction de l’agriculture de la Prevision et des Statistiques agricoles. (2015). (2019, December 18). Director, Crédit Mutuel du Sénégal. (E. Team, Interviewer) Dong, J. (2011). Millet Has Many Faces. Dakar: USDA Foreign Agriculture Service. Famine Early Warning System Network. (2015). Senegal Livelihood Zone Map. Feed the Future. (2018). Feed the Future Innovation Lab for Collaborative Research on Sorghum and Millet Annual Performance Report FY 2018. . Feed the Future Innovation Lab for Collaborative Research on Sorghum and Millet . (2018). Annual Performance Report . Food and Agriculture Organization of the United Nations (FAO) . (1961 to 2013). Food and Agriculture Organization of the United Nations (FAO) Database. G. Basavaraj, R. P. (2010). Availability and utilization of pearl millet in India. Journal of Semi-Arid Tropical Agricultural Research, 1-6. G. Basavaraj, R. P. (2010). Availability and utilization of pearl millet in India. Journal of Semi-Arid Tropical Agricultural Research, 8, 1-6. Guèye, C. e. (2007). Climatic Perturbation and Urbanization in Senegal. The Geographical Journal, 173(1), 88-92. Ibrahima Hathie, B. S.-J. (2017, October). Ending Rural Hunger, the Case of Senegal. Retrieved from Focus on Africa - Brookings Institute: https://www.brookings.edu/wp-content/uploads/2017/10/senegal-erh￾policy-brief.pdf 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 85 Ibrahima Hathie, B. S.-J. (2017). Reducing economic and environmental vulnerability to achieve food and nutrition security in Senegal. Retrieved from Africa in Focus - Brookings Institute: https://www.brookings.edu/blog/africa-in-focus/2017/10/18/reducing-economic-and-environmental￾vulnerability-to-achieve-food-and-nutrition-security-in-senegal/ (2019, December 2011). ICS Sales Manager. (I. E. Team, Interviewer) J. P. Wilson, S. K. (2008). Evaluation of pearl millet for yield and downy mildew resistance across seven countries in sub-Saharan Africa. African Journal Agricultural Research, 3, 371-378. J. P. Wilson, S. K. (2008). Evaluation of pearl millet for yield and downy mildew resistance across seven countries in sub-Saharan Africa. African Journal Agricultural Reserch. Jacques, D. C. (2018, January 12). Social capital and transaction costs in millet markets. Heliyon, 4, 1. JE Austin Inc. (2018). Senegal Millet Business Services Project; Mid-Term Evaluation. Washington, DC: USDA. Jenny C. Aker, S. B. (2010). West African Experience with the World Rice Crisis, 2007–2008. FAO. Jere Brunken, e. a. (1977). The Morphology and Domestication of Pearl Millet. Economic Botany, 31(2), 163-174. John, A. (2015). The origins of Senegal's dependency on rice import. Rice Today(14), p. 28. Masters, W. A. (2007). Distortions to Agricultural Incentives in Senegal. Retrieved from http://documents.worldbank.org/curated/en/408461468336719859/pdf/560430NWP0SN0v1PUBLIC10 Senegal10708.pdf (2019, December 19). MBSP Staff Interview. NCBA CLUSA. (2016). Rapport des Fora Régionaux sur les mé du Mil à Dakar-Fatick-Kaffrine-Kaolack. NCBA CLUSA. (2017). Final Narrative Report 2017, pg 2. NCBA CLUSA. (2017). Rapport De Business Forum de Touba. NCBA CLUSA. (2017). Semi-Annual Narrative Activity Report October 2016 - March 2017. NCBA CLUSA. (2017, October). Semi-Annual Narrative Activity Report April 1, 2017 - September 31, 2017. NCBA CLUSA. (2018, October). Semi-Annual Narrative Report April 1 - September 31, 2018. NCBA CLUSA. (24 November 2015). Millet Business Services Project Presentation. NCBA CLUSA. (n.d.). Terms of References MBSP Final Evaluation. NCBA CLUSA. (n.d.). USDA Millet Services . (2019, December 11). OCP Sales Representative. (I. E. Team, Interviewer) (2019, December 17). President, Khelcom Birane Hub. (2019, December 15). Processoor. Ragasa, C., Sengupta, D., Osorio, M., & al., e. (2014). Gender-specific Approaches, Rural Institutions, and Technological Innovations. Rome: Food and Agriculture ORganization of the United Nations. Ross, C. G. (1982). A Village Level Study of Producer Grain Transactions in Rural Senegal. African Studies Review, 25(4), 65-84. Ross, C. G. (1982). A Village Level Study of Producer Grain Transactions in Rural Senegal. African Studies Review, 25(4). 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 86 Soterco Group. (2016). Baseline Study Report; Millet Business Services Project. Dakar: USDA. (n.d.). Terms of Reference; Final Evaluation; Millet Business Services Project (MBSP). NCBA CLUSA. Waddington, S. R. (2010). Getting the focus right: production constraints for six major food crops in Asian and African farming systems. Food Secure, 2, 27-48. World Food Program. (n.d.). Senegal Strategic Plan (2019 to 2023). Accessible at https://docs.wfp.org/api/documents/5b0e7061163e4ba98d6348b150e588e2/download/?_ga=2.19610 0716.1561643913.1581355362-924581309.1578949931. World Food Programme. (2018). Senegal country strategic plan (2019–2023). Dakar: World Food Programme. Yadav, D. A. (2012). Improved Storage Stability of Pearl Millet Flour Through Microwave Treatment. Agric Res, 1. Documents Reviewed Workplans Draft Workplan USDA Senegal Millet Business Services Program – 2015 New WBS 2019 Plan de Travail, Années fiscale ; Octobre 2016 à Septembre 2017 Session De Revue Des Activites, D’apprentissage Et De Planification; 07-10 Juin, 2017 Rapport De La Retraite Somone, 7 – 9 Juin 2017 ScopeInsight Assessments Aproval, 2017-09-14.SCOPE Basic SME Report Final Bokk Dole, 2017-09-13.SCOPBasic Report Final Centrale d'achat de ndiédieng.2017-09-07.SCOPE Basic Report Final COPAN.2017-09-15.SCOPE Basic Report Final 128 NCBA CLUSA Senegal overview 230218 Entreprise BAKH YAYE.2017-12-03.SCOPE Basic SME Report Final Entreprise de Transformation de Thioffack.2017-10-31.SCOPE Basic SME Report Final Fanx Taxalu Xaleyi de Kathiotte.2017-09-17.SCOPE Basic Report Final FARK (Femmes Artisans Rurales de Kaffrine).2017-09-12.SCOPE Basic SME Report Final GIE Takku Ligudye de Gagu6 Ch6rif.2017-12-18.SCOPE Basic Report Final GIE Xaritu Xaleyi.2017-09-20.SCOPE Basic Report.Final Groupe GIPA.2017-11-27.SCOPE Basic Report Final Groupement des Producteurs de Godaguène Sorokh.2017-11-02.SCOPE Basic Report Final Groupement des Producteurs de Keur Sérigne Bamba.2017-10-17.SCOPE Basic Report Final 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 87 Groupement Dialko Ndjirigne de Niakhar.2017-11-01.SCOPE Basic Report Final Keur Yaye Aby.2017-09-06.SCOPE Basic SME Report.Final.pdf Roseau des producteurs CF.2017-11-27.SCOPE Basic Report Final RESEAU DOOR WAAR.2017-09-16.SCOPE Basic Report Final Reseau mil de Mabo.2017-09-16.SCOPE Basic Report Final RESEAU PASTEF.2017-09-13.SCOPE Basic Report Final Takku Céréales de Yeumbeul.2017-09-15.SCOPE Basic Report Final USDA Presentation.Scopelnsight UTRACEL.2017-09-14.SCOPE Basic SME Report Final Assessments, Reports, and Studies Note D’orientation Pour Le Deroulement Du Business Forum Des Acteurs De La Chaine De Valeur Du Mil; Cœur de Ville de Kaolack les 28, 29 et 30 janvier 2017 Note orientation generale Business forum USDA PSEM Mai 2017-1 CLUSA-PSEM_Rapport d‘évaluation des bons d'achat 18/12/18 Fora Summary 2016 Rapport de I’atelier de concertation des investisseurs du mil FINAL Rapport De La Fiara 2018 Final Rapport des Fora Régionaux sur les metiers du Mil USDA PASEM 29/12/16 Rapport Du Business Forum De Touba Du 11 Au 13 Juillet Rapport final CAM USDA PSEM ECVM Rapport Foire De Foundiougne Rapport Rencontres Acteurs Du Mil Version Finale Touba Business Forum from 11 to 13 July 2017 MBSP Baseline Study MBSP Mid-Term Evaluation Performance Reports Performance Report_Apr2016_Sep2016 Performance Report_Oct2016_Mar2017 Performance Report_Apr2017_Sep2017 Performance Report_Oct2017_Mar2018 View Performance Report November 2018 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 88 View Performance Report Oct 2018 March 2019 Performance Report submitted Apr-Sept 2019 FY17 – FINAL Narrative reportin 11.14.17 FY17 – Narrative Reporting October 2016 to March 2017 FY18 - Reporting Period April 2018 to Sept 2018 FY18 - USDA Narrative Report Oct 2017 to Mar 2018 FINAL FY19 - Reporting Period October 2018 to March 2019 FY19 – Reporting Narrative April to Sept 2019.pdf Final USDA Financial Report Grant Agreements Senegal NCBA Agreement FCC-685-2014-055-00 Modifications 1-4 Administrative documents Grant Trackers Local Partner Agreements for ANCAR, DRDR, IREF, and ITA Monitoring Documents Activity Manuals Activity and Event Reports Program Results Framework 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 89 Annex C: Data Collection Tools Buyer and Seller Questionnaire Name Question Options Instruction A1_interviewer_name Please select the interviewer name from the list below initial_consent_note Hello. I work for the Centre de Recherche pour le Développement Humain (CRDH). We are carrying out a survey about the production of Millet for CLUSA an organisation which you likely know. I would like to ask you a few questions. Your answers will be confidential. They will be put together with the responses of other people we are talking to, to get an overall picture of the Millet Value Chain. It will be impossible to pick you out from what you say, so please feel free to tell us what you think. This interview will take about 30 minutes. Could you please spare a few minutes of your time for this interview? initial_consent Will you participate in this study? Yes No consent Interviewer: Please Make Sure The respondent Has Consented Before Proceeding With Interview. If Consent Is Not Given, Thank respondent And Close Interview. GPS Please capture GPS coordinates Make sure you are outside in view of the open sky, but near the place where the interview took place. Then take the GPS location. province Please select the respondent’s region Dakar Fatick Kaffrine Kaolack 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 90 Name Question Options Instruction commune Please select the respondent’s commune [Full list of 81 commune's available from NCBA CLUSA] village What is the respondent's village? business_type Is this a producer organization, hub, processor or seller? Producer organization Hub Processor unit Womens processing groups Kiosk Restaurant/hotel Other seller (specify) business_name1 Business name [Full list of business names available from NCBA CLUSA] Position Is the respondent the president, owner or manager? Yes No Position_other What is the respondent’s position in the organization? Gender Select the respondent's gender Male Select based on observation Female Module A Background A_01_No_members A_01 How many members does your organization currently have? A_02_eligible A_02 Does your business or organization sell millet or millet related products Yes No 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 91 Name Question Options Instruction A_03_No_suppliers A_03 In the past season (2018/2019) many suppliers did your organization /business buy from? A_04_No_buyers A_04 In the past season (2018/2019), how many buyers/ customers did your organization/business sell to? Module_B Sales B_01_products B_01 Which of the following millet products did you sell this past season (2018/2019)? Thiakry Thiéré Sanxal Araw Ngourbane Farine fortifiée Farine simple other (specify) B_02a_past_quant1 B_02a1 Approximately how much Thiakry did you sell 5 years ago (2013/2014)? (KG) 99 – Don’t know B_02a_past_quant2 B_02a2 Approximately how much Thiéré did you sell 5 years ago (2013/2014)? (KG) 100 – Don’t know B_02a_past_quant3 B_02a3 Approximately how much Sanxal did you sell 5 years ago (2013/2014)? (KG) 101 – Don’t know B_02a_past_quant4 B_02a4 Approximately how much Araw did you sell 5 years ago (2013/2014)? (KG) 102 – Don’t know B_02a_past_quant5 B_02a5 Approximately how much Ngourbane did 103 – Don’t know 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 92 Name Question Options Instruction you sell 5 years ago (2013/2014)? (KG) B_02a_past_quant6 B_02a6 Approximately how much Farine fortifiée did you sell 5 years ago (2013/2014)? (KG) 104 – Don’t know B_02a_past_quant7 B_02a7 Approximately how much Farine simple did you sell 5 years ago (2013/2014)? (KG) 105 – Don’t know B_02a_past_quant8 B_02a8 Approximately how much ${B_01_products_other } did you sell 5 years ago (2013/2014)? (KG) 106 – Don’t know B_03_sales_rev1 B_03_1 How much revenue did you make from Thiakry in the past season (2018/2019) (CFA) This is gross revenue B_03_sales_rev2 B_03_2 How much revenue did you make from Thiéré in the past season (2018/2019) (CFA) This is gross revenue B_03_sales_rev3 B_03_3 How much revenue did you make from Sanxal in the past season (2018/2019) (CFA) This is gross revenue B_03_sales_rev4 B_03_4 How much revenue did you make from Araw in the past season (2018/2019) (CFA) This is gross revenue B_03_sales_rev5 B_03_5 How much revenue did you make from Ngourbane in the past season (2018/2019) (CFA) This is gross revenue 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 93 Name Question Options Instruction B_03_sales_rev6 B_03_6 How much revenue did you make from Farine fortifiée in the past season (2018/2019) (CFA) This is gross revenue B_03_sales_rev7 B_03_7 How much revenue did you make from Farine simple in the past season (2018/2019) (CFA) This is gross revenue B_03_sales_rev8 B_03_8 How much revenue did you make from ${B_01_products_other } in the past season (2018/2019) (CFA) This is gross revenue B_03a_past_rev1 B_03a1 And how much revenue did you make from Thiakry sales 5 years ago (2013/2014)? (CFA) This is gross revenue 99 – Don’t know B_03a_past_rev2 B_03a2 And how much revenue did you make from Thiéré sales 5 years ago (2013/2014)? (CFA) This is gross revenue 99 – Don’t know B_03a_past_rev3 B_03a3 And how much revenue did you make from Sanxal sales 5 years ago (2013/2014)? (CFA) This is gross revenue 99 – Don’t know B_03a_past_rev4 B_03a4 And how much revenue did you make from Araw sales 5 years ago (2013/2014)? (CFA) This is gross revenue 99 – Don’t know B_03a_past_rev5 B_03a5 And how much revenue did you make from Ngourbane sales 5 years ago (2013/2014)? (CFA) This is gross revenue 99 – Don’t know B_03a_past_rev6 B_03a6 And how much revenue did you make from Farine fortifiée This is gross revenue 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 94 Name Question Options Instruction sales 5 years ago (2013/2014)? (CFA) 99 – Don’t know B_03a_past_rev7 B_03a7 And how much revenue did you make from Farine simple sales 5 years ago (2013/2014)? (CFA) This is gross revenue 99 – Don’t know B_03a_past_rev8 B_03a8 And how much revenue did you make from ${B_01_products_other } sales 5 years ago (2013/2014)? (CFA) This is gross revenue 99 – Don’t know B_04_total_rev B_04 Approximately what percent of the business or organization’s total income was from millet in the past season (2018/2019)? 99 – Don’t know B_04a_pasttotal_rev B_04a Approximately what percent of your business or organization’s total income was from millet 5 years ago (2013/2014)? 99 – Don’t know B_05_per_customer B_05 Approximately what percentage of the millet your organization sold in the past season (2018/2019) goes to the following markets? Enter Percentage for each market. Total = 100% local Local (in nearby towns and villages) % regional Regional (within the same region) % national National (outside your region) % export Export % 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 95 Name Question Options Instruction B_06_sales_increase B_06 You said you had more revenue for millet this past season than 5 years ago. Which of the following options best describes why your organization’s income from millet increased? Higher value processed products The quality of our millet is higher We have better customers Our certifications have improved sales The market is bigger More millet supply is available Other B_07_supply_quality_yn B_07 In the past five years, have you seen any improvements in the millet you purchase? Yes No B_08_supply_quality B_08 In the past five years, which improvements have you seen to the millet you purchase? Less debris (e.g. dirty, bad grains, etc) Multiple choice Better color Bigger size Better texture Better taste Other (Specify) B_09_own_quality B_09 What was the main improvement you have made in your millet quality the past five years? B_09_own_quality2 B_09_2 Did you learn this from the CLUSA project? Yes No 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 96 Name Question Options Instruction Module_C Activities C_01_activities1 C_01 Which of the following CLUSA activities was the most useful for improving your organization or business? Training producers Training extension agents Improving access to loans Business planning and marketing training Improving access to seeds Providing post-harvest handling and processing training Providing grants for post￾harvest equipment Threshing training Providing linkages to suppliers Providing linkages to customers C_02_activities2 C_02 Which of the following CLUSA activities was the second most useful for improving your organization or business? Training producers Training extension agents Improving access to loans Business planning and 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 97 Name Question Options Instruction marketing training Improving access to seeds Providing post-harvest handling and processing training Providing grants for post￾harvest equipment Threshing training Providing linkages to suppliers Providing linkages to customers C_03_training C_03 Have you ever provided your members any technical trainings on millet production and post-harvest handling? Yes No C_04_training_type C_04 Which trainings did you provide Personal Hygiene Work area cleanliness Equipment maintenance Quality standards Other Module_D Linkages D_01_events D_01 Have you attended any events hosted by CLUSA to introduce you to buyers, processors or suppliers? Yes No 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 98 Name Question Options Instruction D_02_done_business D_02 Have you done any business with any of these buyers, processors or suppliers you met at these events? Yes No D_03_still_doing D_03 Are you still doing business with them? Yes No D_04_other_links D_04 Has the CLUSA introduced you to buyers, processors, or suppliers through means other than events? Yes No Module_E Finance E_01_access_finance E_01 In the past season (2018/2019), did you access finance? Yes No E_01a_access_finance_wher e E_01a From where did you access finance? Bank Microfinance Buyer Other (Specify) E02_provide_finance E02 In the past season (2018/2019), did you provide finance to your suppliers? Yes No end_note That is the end of the survey. Thank you for your time 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 99 Producer Questionnaire Name Question Options Instructio n interviewer_name Please select the interviewer name from the list below initial_consent_note Hello. I am working for Centre de Recherche pour le Développement Humain (CRDH). We are carrying out a survey about the production of Millet for CLUSA an Organization which you likely know. I would like to ask you a few questions. Your answers will be confidential. They will be put together with the responses of other people we are talking to, to get an overall picture of the Millet Value Chain. It will be impossible to pick you out from what you say, so please feel free to tell us what you think. This interview will take about 30 minutes. Could you please spare a few minutes of your time for this interview? initial_consent Will you participate in this study? Yes No consent Interviewer: Please Make Sure The respondent Has Consented Before Proceeding With Interview. If Consent Is Not Given, Thank respondent And Close Interview. GPS Please capture GPS coordinates Make sure you are outside with the open sky, but near the place where the interview took place. Then take the GPS location. 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 100 Name Question Options Instructio n province Please select the respondent’s province Fatick Kaffrine Kaolack commune Please select the respondent’s commune village Please select the respondent’s village eligible A1.1. Does your household grow millet? Yes If no, end interview No not_eligibe A1.2. Respondent is not eligible for interview. Please thank the respondent and close interview direct_indirect A.02. Is the respondent an MBSP participant or indirect beneficiary farmer MBSP Participant Indirect beneficiary farmer PO Please select he respondent's producer organization ModuleA CONSENT AND IDENTIFICATION A_01_Gender A1. Select the respondent's gender Male Select based on observation Female A_02_name A_03 Name of Respondent A_02a_tel A_03a Respondent telephone number Skip if the respondent is unwilling to provide A_03_literacy A_04 Which of the following best describes your level of literacy? Cannot read or write Can sign only Can read only Can read and write A_04_HOH Yes 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 101 Name Question Options Instructio n A_05 Are you the head of your household? No A_05_group_member A_06 Are you a member of a producer organization? Yes No A_06_member_time A_07 For how many years have you belonged to the producer organization? Module_B Cultivation B_01_total_land B_01_total_land How many hectares in total did your household cultivate for all crops this season (2019)? B_02_millet_land B_02 How many hectares were dedicated to cultivating millet last season (2019)? B_03_millet_past B_03 How many hectares were dedicated to cultivating millet five years ago (2014)? B_04_why_more B_04 Why did you cultivate more millet now than five years ago? The price of millet is higher now than five years ago The market for millet is more stable than five years ago Millet yields are higher Millet is important for food security Other crops are not doing as well as millet I have acquired more land I can produce higher quality millet than I could five years ago Other, specify B_05_why_less B_05_why_less Why did you plant less millet now than five years ago (2014)? The price of millet is lower now than five years ago The market for millet is less stable than five years ago 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 102 Name Question Options Instructio n Millet yields are lower I/my family doesn’t like the taste of millet Other crops are doing better than millet I have less land Other, specify B_06_inputs B_06 This past season, which of the following inputs did you purchase or rent? Improved seeds Fertilizer Pesticides Equipment Hired labor None B_07_finance1 B_07.1 How did you finance these purchases - Improved seeds Personal savings The seller gave me credit A Savings and Internal Lending Community (SILC) loan Microfinance institution loan A bank loan (non-microfinance) A loan from Friends and/0r family An NGO gave the input to me for free A family member gave the input to me for free Other, specify B_07_finance2 B_07.2 How did you finance these purchases - Fertilizer Personal savings The seller gave me credit A Savings and Internal Lending Community (SILC) loan 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 103 Name Question Options Instructio n Microfinance institution loan A bank loan (non-microfinance) A loan from Friends and/0r family An NGO gave the input to me for free A family member gave the input to me for free Other, specify B_07_finance3 B_07.3 How did you finance these purchases - Pesticides Personal savings The seller gave me credit A Savings and Internal Lending Community (SILC) loan Microfinance institution loan A bank loan (non-microfinance) A loan from Friends and/0r family An NGO gave the input to me for free A family member gave the input to me for free Other, specify B_07_finance4 B_07.4 How did you finance these purchases - Equipment Personal savings The seller gave me credit A Savings and Internal Lending Community (SILC) loan Microfinance institution loan A bank loan (non-microfinance) A loan from Friends and/0r family An NGO gave the input to me for free 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 104 Name Question Options Instructio n A family member gave the input to me for free Other, specify B_07_finance5 B_07.5 How did you finance these purchases - Hired labor Personal savings The seller gave me credit A Savings and Internal Lending Community (SILC) loan Microfinance institution loan A bank loan (non-microfinance) A loan from Friends and/0r family An NGO gave the input to me for free A family member gave the input to me for free Other, specify B_08_seedtype B_08 What type of seed did you use for the majority of the millet you harvested Souna 3 Thialack 2 ICMV IS 89305 Gawane Other, specify Unknown Module_C Harvest and Post Harvest C_01_kilos C_01 How many kilos of millet did you harvest this harvest season (2019)? C_02_loss C_02 Did you lose any of your millet before the harvest this past season? Yes No C_03_loss C_03 What percentage do you estimate you lost? C_04_yield C_04 Has your millet harvest per hectare changed since 2014? Yes, it has increased 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 105 Name Question Options Instructio n No, I produce the same amount as in 2014 Yes, I produce less than in 2014 C_05_amt_increase C_05 How much would you estimate that your millet harvest per hectare has increased since 2015? C_06_why_increase C_06 Which of the following reasons do you believe describe why your millet harvest has increased? I am better at preparing the soil. select as many as apply I have better access to seeds The seeds are higher quality I’m better at sowing seeds properly I am better at monitoring my crops I’m better at maintaining my crops I store the millet better I have better storage facilities I am better at managing water I am better at managing pests I have better access to pesticides Better harvesting practices I don’t know C_07_why_decrease C_07 Which of the following reasons do you believe describe why your millet harvest has decreased? Worse access to water select as many as apply Rain or floods at the wrong time Worse land preparation and sowing techniques Worse seeds Poor access to fertilizer 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 106 Name Question Options Instructio n Problems with the soil Problems with pests Problems with harvesting Problems with storage I don’t know C_08_harvest C_08 Did you lose any millet after harvest and before sale or consumption? Yes No C_09_harvestper C_09 What percent did you lose after harvest and before sale or consumption? C_10_harvest_change C_10 Have your post- harvest losses increased, decreased or stayed the same since 2014? Increased Decreased Stayed the same C_11_increased C_11 Why have your post￾harvest losses increased? Worse threshing Worse cleaning Worse drying Worse storage facility Worse storage practices C_12_decreased C_12 Why have your post￾harvest losses decreased? Improved threshing Improved cleaning Improved drying Improved storage facility Improved storage practices C_13_quality C_13 Has the quality of your millet improved in the last 5 years? Yes No C_14_quality_how Better variety 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 107 Name Question Options Instructio n C_14 How has the quality of your millet improved in the last 5 years? Bigger grains Better color Better taste More consistent size and quality Fewer bad grains Other (specify) C_14_quality_how_oth er C_14 How has the quality of your millet improved in the last 5 years? Other, Specify here Module_D Sales D_01_current_sales D_01 How many KGs of millet have you sold so far from the most recent harvest (2019)? D_02_current_revenue D_02 How much money have you made from these sales? (CFA) D_03_future_sales D_03 How much more millet do you plan to sell from this last harvest (2019)? (KGs) D_04_future_revenue D_04 How much more money do you expect to make from these future millet sales this season (2019)? (CFA) D_05_past_revenue D.05 How much money did you make from millet sales last season (2018)? (CFA) Customers_matrix D_06 Customers_matrix D_06_customers D_06 What percent of your sales last season went to the following buyers (2018)? Neighbours Neighbors Local_markets Local markets Local_trader Local trader Producer_organization Producer Organization Processor Processor 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 108 Name Question Options Instructio n other_buyers Other buyers other_customers_des Other Buyers, specify here Module_E Trainings E_01_training E_01 Have you received training in millet cultivation since 2014? Yes No E_02_training_type E_02 Approximately how many of the following trainings have you received in the last five years 99 - I don’t remember Seed_quality_identifica tion Seed quality identification Seed_sowing_and_ma nagement Seed sowing and management Soil_Management Soil Management Crop_monitoring_and_ management Crop monitoring and management Post-harvest_handing Post-harvest handing Business_planning_or_ marketing Business planning or marketing Other_training Other training E_03_training_prod1 E_03 Which training topic did you find to be the most useful for improving the performance of your millet production? Seed quality identification Seed sowing and management Soil Management Crop monitoring and management Post-harvest handing Business planning or marketing Other I don’t remember E_03_training_prod2 E_03.2 Which training topic did you find to be the second most Seed quality identification Seed sowing and management 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 109 Name Question Options Instructio n useful for improving the performance of your millet production? Soil Management Crop monitoring and management Post-harvest handing Business planning or marketing Other I don’t remember E_04_trainers E_04 Approximately how many trainings have you received from each of the following sources in the last 5 years DRDR DRDR Producer_Organization Producer Organization Millet_hub Millet hub Seed_provider Seed provider Fertilizer_provider Fertilizer provider Equipment_provider Equipment provider Other_training_source Other Training Source Other_training_source _des Describe the other training source That is the end of the survey. Thank you for your time. Interview Guides Producer question guide – CLUSA Instructions: Please ask all questions to all respondents. Note responses and indicate the question by number. Note Organization name, location and date 1. How long have you been producing millet? How long has the community been producing millet? 2. How long have you been part of this producer organization? 2.1. Why did you join the producer organization? 2.1.1.Have your expectations been met? 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 110 3. How much millet did you plant last season (hectares)? 3.1. Is the area of millet you grow now more or less than you grew before joining the CLUSA project? Why? 4. How much millet did your harvest this season? 4.1. Have your yields per hectare changed since you joined CLUSA? Why? 5. Have your post-harvest practices changed since you joined CLUSA? How? 5.1. What are the improvements, if any? 6. What about the quality of millet? Has it changed since you joined CLUSA? Why? 7. Where did you sell most of your millet to this season? 7.1. Is it the same buyer as before CLUSA? 7.1.1.If not, why did you change buyers? 8. What trainings have you received through CLUSA? 8.1. Were the trainings provided by your PO, Pole/Hub, the government or someone else? 8.2. Which trainings were most useful? Why? 9. What was the best part of the CLUSA project? Why? 10. What was the weakest part? Why 10.1. What could it have done differently? 11. What’s the most challenging part about being a millet farmer now? Producer organization presidents and pole presidents – CLUSA Instructions: Please ask all questions to all respondents. Note responses and indicate the question by number. Note Organization name, location and date 1. Note whether this is a 1) producer organization president or 2) pole president 2. How old is this producer organization/pole? How long have you been president? 3. What services do you offer to your members? (e.g. financial services, marketing, training) 4. How much millet have you purchased this season? Is it all from members? 4.1. Is there a change in the volume of millet you purchase in recent years? Why? 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 111 4.2. Do you have any challenges in buying enough millet to meet your buyers demands? 5. Has there been a change in the quality of millet you purchase in recent years? Why? 6. Do you do any cleaning or sorting of the millet you purchase? Explain. 6.1. Where did you learn to do these activities? 6.2. How did you get the equipment? 6.3. Is the equipment working well? 6.4. Do these activities result in higher prices? 7. Who are your main buyers? Have you had the same buyers in recent years or new ones? 7.1. Has CLUSA helped you find new buyers? Are you still doing business with them? 8. Do you have contracts with your buyers? 8.1. If yes, are you happy with them? 8.2. If no, do you wish you had contracts? 9. Do you provide finance to members? When did you start doing this? Explain how it works. 10. Which trainings have you received through CLUSA? 10.1. Which were most useful? Why? 11. Do you provide trainings to your members based on what you learned from CLUSA? 11.1. Which do you think are most useful for members? Why? 12. What was the best part of the CLUSA project? Why? 13. What was the weakest part? Why? 13.1. What could it have done differently? 14. What’s the biggest challenge for the producer organization/pole? 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 112 Annex D: Results Table Result Indicator Baseline Final Total Target Achieved Difference Increased Agricultural Productivity Volume of commodities sold by project beneficiaries (MT) 5,334.00 20,951 22,741 1,790 Number of individuals receiving short￾term agricultural sector productivity or food security training as a result of USDA assistance (Continuing) 3,851.00 12,669 10,486 (2,183) Number of individuals receiving short￾term agricultural sector productivity or food security training as a result of USDA assistance ( people in civil society) 0 835 2333 1,498 Number of individuals receiving short￾term agricultural sector productivity or food security training as a result of USDA assistance (people in firms) 323 6,206 5,806 (400) Number of individuals receiving short￾term agricultural sector productivity or food security training as a result of USDA assistance (people in government) 0 1,000 143 (857) Number of individuals receiving short￾term agricultural sector productivity or food security training as a result of USDA assistance (producers) 3,582.00 28,666 35,601 6,935 Number of individuals receiving short￾term agricultural sector productivity or food security training as a result of USDA assistance (new) 0 23,276 30,201 6,925 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 114 Number of individuals who have received short- tenn agricultural sector productivity or food security training as a result of USDA assistance (female) 1,187.00 18,486 20,211 1,725 Number of individuals who have received short-term agricultural sector productivity or food security training as a result of USDA assistance (male) 2,664.00 17,459 22,187 4,728 Value of sales by project beneficiaries 1,443,395.00 11,901,561 12,699,073 797,512 Increased Capacity of Government Institutions Number of millet producers who have been trained by government extension agents 1,470.00 12,385 17,513 5,128 Improved Capacity of Key Groups in the Agriculture Production Sector (Coops and Small Shareholder Farmers) Number of producer organizations/cooperatives with certified extension officcrs/agents 221 199 99 (100) Increased Use of Improved Agricultural Techniques and Technologies Number of hectares under improved techniques or technologies as a result of USDA assistance (new) 9,012.00 24,429 15,393 (9,036) Number of hectares under improved techniques or technologies as a result of USDA assistance (Climate mitigation or adaptation) 6,069.00 9,531 10,744 1,213 Number of hectares under improved techniques or technologies as a result of USDA assistance (continuing) 9,012.00 18,050 14,568 (3,482) Number of hectares under improved techniques or technologies as a result of USDA assistance (Crop genetics) 155 9,610 10,930 1,320 Number of hectares under improved techniques or technologies as a result of USDA assistance (Disease management) 211 6,921 7,088 167 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 115 Number of hectares under improved techniques or technologies as a result of USDA assistance (Other) 15 9,802 1,052 (8,750) Number of hectares under improved techniques or technologies as a result of USDA assistance (Pest management) 809 9,771 10,469 698 Number of hectares under improved techniques or technologies as a result of USDA assistance (Soil- related fertility and conservation) 2,643.00 34,948 15,484 (19,464) Number of hectares under improved techniques or technologies as a result of USDA assistance (Total w/one or more improved techniques or technologies) 9,012.00 43,488 29,962 (13,526) Number of hectares under improved techniques or technologies as a result of USDA assistance (Water management (non-irrigation based) 2,170.00 8,919 5,300 (3,619) Number of individuals who have applied improved farm management practices (i.c. governance, administration, or financial management) as a result of USDA assistance (business) 0 9,307 4,587 (4,720) Number of individuals who have applied improved farm management practices (i.e. governance, administration, or financial management) as a result of USDA assistance (Farmers/Producers) 0 11,634 9,686 (1,948) 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 116 Number of individuals who have applied improved farm management practices (i.e. governance, administration, or financial management) as a result of USDA assistance (Female) 0 11,634 8,970 (2,664) Number of individuals who have applied improved farm management practices (i.c. governance, administration, or financial management) as a result of USDA assistance (government agents) 0 2,326 77 (2,249) Number of individuals who have applied improved farm management practices (i.e. governance, administration, or financial management) as a result of USDA assistance (Male) 0 11,634 5,380 (6,254) Number of individuals who have applied new techniques or technologies as a result of USDA assistance (Climate mitigation or adaptation) 0 2,532 5,700 3,168 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (continuing) 2,921.00 5,000 5,505 505 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (Crop genetics) 0 2,994 8,217 5,223 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (Disease management) 0 2,205 3,711 1,506 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 117 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (female) 492 7,200 4,582 (2,618) Number of individuals who have applied new techniques or technologies as a result of USDA assistance (male) 2,429.00 8,128 11,567 3,439 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (new) 0 10,266 10,644 378 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (Other) 0 4,444 2,944 (1,500) Number of individuals who have applied new techniques or technologies as a result of USDA assistance (Pest management) 0 2,205 5,258 3,053 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (Soil￾related fertility and conservation) 0 7,078 10,035 2,957 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (Total w/one or more improved techniques or technologies) 0 15,328 16,149 821 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (Water management (nonirrigation based)) 0 1,937 7,057 5,120 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 118 Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community￾based organizations (CBOs) that applied new technologies or management practices as a result of USDA assistance (continuing) 223 306 233 (73) Number of private enterprises, producers organizations, water users associations, women’s groups, trade and business associations, and community-based organizations (CBOs) that applied new technologies or management practices as a result of USDA assistance (new) 0 411 470 59 Number of private enterprises, producers organizations, water users associations, women’s groups, trade and business associations, and community-based organizations (CBOs) that applied new technologies or management practices as a result of USDA assistance (POs) 179 458 427 (31) Number of private enterprises, producers organizations, water users associations, women’s groups, trade and business associations, and community-based organizations (CBOs) that applied new technologies or management practices as a result of USDA assistance (private enterprises) 23 205 238 33 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 119 Number of private enterprises, producers organizations, water users associations, women’s groups, trade and business associations, and community-based organizations (CBOs) that applied new technologies or management practices as a result of USDA assistance (trade and business associations) 13 20 8 (12) Number of private enterprises, producers organizations, water users associations, women’s groups, trade and business associations, and community-based organizations (CBOs) that applied new technologies or management practices as a result of USDA assistance (women's groups) 8 60 30 (30) Increased Availability of Improved Inputs Value of inputs and support services sold by agricultural input suppliers to farmers in US dollars 2,310,159.00 3,000,000 1,757,024 (1,242,976) Increased Use of Financial Services Number of individuals receiving Financial services as a result USDA assistance (female) 86 19,981 19,567 (414) Number of individuals receiving Financial services as a result USDA assistance (male) 4,392.00 20,193 20,671 478 Number of loans disbursed as a result of USDA assistance. 242 13,129 10,256 (2,873) Value of loans provided as a result of USDA assistance (female) 0 1,292,129 990,405 (301,724) Value of loans provided as a result of USDA assistance (joint) 150 150 Value of loans provided as a result of USDA assistance (male) 0 1,734,218 1,824,789 90,571 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 120 Value of loans provided as a result of USDA assistance (N/A) 0 521,824 521,824 0 Expanded Trade of Agricultural Products (Domestic, Regional and International) Total number of individuals benefiting directly as a result of USDA assistance (continuing) 9,065.00 27,139 167,747 140,608 Total number of individuals benefiting directly as a result of USDA assistance (female) 4,098.00 31,579 36,710 5,131 Total number of individuals benefiting directly as a result of USDA assistance (male) 4,967.00 37,797 45,691 7,894 Total number of individuals benefiting directly as a result of USDA assistance (new) 0 36,287 45,052 8,765 Total number of individuals benefiting indirectly as a result of USDA assistance 33,618.00 437,277 282,768 (154,509) Improved Capacity of Key Organizations in the Trade Sector (Processing Organizations and Trade Associations) Number of producer organizations/groups with marketing specialists 34 110 108 (2) Increased Value Added to Post Production Agricultural Products Number of jobs attributed to USDA assistance (female) 91 364 342 (22) Number of jobs attributed to USDA assistance (male) 139 350 203 (147) Improved Post-Harvest Infrastructure Number of agriculture producers in target region that have access to effective off-farm storage for their product 3,422.00 12,404 10,894 (1,510) Total increase in installed storage capacity as a result of USDA assistance (new) 0 9,286 9,786 500 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 121 Total increase in installed storage capacity as a result of USDA assistance (refurbished) 0 7,696 6,305 (1,391) Increased Access to Markets to Sell Agricultural Products Number of market outlets in which selected agricultural products is sold 93 599 783 184 Improved Linkages Between Buyers and Sellers Number of agreements signed (contracts, MOUs etc.) between buyers/ sellers 31 304 281 (23) Performance Indicators Activity Indicator LoP Target Achieved Capacity Building: Agricultural Extension Agent Services, Producers, and the Private Sector Number of demonstration plots created 360 427 Number of input demonstration fairs 12 12 Number of input suppliers trained in extension skills by input firms 310 199 Number of Producer Organization (PO) members trained in extension skills 360 567 Financial Services: Facilitate Agricultural Lending Number of individuals receiving financial services as a result of USDA assistance (female) 5,300 17,981 Number of individuals receiving financial services as a result of USDA assistance (male) 2,500 15,193 Number of loans disbursed as a result of USDA assistance 800 10,129 Value of agricultural and rural grants provided as a result of USDA assistance (USD) 834,500 931,609 Value of loans provided as a result of USDA assistance (female) 1,392,129 921,511 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 122 Value of loans provided as a result of USDA assistance (joint) 969,096 0 Value of loans provided as a result of USDA assistance (male) 1,246,063 1,134,219 Value of loans provided as a result of USDA assistance N/A) 323,038 0 Infrastructure: Postharvest Handling and Storage Total increase in installed storage capacity as a result of USDA assistance (new) 4,000 6,976 Total increase in installed storage capacity as a result of USDA assistance (Refurbished) 8,500 6,696 Infrastructure: Post￾Harvest Processing Number of agricultural and rural grants provided as a result of USDA assistance (USD)- ONLY Post harvest processing infrastructure 324 131 Inputs: Develop Agrodcalcrs and Input Suppliers Number of agro dealers developed into networks 250 221 Market Access: Facilitate Buyer-Seller Relationships Number of jobs attributed to USDA assistance (female) 310 337 Number of jobs attributed to USDA assistance (male) 425 191 Volume of commodities (metric tons) sold by project beneficiaries 21,454 19,596 Training: Improved Agricultural Production Techniques Number of hectares under improved techniques or technologies as a result of USDA assistance (Climate mitigation or adaptation) 921 1,191 Number of hectares under improved techniques or technologies as a result of USDA assistance (continuing) 23,387 2,121 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 123 Number of hectares under improved techniques or technologies as a result of USDA assistance (Crop genetics) 3,359 3,065 Number of hectares under improved techniques or technologies as a result of USDA assistance (Disease management) 7,980 1,457 Number of hectares under improved techniques or technologies as a result of USDA assistance (new) 27,896 12,537 Number of hectares under improved techniques or technologies as a result of USDA assistance (Other) 2,035 1,052 Number of hectares under improved techniques or technologies as a result of USDA assistance (Pest management) 11,147 1,745 Number of hectares under improved techniques or technologies as a result of USDA assistance (Soil- related fertility and conservation) 32,105 3,980 Number of hectares under improved techniques or technologies as a result of USDA assistance (Total w/onc or more improved techniques or technologies) 62,458 14,658 Number of hectares under improved techniques or technologies as a result of USDA assistance (Water management (non-irrigation based) 4,911 3,628 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 124 Number of individuals receiving short- term agricultural sector productivity or food security training as a result of USDA assistance ( People in firms-NOT Processors) 3,200 2,385 Number of individuals receiving short- term agricultural sector productivity or food security training as a result of USDA assistance (female) 17,700 20,130 Number of individuals receiving short- term agricultural sector productivity or food security training as a result of USDA assistance (male) 18,500 21,933 Number of individuals receiving short- term agricultural sector productivity or food security training as a result of USDA assistance (New) 18,250 30,171 Number of individuals receiving short- term agricultural sector productivity or food security training as a result of USDA assistance (People in civil society) 600 848 Number of individuals receiving short- term agricultural sector productivity or food security training as a result of USDA assistance (Producers) 30,000 35,375 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 125 Number of individuals who have applied improved farm management practices (i.e. governance, administration, or financial management) as a result of USDA assistance (business- NOT processors) 9,007 1,944 Number of individuals who have applied improved farm management practices (i.e. governance, administration, or financial management) as a result of USDA assistance (Farmers/Produccrs) 12,635 9,596 Number of individuals who have applied improved farm management practices (i.e. governance, administration, or financial management) as a result of USDA assistance (Female) 12,134 8,270 Number of individuals who have applied improved farm management practices (i.e. governance, administration, or financial management) as a result of USDA assistance (government agents) 2,351 77 Number of individuals who have applied improved farm management practices (i.e. governance, administration, or financial management) as a result of USDA assistance (Male) 13,084 5,260 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 126 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (Climate mitigation or adaptation) 745 1,598 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (continuing) 8,500 1,747 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (Crop genetics) 1,300 4,407 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (Disease management) 1,358 1,606 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (female) 5,500 3,734 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (male) 9,500 6,972 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (new) 6,500 8,959 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (Other) 100 2,944 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 127 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (Pest management) 658 1,606 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (Soil-related fertility and conservation) 7,986 4,919 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (Total w/one or more improved techniques or technologies) 15,078 10,706 Number of individuals who have applied new techniques or technologies as a result of USDA assistance (Water management (non￾irrigation based)) 2,930 6,541 Number of individuals who have received short- term agricultural sector productivity or food security training as a result of USDA assistance (Continuing) 18,000 11,893 Number of individuals who have received short- term agricultural sector productivity or food security training as a result of USDA assistance (People in government) 1,025 143 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 128 Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and CBOs that applied new technologies or management practices (trade and business associations) 15 8 Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community- based organizations (CBOs) that applied new farm management practices (i.e. governance, administration, or financial management) (continuing) 260 779 Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community- based organizations (CBOs) that applied new farm management practices (i.e. governance, administration, or financial management) (new) 190 771 Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community- based organizations (CBOs) that applied new farm management practices (i.e. governance, administration, or financial management) (POs) 140 752 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 129 Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community- based organizations (CBOs) that applied new farm management practices (i.e. governance, administration, or financial management) (private enterprises - NOT Processors) 30 104 Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community- based organizations (CBOs) that applied new farm management practices (i.e. governance, administration, or financial management) (trade and business associations) 36 28 Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community- based organizations (CBOs) that applied new farm management practices (i.e. governance, administration, or financial management) (women's groups NOT Processors) 24 60 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 130 Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community- based organizations (CBOs) that applied new technologies or management practices (continuing) 270 219 Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community- based organizations (CBOs) that applied new technologies or management practices (new) 86 470 Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community- based organizations (CBOs) that applied new technologies or management practices (Pos) 190 425 Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community- based organizations (CBOs) that applied new technologies or management practices (Private Enterprises -NOT processors) 22 43 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 131 Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community- based organizations (CBOs) that applied new technologies or management practices (Woomens Groups -NOT processors) 13 3 Training: Improved Marketing and Branding Number of service provider/ input supplier employees trained in marketing techniques 525 95 Training: Post-Harvest Handling and Processing Number of individuals who have applied improved farm management practices (i.e. governance, administration, or financial management) as a result of USD A assistance (business- ONLY processors) 1,175 1,913 Number of individuals who have received short- term agricultural sector productivity or food security training as a result of USDA assistance (People in firms- ONLY Processors) 1,200 3,313 Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community- based organizations (CBOs) that applied new farm management practices (i.e. governance, administration, or financial management) (private enterprises - ONLY Processors) 140 411 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 132 Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community- based organizations (CBOs) that applied new farm management practices (i.e. governance, administration, or financial management) (women's groups ONLY Processors) 94 195 Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community- based organizations (CBOs) that applied new technologies or management practices (private enterprises- ONLY Processors) 85 183 Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community- based organizations (CBOs) that applied new technologies or management practiccs(women's groups- ONLY Processors) 31 27 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 133 Annex E: Team Composition  Mr. Jason Wares, Team Leader, an M&E Expert at IMC, is an international development expert with 16 years’ experience specializing in monitoring and evaluation. Most recently, Mr. Wares led the Baseline evaluations for Counterpart in Senegal, as well as the Mid-term and Final evaluations of the USDA-funded McGovern Dole International Food for Education and Child Nutrition Program (MGD-FFE) in Senegal, which included a large quantitative data collection component. Additionally, he conducted the final evaluations of FINCA International’s Food for Progress USDA-funded projects in Haiti and Tanzania.  Mr. Chris Root, Co-Evaluator, is an agricultural economist with over 10 years’ experience in program evaluation and agricultural value chains. He has designed, led and implemented a wide range of evaluations, including several evaluations in Senegal. He strong technical skills in survey design and management, data analysis and qualitative data collection. Additionally, he has subject matter expertise and has led many value chain assessments, including of millet. He has worked with Mr. Wares on all of our previous evaluations in Senegal of USDA-funded programs including the LRP program.  Mr. Alioune Sow, is a Senegalese monitoring and evaluation expert with over 20 years conducting research in the Agriculture sector in Senegal. 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 134 Annex F: Evaluation Matrix Evaluation Question Measure Data Collection Instrument Data Analysis Operationalization of Evaluation Questions Goal Achievement 1. To what extent have project goals been achieved? Project outcome indicators  Desk review  Staff interviews  Strategy and project documents if available  Key Informant Interviews  Survey of Beneficiaries  Confirmation of reported achievement through documentation  Analysis of survey data  Analysis of qualitative data  Triangulation of findings FFPr SO1 - Yield of targeted agricultural commodities among project participants with USDA assistance (USDA Indicator Handbook) FFPr SO2 - Value of annual sales of farms and firms receiving USDA assistance (USDA Indicator Handbook) 2. Which activities have been successful in meeting their targets? Project outcome indicators  Review of monitoring and evaluation data  Partner organization and staff interviews  Strategy and project documents if available  Key Informant Interviews  Focus Groups of beneficiaries  Beneficiary surveys  Quantitative and qualitative data analysis  Triangulation of findings with project documentations 3. Which activities failed to reach their targets and why? Include analysis of factors that delay/prohibit adequate progress towards results? Project outcome indicators  Review of monitoring and evaluation data  Partner organization and staff interviews  Strategy and project documents if available  Key Informant Interviews  Focus Groups of beneficiaries  Beneficiary surveys  Identification of missed targets  Analysis of internal and external factors that contributed to missing targets 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 135 Evaluation Question Measure Data Collection Instrument Data Analysis Operationalization of Evaluation Questions Relevance 4. Are the project activities in alignment with Senegal Government priorities and strategies?  Desk review  Secondary research  KIIs with government officials  Comparison of Senegal Government priorities and strategies with Project activities and expected results 5. Do the project activities respond to the aspirations and needs of the project beneficiaries/participants?  Review of MBSP conducted assessments  Review of management system to show responsiveness to identified needs  Qualitative and quantitative data collection  Analysis of MBSP’s system for understanding participant needs and incorporating them into project design. 6. Are the project activities in alignment with USDA/US Government development goals, objectives, and strategies?  Review of project activities and the USDA agreement  Identification of discrepancies, if any, between the USDA approved proposal and the implementation of project activities. 7. Are the project activities in alignment with award agreement with USDA?  Identification of discrepancies, if any, between the terms of the award and implemented activities. Effectiveness 8. To what extent have different actors in the value chain been able to increase quantity e.g., yield/ha and minimize waste since the inception of the project? Increase in yield/ha and decrease in amount wasted/ha  Quantitative Data Collection tools.  KIIs and FGDs  Analysis of survey data  Comparison with baseline and mid-term data if possible  Triangulation with qualitative data 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 136 Evaluation Question Measure Data Collection Instrument Data Analysis Operationalization of Evaluation Questions 9. To what extent have quality standards been adopted by all actors in the value chain? Percentage of beneficiary groups that have adopted quality standards as a result of the project  Quantitative Data Collection tools.  KIIs and FGDs  Review of project records  Analysis of survey data  Comparison with baseline and mid-term data if possible  Triangulation with qualitative data 10. To what extent have value chain actors been trained in quality standards? Percentage of value chain participants that attended quality standards training  Quantitative Data Collection tools.  Project records  Analysis of survey data  Comparison with baseline and mid-term data if possible  Triangulation with qualitative data 11. Are partner organizations providing training to beneficiaries that are tailored to the local context and achieving increased use of agricultural techniques and technologies? Training system assessment  Quantitative Data Collection tools.  KIIs and FGDs  Review of project records  Description of the degree to which partner organizations use local knowledge and information to increase uptake of new practices 12. Are local government extension officers offering technical support to farmers in new techniques and technologies?  Frequency of contact with extension officers  Range of services provided  Quantitative Data Collection tools.  KIIs and FGDs  Review of project records  Analysis of frequency of contact with extension officers.  Analysis of range of services provided. 13. Have effective linkages been established between different project participants e.g., between producers and processors?  Percentage of participants that report the establishment of linkages.  Percentage of participants that report benefiting  Quantitative Data Collection tools.  KIIs and FGDs  Review of project records  Analysis of quantitative data on linkages.  Triangulation with qualitative data collection and project records 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 137 Evaluation Question Measure Data Collection Instrument Data Analysis Operationalization of Evaluation Questions in expected ways from linkages. Efficiency 14. What internal and external factors have influenced the ability of the project to meet expected results and targets? Strengths, Weaknesses, Opportunities, and Threats (SWOT) Analysis  Qualitative data collection  Review of MBSP narrative reports  Review of strategy documents  SWOT analysis to show factors that catalysed project implementation or served as obstacles to implementation 15. Do project staffing, training activities and management structures efficiently support the achievement of expected project outcomes? Management Review  Review of management, planning, and design documents  Interviews with staff and partners  Analysis of the project’s management systems and structures 16. Does the project resource allocation to activities and to the project’s geographic areas efficiently support the achievement of expected project outcomes? Budget Review and results analysis  MBSP financial reports  Analysis of project indicators and results  Analysis of the degree to which resources were spent to achieve optimal results. Sustainability 17. What factors are likely to contribute towards the sustainability of the following factors in relation to the MBSP: policy environment, financial incentives (i.e., profit motive), access to inputs and information, market access, etc.?  All primary and secondary data collection  Definition of sustainability for the project and description of contributing factors 18. What factors are likely to hinder sustainability of the following factors in relation to the MBSP: policy environment,  All primary and secondary data collection  Definition of sustainability for the project and description of contributing factors 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 138 Evaluation Question Measure Data Collection Instrument Data Analysis Operationalization of Evaluation Questions financial incentives (i.e., profit motive), access to inputs and information, market access, etc.? 19. What project activities are likely to contribute towards sustainability and is there evidence that project activities or benefits will be sustained after the project ends?  All primary and secondary data collection  Definition of sustainability for the project and description of contributing factors Impact 20. What impacts are the project activities having on the program participants, both positive and negative, especially in relation to the expected results and strategic objectives?  All primary and secondary data collection  Assessment of the objective level changes the project has created. 21. To what extent target groups (farmers, processors, vendors, consumers, etc) have adopted millet as a viable crop that has economic and nutritional benefits? Expansion of millet cultivation  Quantitative data collection tools  Description of expansion of millet cultivation and motivation for the expansion 22. What is the extent of the project reach beyond direct and targeted beneficiaries?  Indirect beneficiary survey  Analysis of qualitative data 23. What are the unexpected outcomes that could have been due to the project?  Description of unexpected outcomes, if any, found in qualitative data collection 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 139 Evaluation Question Measure Data Collection Instrument Data Analysis Operationalization of Evaluation Questions 24. What is the overall impact on increasing trade of millet? (marketing strategies, creation of new partnerships, market information systems, certifications, etc.) Response in EQ1, which addresses S02 25. How has the trade increased as a result of USDA MBSP on local, regional, national, and international level? Response in EQ1, which addresses S02 Best Practices and Lessons Learned 26. What lessons learned can be documented at this point to inform implementation of the program until completion? Project is complete  All data collection  Lessons learned from qualitative data, reports, and descriptions 27. What best practices can be shared with stakeholders and donors?  All data collection  Good practices distilled from project implementation 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 140 Annex G: Application of Recommended Mid-Term Actions Midterm recommendation Implementation summary Recommendation #1: That MBSP continues to support the Producer Organizations in Fatick in acquiring formal and legal status. Following the mid-term report, the project recognized that acquiring formal legal status for the hubs was important to their efficient operations. In its 2019 narrative report (April – September 2019), the project recognized that the time and expense of formalizing the hubs remained a challenge. Recommendation #2: That NCBA CLUSA continues to promote company registration, product certification, and labelling as well as supporting Processing Units to meet the quality standards necessary for obtaining the FRA number, especially in Fatick, Kaffrine, and Dakar. This recommendation essentially suggests that the MBSP do what it was already attempting to do across a number of initiatives. The MBSP did continue with these activities. Recommendation #3: That NCBA CLUSA investigates why the yield per hectare varies between regions. While the information the recommendation discussed may be interesting, it is not clear how it would have been useful for the project especially as it switched its focus from production to marketing. Recommendation #4: That NCBA CLUSA reinforce its training for Millet Producers on producing and identifying high quality seeds given that farmers are using the seeds that they produce for seeding the following year. In the last phase of the project, MBSP focused on marketing. Therefore, this recommendation was less relevant. Recommendation #5: That NCBA CLUSA and CRS support the Millet Producers to find credit and/or to save to buy seeds from the APS Semenciers. In the final stage of the project, the MBSP worked with hubs as a link to helping producers finance seed purchases. Recommendation #6: That NCBA CLUSA assist the APS Semenciers in negotiating contracts with the Producer Organizations and further build their skills in marketing their products. After the mid-term evaluation, the project focused on working with a large input provider and assisting that provider in working with the hubs. 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 141 Recommendation #7: That NCBA CLUSA and CRS investigate how Millet Producers, the PO, and the Village Council can access funds through financial institutions and savings to purchase rippers, making this tool a priority piece of equipment in millet cultivation. The beneficiaries should be further encouraged to apply for the MBSP grants for the acquisition of equipment. The ripper initiative did not appear to continue after the mid-term evaluation. An equipment manufacturer engaged by the project told the final evaluation team that 14 rippers were produced and that seemed to be the end of the initiative. Recommendation #8: That NCBA CLUSA and the Agricultural Extension Agents investigate why certain practices are not being followed; that retraining and/or support is provided to farmers where necessary. The final evaluation did not find evidence that this recommendation was implemented. Recommendation #9: That NCBA CLUSA reviews the role of the Extension Agents and investigate how to sustain their position within the PO to ensure that they can continue to support and coach the Millet Producers when the Program ends This recommendation was not realistic given the ANCAR’s limited budget. Recommendation #10: That NCBA CLUSA reinforces its training in composting, encouraging and supporting the use of compost to fertilize the soil. Following the mid-term evaluation the MBSP supported training in composting through the hubs for all hub members. Composting was found to be one of the project’s most useful trainings. Recommendation #11: That NCBA CLUSA works with its partners and the POs to reinforce knowledge and promote adoption of methods taught with reminders and refresher sessions prior to the wintering season. This is a vague recommendation. Most of MBSP’s trainings were delivered through intermediaries, including POs and Hubs. They delivered trainings to their members as they saw fit. Recommendation #12: That NCBA CLUSA verifies and ensures that the different components of training are being provided uniformly to the beneficiaries This again is an overly broad recommendation and in the opinion of the evaluation team, not appropriate. Trainings should be customized to the needs of trainees, not provided uniformly. 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 142 Recommendation #13: That NCBA CLUSA reinforces its training in sales and marketing, especially in Fatick and Kaffrine. It is unclear which actor types this recommendation refers to. Nevertheless, the project did focus on marketing, including through training, in the final phase. Recommendation #14: That NCBA CLUSA finds alternative methods to teach illiterate farmers and facilitate their learning. The final evaluation did not find that literacy was a constraint to farmer learning. Trainings were delivered by local intermediaries who were familiar with the learning abilities of farmers. See above. Recommendation #15: That NCBA CLUSA customizes its training methods according to the beneficiaries’ literacy level, agricultural experience and knowledge of millet cultivation. Recommendation #16: That NCBA CLUSA emphasizes the value chain in its training program and use case studies and hands-on methods to teach business concepts and marketing and to explain better the role of each actor along the value chain. Unfortunately the final evaluation did not find evidence that the project had taken a value chain approach. This was a good recommendation that should have been implemented. Recommendation #17: That NCBA CLUSA reviews the training material to make sessions more concise and practical; that training is scheduled to be least disruptive to trainers and beneficiaries. This recommendation was made in response to the mid-term survey that found that trainers and participants had complaints about the scheduling and length of trainings. The recommendation is a non-specific response to those complaints. Recommendation #18: That NCBA CLUSA verifies the membership status of its beneficiaries in the PO that it is supporting. This was not implemented thoroughly. The database of members and POs was incomplete and partly unusable. The project did attempt to improve hub registration, holding events to promote hub engagement with producers. Ultimately, membership listing was incomplete. Recommendation #19: That NCBA CLUSA continues to emphasize hygienic standards and draw attention to gaps and deficiencies during site visits The recommendation suggests that the project continue doing what it is already doing. Coaches continued to emphasize hygienic standards with project participants, as reported in the final evaluation section 3.3.2 Quality Adoption. 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 143 Recommendation #20: That NCBA CLUSA reviews the relevance of its packaging initiative and its partnership with the APS Packaging. It is unclear if this review was carried out. However, there was some evidence of packaging not being consistently relevant. Some processors preferred plastic packaging; however APS recommended paper packing only. Recommendation #21: That NCBA CLUSA and Extension Agents reinforce the business and marketing training of the Millet Producers and emphasize price setting based on supply and demand. ANCAR extension agents mostly are agricultural specialists without the background to deliver business training. Business trainings were delivered by the project to Hubs and POs. Price setting based on supply and demand is too general a recommendation to be useful. Producers have no choice but to accept market price. Recommendation #22: That NCBA CLUSA reviews the marketing strategy of the Multi-Service Hub; that they encourage negotiation with Processing Units, especially the beneficiaries of the MBSP secure long- term agreements for program sustainability. MBSP facilitated several long term relationships between hubs and processors. Recommendation #23: That NCBA CLUSA verifies whether all Processing Units are currently in operation. If not, it is recommended that NCBA CLUSA documents closure reasons to help identify more effective beneficiary selection criteria. The final evaluation found several processing units not in operation. Not all businesses succeed, so a less than 100% success rate is not necessarily evidence of ineffective beneficiary selection, as the recommendation suggests. Recommendation #24: That NCBA CLUSA investigates why the MBSP Processing Units are not in operation throughout the year. It is unclear whether or not this investigation was carried out. However the final evaluation did not find that an unexpectedly high number of processors were not in operation. Recommendation #25: That NCBA CLUSA reviews its strategy for supporting the Processing Units, focusing on equipment that can increase efficiency and productivity. Following the mid-term evaluation, the MBSP delivered grants of equipment to 50 processing units. Equipment included drying tables, gas cookers, dryers, and hardware. 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 144 Recommendation #27: That NCBA CLUSA continues to promote synergy between processors, consumers, wholesalers, retailers, and the export market and encourage the establishment of long-term partnerships among the trading actors in the Millet Value Chain. See reply to recommendation 22. Recommendation #28: That CRS investigates the possibility of depositing SILC funds at banks to earn interest when large sums of money are waiting to be spent on the next wintering season. The evaluation team did not find evidence that this recommendation was acted upon. Recommendation #29: That CRS and the APS-IF keep track of the disbursement of the SILC savings, categorizing them in millet and non￾millet expenditure and investments for monitoring purposes and research This recommendation is for CRS. CRS was not an active partner after mid-term Recommendation #30: That NCBA CLUSA reviews the objectives of the SILC Program and any other financial activities that is being promoted within MBSP Catholic Relief Services. That targets are set for the number of SILC Groups formed and funds raised for investments in the millet sector. That CRS and the APS-IF form and train SILC Groups only with producers who formally agree to direct the SILC Savings to Millet related activities and investments. The MBSP refocused its financial services activities after the mid-term. Recommendation #31: That initiatives such as SILC training and SILC Groups initiated within MBSP by the APS-IF be directly related to the Millet sector and formally tied to a specified investment in millet production or trade before the initiative can start with an PO or in a village community. This recommendation explains why SILCs were de-emphasized. It was too difficult for MBSP to ensure that loans through SILCs were spent on either millet specifically, or agriculture generally. 1. PROJECT TITLE – REPORT TYPE (E.G. DRAFT BASELINE, FINAL REPORT) – DATE 145 Recommendation #32: That the role and responsibilities of the APS-IF be reviewed; that the supervision and monitoring of the APS-IF be formally assigned to one of the stakeholders; that NCBA CLUSA ensures that there is no misuse of the CRS Certification and MBSP by the APS-IF. The MBSP did not appear to support the APS-IF role following mid-term. Recommendation #33: That the needs of the smallholder millet farmers be further investigated for a better understanding of how the SILC savings are being spent and what are the aspirations and expectations of the beneficiaries when it comes to their commitment to the production of millet. Please see response to recommendation 31. Recommendation #34: That NCBA CLUSA reviews its staff’s workload and introduces quality assurance measures in its activities and program deliverables. The MBSP management had a process of work review as described in section 3.4.2 of the final evaluation. However, this system was in place before the mid-term evaluation. Recommendation #35: That NCBA CLUSA reviews its monitoring system and sets clear procedures for the registration of beneficiaries; identifies and collects accurate information that is needed for performance monitoring. This was not implemented as evidenced by the poor quality of its beneficiary registration data and lack of activity monitoring data. Recommendation #36: That NCBA CLUSA makes important project documents available in English and French. This is a poor recommendation. It was made in response to the mid-term evaluator’s frustration that important documentation could not be made available in a timely manner. While it is certainly true that the project’s documentation of its activities were insufficient, it is not the project’s responsibility to make sure that they are available in the evaluator’s language of choice. A more appropriate recommendation would have been that NCBA CLUSA improve the documentation of its project activities and make sure that information is easily accessible.