EVALUATION REPORT Final Performance Evaluation of USAID/Sri Lanka Supporting Opportunities for Livelihoods Development (SOLID) Activity October 2017 USAID Contract AID–486–I–14-00001 (IDIQ); AID–383–TO–17–00003 (Task Order) DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. PHOTO BY DAVID RINCK FOR SOCIAL IMPACT II USAID.GOV Final Performance Evaluation for SOLID Activity ABSTRACT The United States Agency for International Development/Sri Lanka launched the Supporting Opportunities for Livelihoods Development (SOLID) activity in 2013 to increase economic opportunities for vulnerable families in target provinces of Sri Lanka. This mixed-methods, final￾performance evaluation focuses on the following: (1) how effective SOLID was in achieving the project’s intended outcomes, (2) whether project activities influenced USG and Government of Sri Lanka (GoSL) relations, (3) if there were any intended or unintended effects on agricultural value chains and their sustainability, and (4) if, and how, beneficiaries are implementing skills and techniques taught by the project. Overall, SOLID improved agricultural output in all three of the target regions evaluated and in all three target sectors (dairy, horticulture, and poultry). This output was achieved in part by deploying effective techniques and technologies derived from commercial value chains, such as silage. SOLID succeeded in engaging women in agriculture and in providing access to productive assets, but SOLID’s success was more mixed with respect to engaging youth. SOLID significantly improved household income and generated positive changes in household nutrition due to increased income generated by farming activities. The relationship between SOLID and the GoSL was characterized more by coordination and support, with limited influence on relevant GoSL activities and approaches. SOLID had limited partnerships with previous USG projects/activities and with other donors working in this space. Finally, the vast majority of past beneficiaries are currently implementing the skills and techniques promoted by SOLID, and most farmers involved in demonstration activities have adopted recommended practices. III USAID.GOV Final Performance Evaluation for SOLID Activity CONTENTS ACRONYMS V EXECUTIVE SUMMARY VI EVALUATION PURPOSE AND EVALUATION QUESTIONS VI PROJECT BACKGROUND VI EVALUATION DESIGN, METHODS, AND LIMITATIONS VI KEY FINDINGS VII CONCLUSIONS IX RECOMMENDATIONS X INTRODUCTION 1 PROJECT DESCRIPTION 1 DEVELOPMENT HYPOTHESIS 2 CRITICAL ASSUMPTIONS 2 EVALUATION PURPOSE AND QUESTIONS 2 METHODS AND LIMITATIONS 3 LIMITATIONS AND POTENTIAL BIASES 6 FINDINGS AND CONCLUSIONS 7 EVALUATION QUESTION ONE - FINDINGS 7 EVALUATION QUESTION ONE - CONCLUSIONS 15 EVALUATION QUESTION TWO - FINDINGS 16 EVALUATION QUESTION TWO - CONCLUSIONS 18 EVALUATION QUESTION THREE - FINDINGS 18 EVALUATION QUESTION THREE - CONCLUSIONS 20 EVALUATION QUESTION FOUR - FINDINGS 21 EVALUATION QUESTION FOUR - CONCLUSIONS 23 RECOMMENDATIONS 23 ANNEX 1: SCOPE OF WORK 26 ANNEX 2: EVALUATION TEAM 33 ANNEX 3: DATA COLLECTION SCHEDULE 35 ANNEX 4: REFERENCES 36 ANNEX 5: KEY INFORMANT INTERVIEWS 38 ANNEX 6: FOCUS GROUP DISCUSSIONS 39 ANNEX 7: MINI-SURVEY PARTICIPANTS 40 ANNEX 8: DATA COLLECTION PROTOCOLS 41 ANNEX 9: PROJECT BACKGROUND 51 IV USAID.GOV Final Performance Evaluation for SOLID Activity ANNEX 10: SOLID ACTIVITIES BY PROVINCE 53 ANNEX 11: SOLID RESULTS FRAMEWORK 54 ANNEX 12: EVALUATION DESIGN MATRIX 55 ANNEX 13: EVALUATION TEAM CONFLICT OF INTEREST FORMS 57 LIST OF FIGURES AND TABLES TABLE 1: KEY INFORMANT INTERVIEWS BY STAKEHOLDER GROUP 4 TABLE 2: FOCUS GROUP DISCUSSIONS BY SECTOR 4 TABLE 3: MINI-SURVEY PARTICIPANTS BY SUB-GROUP 5 GRAPH 1: FARM ACREAGE AREA INCREASE 9 GRAPH 2: CHANGE IN INCOME BY SECTOR 10 GRAPH 3: WOMEN WERE AS LIKELY AS MEN TO BENEFIT FROM SOLID ACTIVITIES 13 GRAPH 4: SOLID ACTIVITY’S EFFECTIVENESS IN IMPROVING NUTRITION 14 GRAPH 5: LIKELINESS TO CONTINUE IMPLEMENTING TECHNIQUES LEARNED THROUGH SOLID 21 GRAPH 6: SOLID PRACTICES WERE EASY TO IMPLEMENT 22 FIGURE 1: ILLUSTRATIVE CROP BUDGET 25 V USAID.GOV Final Performance Evaluation for SOLID Activity ACRONYMS ALME Asia Learning and Monitoring and Evaluation Support Project AMEG Asia and Middle East Economic Growth CDCS Country Development Cooperation Strategy CIC CIC Holdings CII Chemonics International, Inc. DO Development Objective EQ Evaluation Question EQUI® Evaluation Quality Use and Impact ET Evaluation Team FAO Food and Agriculture Organization FGD Focus Group Discussion FO Farmer Organization GAP Good Agricultural Practices GDP Gross Domestic Product GoSL Government of Sri Lanka IP Implementing Partner KII Key Informant Interview LKR Sri Lankan Rupees M&E Monitoring and Evaluation MED Ministry of Economic Development NGO Non-Governmental Organization PE Performance Evaluation PMMU Project Monitoring and Management Unit RISEN Reintegration and Stabilization in the East and North Project SI Social Impact Inc. SOLID Supporting Opportunities for Livelihoods Development SOW Scope of Work TOT Training of Trainers US$ United States Dollars USAID United States Agency for International Development USDA United States Department of Agriculture USG United States Government VI USAID.GOV Final Performance Evaluation for SOLID Activity EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The purpose of this performance evaluation (PE) is to identify factors that facilitated or inhibited the achievement of the Supporting Opportunities for Livelihoods Development (SOLID) results and to document best practices to inform and enhance future agriculture and economic development interventions in Sri Lanka and other countries. This evaluation is intended to be a learning tool for the United States Agency for International Development (USAID)/Sri Lanka to assess the effectiveness of the SOLID program and to serve as a reference work for USAID, Chemonics, the Government of Sri Lanka (GoSL) entities, and other donor organizations for similar activities. The evaluation is intended to be used for future planning of similar programs aimed at accelerating livelihood opportunities, economic growth, and reducing poverty in post-conflict regions. The summarized evaluation questions (EQ) focused on the following: EQ1: Project activity effectiveness and if or how results varied by key stakeholder group; EQ2: If or how SOLID activities influenced relationships between the United States Government (USG) and the GoSL (at national, provincial, and district levels) and how stakeholders view a productive relationship; EQ3: Identification of any intended and unintended effects on agricultural value chains and their potential sustainability due to the participation of various stakeholders; and EQ4: Identification of if or how past beneficiaries, including farmers, are implementing the skills, techniques, and other practices SOLID introduced. PROJECT BACKGROUND SOLID is a US$10,785,000, 46-month pilot activity conducted through the Asia and Middle East Economic Growth (AMEG) Best Practices Project and implemented by Chemonics International, Inc. (CII). SOLID’s target was to assist 3,000 vulnerable households in conflict-affected provinces in the north and east of Sri Lanka to strengthen their livelihoods through improvements in farm income and nutrition. SOLID hypothesized that transitioning subsistence farming activities to commercial operations through the introduction of new technologies would result in the production of higher volumes and higher-value agricultural products and strengthened linkages between target farmers and the rest of the value chain. An initial livelihoods analysis suggested that opportunities for these upgrades lie in the dairy, horticulture, and poultry sectors.1 According to the SOLID scope of work (SOW), the activity also aimed to collaborate with other agencies, donors, and organizations in order to promote local capacity building, leverage donor resources and promote gender equality and women’s empowerment. EVALUATION DESIGN, METHODS, AND LIMITATIONS The evaluation’s period of performance was between June 3 and October 31, 2017, with fieldwork occurring between July 17 and August 14. Fieldwork was conducted in three provinces of Sri Lanka (Eastern, Northern, and North Eastern). The evaluation team (ET) was composed of five members supported by Social Impact headquarters in Washington DC. This evaluation employed a mixed methodology, incorporated gender considerations, and included youth in the design and analysis. Data collection methodologies included a document review, site visits, 40 key informant interviews (KIIs), 21 focus group discussions (FGD) with 181 farmers in the dairy, horticulture, and poultry sectors (beneficiaries), and a quantitative mini-survey of 174 farmers. Data analysis methods employed by the ET to produce findings and conclusions included a content analysis of KII and FGD data to 1 “Supporting Livelihoods for Vulnerable Populations in Sri Lanka - Assessment of Findings and Opportunities,” AMEG and USAID, March 29, 2013. VII USAID.GOV Final Performance Evaluation for SOLID Activity identify and highlight notable examples of factors that facilitated or hindered the project, a trend analysis to identify patterns of convergence (or divergence) that affected activity outputs and outcomes, and a gender and youth analysis to capture and compare the results of the program as it specifically benefited (or did not benefit) women, men, and youth. Limitations of this PE include sampling related to the purposive selection of an illustrative sub-set of beneficiaries for FGDs and a survey to include at least two districts in each region evaluated and to reflect the proportional distribution of overall project beneficiaries with regard to sector. In addition, time and resource constraints limited the ability of the ET to meet with beneficiaries in several of the horticultural sub-sectors addressed by SOLID. KEY FINDINGS EQ 1: To what extent did the intended outcomes in agriculture, household income, women and youth’s engagement, household nutritional intake and access to productive economic assets occur over the course of SOLID? Did outcomes differ by region, gender of beneficiary, or sector (dairy, poultry and horticulture)? In FGDs, both male and female beneficiaries in the dairy and poultry sectors cited the effectiveness of technology introduced by the project as a catalyst for expanding the number of animals they held.2 Likewise, both male and female poultry farmers cited improved chicken coops as the catalyst for expanding poultry operations and facilitating access to new commercial markets. Horticulture farmers reported improved market linkages through the adoption of new commercial crops. Many dairy farmers specifically described how livestock had overtaken paddy production as their primary occupation due to expansions in livestock activity. However, severe weather patterns (drought, flooding) negatively affected all regions evaluated, reduced water for irrigation, and undermined production of horticultural crops and silage. Despite adverse weather, the majority of beneficiaries of both genders surveyed reported an increase in incomes across all sectors, with dairy farmers experiencing the most significant average rise at 33.4 percent, followed by poultry farmers at 31.3 percent and horticulture farmers at 28.7 percent (both male and female farmers expressed a high level of confidence in their income statements and had a good grasp of costs of production and the profitability of activities, due to active recordkeeping promoted by SOLID). Although starting from a lower level, reported overall income gains were higher for female beneficiaries surveyed (36.7 percent) compared to male beneficiaries surveyed (25.5 percent). The difference in these income gains was largest in the poultry sector, followed by horticulture and dairy. In terms of women’s participation, beneficiaries reported significant improvements in women’s engagement in agriculture and increased access to productive economic assets, due to SOLID’s focus on intensive components of household farming systems, i.e., horticulture and poultry, which in traditional South and Southeast Asian farming systems are generally managed by women, as opposed to the extensive component, i.e., staples grains, such as paddy. In contrast, responses regarding the extent of youth engagement in agriculture were ambiguous in some cases, and the opinions expressed in FGDs suggested that youth engagement is dependent on personal career preferences and regional proximity to urban conveniences. Horticulture and poultry farmers stated that SOLID activities helped increase income and facilitate improved purchasing power for food and, hence, household nutrient intake. Dairy farmers stated that, in 2 In all cases, SOLID “beneficiaries” were farmers and, hence, the two terms are interchangeable. VIII USAID.GOV Final Performance Evaluation for SOLID Activity addition to reporting significantly increased income, increased production of milk led to increased home consumption. However, virtually no farmers reported changing their nutritional behavior significantly, i.e., increased food diversity and changes in dietary preferences. EQ2: What influence did SOLID have on the USG/GoSL relationship at the national and local levels? What does a productive relationship look like based on the SOLID experience? In KIIs, GoSL and IP (Implementing Partner) respondents explained that SOLID’s start-up struggled for approval from the GoSL, who restricted its early operations and created pressure to transfer resources to government entities. SOLID was assigned to the Ministry of Economic Development (MED), which was abolished, and the project was shifted to the Ministry of National Policy Planning, severing it from the Project Monitoring and Management Unit (PMMU); subsequently, reporting has not taken place. GoSL representatives indicated that SOLID activities complemented GoSL priorities in the agriculture and dairy sectors at the local level (provincial Departments of Animal Production and Health and Agriculture) but also emphasized the need for deeper coordination of objectives and integration of resources, as well as a high level of interest in “assimilating” resources and effective approaches. Specifically, staff at the Ministry of National Policy Planning expressed an interest in adopting successful SOLID methodologies and training materials to support the legitimacy of the agency. EQ3: Identify planned and unexpected results in relation to ensuring potential sustainability and effect on the local agricultural value chain through partnering with other USG projects/activities, donors, GoSL, or international, national, and local private sector firms. Project documentation describes only a very limited partnership with existing USG and other donor projects, noting that, despite widespread donor interest in the country, the focus of most programs at the time of SOLID start-up was on humanitarian assistance to address post-conflict and tsunami-related reconstruction. Respondents described a difficult operating environment prevailing during the initial year of the project, which meant that few donor programs were compatible with SOLID, limiting effective partnership opportunities. However, GoSL and IP staff stated that SOLID support for the introduction of silage to increase milk yield was likely to advance GoSL agency activities in this area, thereby expanding the impact of this technology. Likewise, they stated that SOLID support for onion and chili seed production is likely to facilitate enhancing GoSL activities around increasing available seeds. Dairy and horticulture (especially passion fruit) farmers stated that they perceive a strong commitment from private sector buyers to continue support for production and preserve market linkages. Nestlé staff in particular expressed a commitment to investing in maintenance of milk chilling units provided by SOLID. EQ4: Are past beneficiaries currently implementing the skills and techniques promoted by SOLID? Did farmers involved in demonstration activities adopt the recommended practices? The majority of past beneficiaries included in FGDs and KIIs report that they are currently implementing the techniques promoted by SOLID. Furthermore, 92 percent of survey respondents in dairy, 75 percent of respondents in horticulture, and 100 percent of respondents in the poultry sector agreed or strongly agreed that they are likely to continue to implement these skills and techniques. With the exception of morunga tree farmers in Vavuniya, the only agriculture-related reason cited for not continuing to practice the skills and techniques promoted by SOLID was related to adverse effects of weather. In a related finding, many farmers expressed appreciation of the SOLID program, stating that they had previously had trouble accessing technical information related to agriculture. They also stated that low Internet literacy, especially for older farmers, prevented them from accessing information online, and that youth often facilitated this access (a web search revealed a wealth of online information related to SOLID crops in both Sinhalese and Tamil). IX USAID.GOV Final Performance Evaluation for SOLID Activity CONCLUSIONS EQ1: SOLID improved agricultural outputs in all three of the target regions evaluated and across genders in all three of the target sectors by deploying effective techniques and technologies derived from commercial value chains, such as silage. This facilitated an increase in production volumes in order to reach commercially viable scales, or access to markets through the adoption of new commercial crops. Outcomes did not vary across the regions evaluated. However, climactic challenges reduced horticultural production. Improved outputs, demonstrated profitability, and market linkages translated farmers’ investments into expansions of their farming systems, resulting in significant improvements in household income. Promotion of farmers’ recordkeeping also enhanced the adoption of SOLID skills and techniques, and facilitated project reporting and evaluation. SOLID’s selection of activities related to intensive components of household farming systems led to a high-level of women’s engagement in agriculture and access to productive assets. Regarding youth engagement in agriculture, the level of youth engagement has been ambiguous. While several general assumptions regarding youth engagement in agriculture can be deduced, youth engagement in agriculture is primarily subject to individual interests and preferences. The SOLID project successfully generated positive changes in household nutrient intake through increased home consumption of milk, due to increased income generated by farming activities and families engaged in dairy production. However, activities specifically related to nutritional behavior change were not sufficient to affect major changes in nutritional preferences and behaviors. EQ2: The relationship between SOLID and the GoSL was characterized more by coordination and support, with limited influence on relevant overall GoSL activities and approaches. The dissolution of the MED following the change in administration in 2015 resulted in a lack of official reporting and, therefore, influence, on the USG/GoSL relationship. The change in administration did, in fact, result in a more collaborative relationship. A productive relationship would entail understanding the goals and capacities of the GoSL’s relevant ministries and identifying entry points for GoSL assistance and support (by Ministry) at national and local levels. GoSL interest in assimilating resources and contributing to project impacts is likely to have a positive effect on spreading the impact of these resources. EQ3: SOLID had limited partnership with previous USG projects/activities. SOLID support for the Department of Agriculture and Department of Animal Production and Health objectives in the dairy (silage) and horticulture (onion and chili seed) sectors are likely to broaden the effect on agricultural value chains through wider dissemination to farmers. Stated private sector intentions to continue to support SOLID approaches and resources are likely to increase sustainability and intensify the effect of SOLID activities on the dairy and horticultural value chains. Notably, the partnership with Nestlé is especially likely to foster sustainability and impact as the company continues to support maintenance of milk-chilling equipment and silage. EQ4: The vast majority of past beneficiaries are currently implementing the skills and techniques promoted by SOLID, and most farmers involved in demonstration activities have adopted recommended practices. Older farmers perceive that there will be challenges in accessing information on new techniques and technologies that can improve their farming systems since they do not regard themselves X USAID.GOV Final Performance Evaluation for SOLID Activity as Internet literate and anticipate challenges in accessing information in local languages. Youth can act as intermediaries, connecting older household members with available online information. RECOMMENDATIONS Recommendation 1 (EQ1-4): Implementers should adopt a two-step approach to livelihoods assessment to inform project design: 1. Initially, identify technical areas of focus (agriculture, access to finance, SMEs, etc.) in order to identify activities relevant to context (post-conflict reconstruction, disaster recovery, etc.). 2. Subsequently, apply specialized technical analysis to specific activities. Implementers should assess the prevailing level of development of value chains vis-à-vis more developed comparable commercial value chains and focus on adoption of commercial practices and technologies. Recommendation 2 (EQ1,3&4): USAID and implementing partners should include record keeping (e.g., farm budgets, cost of production data) as an integral activity to demonstrate profitability of new technologies and inputs, develop producers’ analytical skills to evaluate new technologies and inputs, promote innovation, and facilitate Monitoring and Evaluation (M&E) data collection through aggregation of records. Recommendation 3 (EQ1): Given the prevailing gender roles relating to management of the various components of family farming systems in the region, USAID and IPs in Sri Lanka and other similar countries should focus on intensive (as opposed to extensive) components of household farming systems in order to promote women’s engagement in agriculture. Intensive components include horticulture and poultry, as opposed to staple grains. Recommendation 4 (EQ1-3): Given Nestlé’s established track record of active support for USG and other donor development initiatives in various value chains, USAID should explore increasing partnership with Nestlé as a potential development alliance. Recommendation 5 (EQ1-4): USAID and IPs should include a web presence (e.g., tutorial videos, technical resources) in future projects in Sri Lanka in order to engage youth in activities to facilitate the transfer of knowledge to older farmers, who may not feel adequately skilled to access online information that exists, and increase the scale of impact of similar projects on selected value chains. However, USAID should take care not to duplicate information already available online in Sinhala and Tamil. Recommendation 6 (EQ2-3): Given GoSL interest in “assimilating” resources and responsibility for positive impact, improved coordination between implementers and the GoSL is likely to increase positive impact of activities on value chains. Improved coordination can take the form of greater coordination in terms of assessments, selecting target sectors, and developing methodologies, and exchanges, as well as increasing engagement of GoSL staff in activities such as training of trainers (TOT) and seconding of project staff. INTRODUCTION PROJECT DESCRIPTION SOLID is a US$10,785,000, 46-month pilot activity conducted through the Asia and Middle East Economic Growth (AMEG) Best Practices Project and implemented by Chemonics International, Inc. (Chemonics), working in partnership with the Government of Sri Lanka (GoSL) and the private sector. Sri Lanka suffered from a 26-year civil war that displaced more than one million people, destroyed major infrastructure, and devastated livelihoods. While the country has seen rapid economic growth since the conflict’s end in 2009, this growth is highly unequal among the country’s nine provinces, especially in the areas most affected by conflict, which are the northern and eastern parts of the country. Relative to the Western Province, which had a per capita gross domestic product (GDP) of Sri Lankan Rupees (LKR) 77,117 in 2015, the Northern and Eastern Provinces had GDP per capita of LKR 357,120 and LKR 415,331 respectively.3 Against this backdrop, the United States Agency for International Development (USAID)/Sri Lanka’s Development Objective (DO) #2 of its 2016-2019 Country Development Cooperation Strategy (CDCS) aimed to foster “sustained and inclusive economic growth” through SOLID. The activity specifically aimed to increase economic opportunities for vulnerable families in target provinces of Sri Lanka. Originally slated to be a two-year pilot valued at US$8 million, SOLID received two extensions of the period of performance and corresponding budget increases, and was ultimately implemented from November 2013 to August 2017. SOLID’s original target was to assist 3,000 vulnerable households during the life of the Activity; an additional 1,000 households were added during the extension period. SOLID’s objectives were to strengthen the livelihoods and productivity of farming households, in turn improving their nutrition. Project activities focused on dissemination of productivity-enhancing technologies, improved market access, more widespread use of good agricultural practices (GAP), and measurable improvements in nutritional intake. Other important goals included increasing access to finance and ensuring that smallholder farmers understood business practices to improve competitiveness. Achieving these objectives often entailed increasing smallholder farmer membership in farmer organizations, as well as increasing their links to financial service providers. Geographically, the project also focused on reaching conflict-affected provinces in the north and east to serve vulnerable groups and promote inclusive development (though SOLID was ultimately expanded to non-conflict Uva Province in the south of the country as well, at the request of the GoSL). SOLID also aimed to “collaborate, wherever possible, with other agencies, donors, and organizations, to promote the building of capacity within local organizations, leverage donor resources, and promote gender equality and female empowerment.” 4 3 The exchange rate at the time of writing is approximately US$1.00 = LKR 154. 4 “Scope of Work – Sri Lanka Supporting Opportunities in Livelihoods Development Pilot Activity,” Chemonics, November 1, 2013. 2 USAID.GOV Final Performance Evaluation for SOLID Activity DEVELOPMENT HYPOTHESIS The SOLID Activity development hypothesis stated that, by increasing financial capital (household-level income), physical capital (productive assets), social capital (access to markets), and human capital (knowledge and nutrition), SOLID would strengthen target groups’ livelihoods and encourage more equitable and sustainable economic growth in post-conflict regions. Increasing capital and assets would be achieved by transitioning subsistence farming activities to commercial operations through the introduction of new technologies and modernization and diversification of agriculture processes and products, resulting in production higher volumes and higher-value agricultural products, and strengthened linkages between target farmers and the rest of the value chain, especially linkages with commercial input suppliers and private sector buyers and local markets. Initial analysis suggested that these opportunities would lie in the dairy, horticulture, and poultry sectors.5 CRITICAL ASSUMPTIONS In the activity’s initial design, Chemonics assumed that SOLID activity would complement and derive synergies with GoSL investments in infrastructure, nutrition and health (and social and spiritual development). This was envisioned to occur due to similarities with the GoSL Divi Neguma strategy, which promoted home-based horticulture and livestock production to “improve not only the household’s nutritional food security, but also its financial stability and economic wealth”6. The SOLID activity selected the dairy, horticulture, and poultry sectors as potential drivers of economic growth and, hence, improved livelihoods due to several considerations, including: a) Several districts in the targeted geographic areas demonstrating a comparative advantage in these sectors before the hostilities; b) The growing levels of dairy and poultry consumption in Sri Lanka commensurate with increasing GDP; c) The potential to increase food security and household resiliency; d) The existence of a competitive market in the industries, all of which rely on household producers; e) The robust availability of skilled public and private agricultural extension services; and f) The large number of individuals within the targeted vulnerable populations who have familiarity and/or experience in these sectors.7 In this context, Chemonics assumed that the project would provide model pilot activities that would be “replicable, scalable, and sustainable” 8. EVALUATION PURPOSE AND QUESTIONS In 2017, USAID/Sri Lanka engaged Social Impact, Inc. (SI) to conduct a final performance evaluation (PE) of the SOLID activity (see Annex 1: Scope of Work). This PE illustrates the factors that facilitated or 5 Ibid 6 Ibid 7 Ibid 8 Ibid 3 USAID.GOV Final Performance Evaluation for SOLID Activity inhibited the achievement of SOLID results and documents best practices to inform and enhance future agriculture and economic development interventions in Sri Lanka and other countries. This evaluation is intended to be an effective learning tool for USAID/Sri Lanka to assess the effectiveness of the SOLID program. The PE is also intended as a reference work for USAID, Chemonics, GoSL entities, and other donor organizations for future planning of similar agriculture programs aimed at accelerating livelihood opportunities, economic growth, and reducing poverty in post-conflict regions. The PE seeks to inform four Evaluation Questions (EQs), including: EQ1: To what extent did the intended outcomes in the bulleted list below occur over the course of SOLID? Did outcomes differ by region, gender of beneficiary, or sector (dairy, poultry and horticulture)?: • Improved agricultural outputs • Household income • Women’s engagement in agriculture and access to productive economic assets • Youth engagement in agriculture • Household nutritional intake EQ2: What influence did SOLID have on the USG/GoSL relationship at the national and local levels? What does a productive relationship look like based on the SOLID experience? EQ3: Identify planned and unexpected results in relation to ensuring potential sustainability and effect on the local agricultural value chain through partnering with: • Other USG projects/activities • Other donors • Government of Sri Lanka • International, national, and local private sector firms EQ4: Are past beneficiaries currently implementing the skills and techniques promoted by SOLID? Did farmers involved in demonstration activities adopt the recommended practices? METHODS AND LIMITATIONS The evaluation team (ET) consisted of a six-person field team led by the Team Leader, Mid-Level Livelihood and Agriculture Specialists, a Research Assistant, and an Interpreter fluent in both Sinhala and Tamil. A Logistician based in Colombo provided support as well (see Annex 2: Evaluation Team for additional information on the team’s qualifications, experiences and roles). The field team also received support from SI’s HQ-based Management Team, including, a Project Director, Project Manager, Project Assistant, and Gender Specialist. The ET collected data between July 17 and August 14 (see Annex 3: Data Collection Schedule). The ET employed a mixed-methodology, which incorporated gender and social inclusion throughout. All respondents were purposively selected from a sub-set of beneficiaries; the lists were provided by 4 USAID.GOV Final Performance Evaluation for SOLID Activity Chemonics and disaggregated by sector. This selection allowed for a proportional distribution of overall project beneficiaries in each sector. Data collection methodologies included: 1. Document Review, including annual reports, quarterly reports and M&E Plans, as well as review of analytical documents, including the initial livelihoods assessment9, in order to understand the project design process and assumptions, and assess reported implementation challenges and progress toward objectives (see Annex 4: References). 2. Key informant interviews (KII) to evaluate programmatic effectiveness (EQ1), as well as to solicit opinions on how SOLID activities influenced the relationship between the USG and the GoSL (EQ2). The ET implemented 40 KIIs with USAID, GoSL, IPs (Chemonics), and private sector staff, as well as with farmers in each selected sector (dairy, horticulture, and poultry). Twenty-eight percent of KIIs were with females. Key informants were selected in consultation with USAID staff to represent a cross-section of stakeholder groups, and the ET also identified additional key informants during the interview process (see Annex 5: Key Informant Interviews). TABLE 1: KEY INFORMANT INTERVIEWS BY STAKEHOLDER GROUP Stakeholder Group KIIs (#) Females (#) Males (#) Females (%) Males (%) USAID Staff 5 3 2 60% 40% Implementing Partner Staff 8 2 6 25% 75% GoSL Staff 10 1 9 10% 90% Private Sector Staff 8 0 8 0% 100% Farmers (Beneficiaries) 9 5 4 56% 44% Total (n) 40 11 29 28% 72% 3. Focus group discussions (FGD) with 185 participants (118, or 63 percent, were males and 67, or 37 percent, were females) to evaluate programmatic effectiveness (EQ1), identify intended and unintended activity impacts (EQ3), and examine whether past beneficiaries are implementing the skills, techniques and other practices introduced (EQ4). The ET undertook 21 FGDs, distributed to include proportions of participants in line with overall populations of farmers by selected sector. Where possible, the ET sought to have single-sex FGDs, dependent on the number of female farmers in a given sector. As such, the ET implemented nine FGDs with beneficiaries engaged in dairy (40 percent of total beneficiaries), ten FGDs with beneficiaries engaged in horticulture (55 percent of total beneficiaries), and two FGDs with beneficiaries engaged in poultry (5 percent of total beneficiaries). 9 “Supporting Livelihoods for Vulnerable Populations in Sri Lanka - Assessment of Findings and Opportunities,” AMEG and USAID, March 29, 2013. TABLE 2: FOCUS GROUP DISCUSSIONS BY SECTOR Sector Beneficiaries (#) Sector (%) FGD's (#) Participants Females (#) Females (%) Dairy 1,739 40% 9 78 21 27% Horticultu re 2,425 55% 10 93 40 43% Poultry 214 5% 2 10 6 60% Total (n) 4,378 21 181 67 37% 5 USAID.GOV Final Performance Evaluation for SOLID Activity FGDs were also scheduled to include a selection of beneficiaries in at least two districts in each of the three provinces evaluated (Northern, North Central, and Eastern). Ultimately, the ET conducted FGDs in six districts in Northern Province and two each in the North Central and Eastern Provinces. Finally, the FGDs were also designed to include proportions of women and youth by organizing FGD sites at women- and youth-owned farms, holding discussions at times convenient for women participants and by specific invitation. In cases where they existed, the ETs held FGDs with women- and youth-based FOs (farming organizations), including the Youth Farmer Initiative horticulture group in Upuvali and a women’s poultry group in Kinniya. As implementation progressed, the ET developed probing questions for use in the FGDs in order to gain insights into emerging trends, using an “evolving subject-driven” format, which refers to an iterative process that allows for an open conversation with stakeholders while adhering to a basic format so that information gathered across interviews can be aggregated and analyzed in a cohesive and consistent manner (see 6: Focus Group Discussions). 10 4. Site Visits: Over the course of conducting KIIS with farmers and FGDs, the ET also conducted site visits to farming sites in order to provide beneficiaries with an opportunity to demonstrate firsthand the factors that facilitated or hindered changes, such as water cisterns affected by drought and stands of new crops the activity promoted for evaluation. Site visits were selected to include both male and female farmers producing all of the crops evaluated. 5. Mini-Survey: Under the management of the Research Assistant, the ET implemented a mini-survey of 174 SOLID beneficiaries drawn from FGD participants (108, or 62 percent, were males and 66, or 38 percent, were female). The ET designed the mini-survey questionnaire to test the accuracy of project reporting and to support triangulation of data derived from KIIs and FGDs. As with FGDs, mini-survey implementation attempted to include a representative proportion of women and youth, defined as individuals aged 18 to 29 (see Annex 7: Mini-Survey). 11 The ET randomly selected specific beneficiaries from site lists provided by Chemonics that were disaggregated by crop, in line with the purposively-selected sample defined by the ET. Contacts were brokered by the staff at provincial Departments of Animal Health and Production and Departments of Agriculture (with the exception of the Youth Horticulture Initiative in Trincomalee, which was deliberately selected). All KIIs, FGDs, and mini-surveys included an informed consent statement, and all data were stored in secure files to protect the confidentiality of informants (see Annex 8: Data Collections Protocols). The ET designed the format of this report to omit specific identifiers. 10 Gary King, Robert Keohane, and Sydney Verba Designing Social Inquiry: Scientific Inference in Qualitative Research (Princeton University Press, 2016). 11 The National Youth Policy of Sri Lanka defines “youth” as between 18 and 29. See “National Youth Policy of Sri Lanka,” Ministry of Youth Affairs and Skills Development, 2014. TABLE 3: MINI-SURVEY PARTICIPANTS BY SUB￾GROUP Sub￾Group Participants Percent (%) Males 108 53% Females 66 32% Youth 30 15% Total 204 100% 6 USAID.GOV Final Performance Evaluation for SOLID Activity Notes from KIIs and FGDs were compiled and compared on a real-time basis to discern emerging trends and develop probing questions. Following the conclusion of data collection, the ET aggregated data obtained from the KIIs and FGDs around common themes related to the four EQs. For the quantitative mini-survey data, the ET input data electronically into an electronic database on a regular basis throughout data collection and analyzed frequencies of recurring responses to identify emerging trends. To the extent possible, the team also compared SOLID indicators and targets as a standard of achievement. Data analysis methods employed by the ET to produce findings and conclusions included: 1. Content Analysis – Content analysis entailed the ET’s intensive review of KII and FGD data to identify and highlight notable examples of factors that facilitated or hindered the project. 2. Trend Analysis – The ET used trend analysis to identify patterns of convergence (or divergence) that affected activity outputs and outcomes. 3. Gender and Youth Analysis – The ET worked across all SOLID EQs to capture and compare the results of the program as it specifically benefited (or did not benefit) women, men, and youth. All data collected through KIIs, FGDs, and mini-surveys were disaggregated by sex and youth in order to analyze effects on female and youth beneficiaries. While evaluating the performance of the SOLID Activity, the ET also documented the positive and negative impacts of external factors—such as changes in economic, political, environmental and financial conditions—which occurred over the length of the project. This data was obtained through the document review and in KIIs and FGDs. LIMITATIONS AND POTENTIAL BIASES There are several limitations and potential biases to the methods employed by the ET, which may affect the conclusions and recommendations reached by the ET. These include: Sampling – The ET purposively selected an illustrative, as opposed to a statistically significant, representative sub-set of beneficiaries for the FGDs and mini survey; this included at least two districts in each region evaluated (North, North Central, Eastern) and reflected the proportional distribution of overall project beneficiaries with regards to sector. To counter the limitations of this sampling methodology, the team attempted to employ reasonable assumptions as to the broader impact of factors identified as facilitating or hindering changes. In addition, time and resource constraints limited the ability of the ET to meet with beneficiaries in several of the horticultural sub-sectors addressed by SOLID, including sweet potato, jalapeño, black gram, and groundnut. Beneficiaries in these sub-sectors, as well as beneficiaries in micro-zones that feature unique soil or climatic conditions, and unusual marketing opportunities or disadvantages, are likely to have experienced discreet project outcomes. In limited cases, non-beneficiaries inadvertently attended FGDs, a fact only subsequently made clear to the ET (nonetheless, these individuals were limited in number). Indirect effects – The evaluation methodology and timeframe obliged the ET to focus on direct project beneficiaries. There may be unrepresented direct or indirect beneficiaries, especially including farmers that adopted techniques by observation. 7 USAID.GOV Final Performance Evaluation for SOLID Activity Subjective interpretation – The professional and technical opinions and experience of the ET may have limited their ability to analyze certain data, identify certain factors that influenced changes, or recognize lessons learned and best practices, such as familiarity with the procurement modality, certain GoSL methodologies or approaches, and climactic impacts. To the degree possible, the ET consulted with outside experts in cases where they recognized such limitations may have been present. Selection bias – Due to time and resource limitations, the ET was dependent on existing beneficiary lists provided by Chemonics and the staff at the provincial Department of Animal Health and Production and the Department of Agriculture, which facilitated contact with beneficiaries. This process may have resulted in a bias toward contact with beneficiaries that experienced more positive project outcomes, especially low drop-out rates. To counter this, the ET employed reasonable assumptions to the degree possible and probed for examples of drop-outs known to the beneficiaries and key informants interviewed. Positive response bias – Probing questions regarding income, growth, and outcomes during FGDs may have resulted in a bias toward positive responses. To the degree possible, the ET probed for quantitative data to verify unusual claims. FINDINGS AND CONCLUSIONS The evaluation team conducted KIIs with key stakeholders, including government representatives, USAID, and IP staff, to understand the origins of the SOLID project and the environment in which it started. This detailed summary provides the reader with findings and clarity by which to understand the development of the project, project dynamics and reporting and is included as an annex to this evaluation report (see Annex 9: Project Background). EVALUATION QUESTION ONE - FINDINGS EQ1: To what extent did the intended outcomes in agriculture, household income, women and youth’s engagement, household nutritional intake and access to productive economic assets occur over the course of SOLID? Did outcomes differ by region, gender of beneficiary, or sector (dairy, poultry and horticulture)? IMPROVED AGRICULTURAL OUTPUTS The SOLID activity proposes facilitating a transition from subsistence to commercial production through the introduction of new techniques and technologies that lead to improved agricultural outputs. Improvement in output entails adopting commercial practices to either increase production volumes to reach commercially viable scales (in the case of dairy and poultry, as well as some horticulture crops) or adopting new commercial crops (as was the case for most of the horticulture crops). Document review indicates that both project-assisted dairy farmers and poultry farmers have experienced improved agricultural outputs through increased production. For example, “milk producers who have worked at least one year with SOLID have increased monthly milk production by 27 percent, from 404 liters to 8 USAID.GOV Final Performance Evaluation for SOLID Activity “Rice was our most important crop in the past, but adopting silage means that we are now more interested in milk.” FGD respondent, Thalgaswewa 513 liters,” representing an increase of approximately 24 percent (over the course of the year following the adoption of silage). 12 Likewise, project reporting indicates that most beneficiary poultry farmers transitioned from small-scale “backyard operations,” typically consisting of between 20-50 birds, to commercial-scale production. For example, it was reported that “during the first phase of the program, 80 farmers from the Batticaloa District completed the five-module training course, built a 100-bird capacity broiler coop to specifications provided by SOLID, and successfully passed a final examination.” 13 Analysis of mini-survey responses supports project-reporting data in terms of expansion of livestock activities. Dairy farmers reported expanding the size of their herds from, on average, three to seven cows (or 133 percent), while poultry farmers report even larger expansions over the life of the project. Female respondents reported a relatively similar percentage expansion to males in terms of herd size; however, gross increases were typically smaller due to initially holding fewer cattle (3.6 to 8.2 cows, or 127 percent for males, and 2.8 to 5.9 cows, or 110 percent for females). In the poultry sector, respondents reported an average increase of 537 percent, from 37 to 237 broilers. Females initially held some poultry, as opposed to males who held almost none. While males adopted poultry as a result of SOLID activities, their post-project flock sizes were lower than those of their female counterparts, who expanded flocks nearly seven times on average (25 to 150 broilers, or 500 percent for males, and 50 to 325 broilers, or 550 percent for females). One possible explanation for this is that poultry activities were implemented in primarily Muslim communities where poultry is largely female-dominated, because poultry farming can be practiced within the household grounds. In FGDs, respondents engaged in both the dairy and poultry sectors cited the effectiveness of new technology introduced by the project as the catalyst for expanding the number of animals they held. Dairy farmers specifically cited the dramatic impact of adopting high-protein sorghum fodder (silage) on milk production as the primary reason that they expanded their operations. In many cases, participants in FGDs expressed astonishment at the resulting increase in per head milk production. One FGD respondent from Vattakachchi said, “I had no idea that changing the feed (of my cows) could make such a difference. I was astounded. This is a simple technique, but has a dramatic impact.” Likewise, poultry farmers cited improved chicken coops as the catalyst to expanding poultry operations, stating that the opportunity to increase flock size, which this permitted, facilitated access to new commercial markets. In both cases, numerous FGD participants stated that the livestock component of their farming system had overtaken paddy production as their primary occupation and income source due to expansions of these activities. In a significant number of cases, FGD participants reported accessing new finance to expand animal holdings in both the dairy and poultry sectors. However, there were significant differences between males and females in accessing finance, with much larger numbers of males accessing finance for cattle, while finance to increase poultry holdings was accessed only by females. Forty-one percent of dairy 12 SOLID Annual Report: October 2015 – September 2016,” Chemonics, November 15, 2016. 13 Ibid 9 USAID.GOV Final Performance Evaluation for SOLID Activity Source: SOLID PE Mini-Survey. farmers surveyed reported accessing finance for dairy activities over the course of the SOLID project (42 percent for males and 41 percent for females). In contrast, only one of all poultry farmers surveyed (10 percent, n=10), a female farmer, reported accessing finance for poultry activities over the course of the SOLID project, representing 17 percent of female poultry farmers surveyed. Most farmers reported that expenses for material required for expanded chicken coops were within their means, while the sole (female) farmer undertook a significant expansion, saving for which would have delayed her expansion plans. In the horticulture sector, “during the two-year project implementation period, SOLID supported 20 horticultural activities.”14 Specific horticultural products included a wide range of crops, all of which were either new or expanded to commercial scale, and all of which have a commercial market (see Annex 10: SOLID Activities by Province). In some cases, horticultural activities replaced existing crops with higher value alternatives. For example, KIIs and FGDs with passion fruit farmers indicted that irrigated passion fruit largely replaced watermelon in the North Central Province. Other crops, such as mushrooms, were additional new high-value crops, which farmers integrated into existing farming systems. In some cases, the production cycle of new crops contributed to improvement in outputs by increasing volumes comparable to existing crops. For example, irrigated passion fruit can be produced year-round in three-month cycles while watermelon can only be produced in a single annual cycle. Although less relevant as an indicator of growth due to the intensive, as opposed to extensive, 15 nature of most of the horticultural crops addressed by the SOLID project, mini-survey respondents reported expanding their acreage 100 percent on average, from 0.3 acres to 0.6 over the course of the project. Acreage expansion by female farmers was far larger than the acreage expansion of their male counterparts (60 percent, or 0.5 to 0.8 acres for males, and 0.1 to 0.4 acres, or 300 percent for females). In contrast, male farmers accessed new finance at over twice the rate as female farmers in order to expand horticultural production. Male farmers explained that, as the titleholders to collateral assets, finance institutions generally require that they themselves apply for loans. Overall, 22 percent of horticulture farmers surveyed reported accessing finance for horticulture activities over the course of the SOLID project (31 percent for males and 11 percent for females). In FGDs, respondents cited capital assets required for the set-up of new crops, such as stake and wires for passion fruit vines, as well as new irrigation equipment, as the most common investments. As such, despite being financed by males, in KIIs with private sector and FGDs with horticulture farmers, respondents described that in some cases these assets may pass to females, as women take over the management of new crops in the horticultural component of family farming systems. 14 Ibid 15 i.e., the increased value of crops versus an increased production through expansion of acreage. 10 USAID.GOV Final Performance Evaluation for SOLID Activity In response to a probing question regarding the high level of finance, farmers in FGDs indicated that access to new markets was one of the primary reasons that they had invested in the expansion of farming systems. In a KII in North Central Province, one farmer stated that the new buyer had provided significant guidance in production practices providing him confidence to transition large areas of his farm from watermelon to passion fruit. In FGDs in the Northern Province, dairy farmers described their relationship with Nestlé buyers as “very close.” A notable exception was morunga farmers at Sannasiparanthan in the Vavuniya District of Northern Province. These farmers explained that they had received seeds for a domestic variety of morunga, apparently due to a project oversight, which had a lower market demand than “red” Indian varieties they had previously grown. As a result, they stated that they were removing the new trees and replacing them with the former variety. Other localized problems included access to seeds for sugar sorghum, which farmers in Killinochchi indicated were not available in their region. Farmers in several areas of North and North Central Provinces stated that elephant invasions had damaged crops as well. The analysis of mini-surveys data, as well as FGD responses, does not suggest significant regional variation with regard to improvements in agricultural output. One notable though minor exception occurred in Mannar, where horticulture farmers reported that their big onion production had failed. KIIs with GoSL provincial staff at the Department of Agriculture cited that saline soils on the island regions of this district rendered the area unsuitable for big onion production. However, severe ongoing droughts across all regions evaluated (Northern, North Central, and Eastern) resulted in the failure of the 2016 Yala and Maha seasons and the 2017 Yala season. These events lowered water tables in all three regions, reducing water for irrigation, damaging horticultural crops and reducing water available for silage. In Manna, flooding inundated farmland, destroying crops. 16 Due to time and resource constraints, the ET was unable to visit producers of all horticultural crops supported by SOLID. However, a reasonable assumption is that drought impact was covariate with drought vulnerability across all regions and hence did not impact crops such as mushroom. On the other hand, irrigated crops fared poorly. FGDs with red onion producers in Jaffna and KIIs with passion fruit farmers throughout North Central Province indicated that these crops were particularly hard hit. In general, farmers reported that the drought had reduced or undermined horticultural production, but that previous crops would have been similarly impacted and that they intended to continue to produce the new crops once weather conditions improved. HOUSEHOLD INCOME SOLID set a 10 percent target in terms of percentage change in household income (IR1: Percentage change in the income of beneficiary households due to SOLID Interventions). 17 Reporting data claims significant improvements in household income as a result of activities in all sectors. For example, “an average crop farmer working with SOLID receives approximately 30 16 “Drought, Floods Slash Sri Lanka's Rice Production, Threaten Food Security - U.N.,” Reuters, June 22, 2014. 17 “SOLID Annual Report: October 2015 – September 2016,” Chemonics, November 15, 2016. % GRAPH 2: CHANGE IN INCOME BY SECTOR 0 10 20 30 40 50 Male Female Source: SOLID PE Mini-Survey. % 11 USAID.GOV Final Performance Evaluation for SOLID Activity Source: Source: SOLID PE Mini SOLID PE Mini-Survey percent more income, and earns an average seasonal farming income of Sri Lankan Rupees (LKR) 21,500.”18 Likewise, annual SOLID reports show the project exceeding end-of-project targets related to improved household income, notably the stated target in the SOLID Phase 2 M&E Plan under Indicator 2 (Percent change in income in dairy activities among beneficiary households).19 The SOLID Year Two Annual Report stated, “SOLID horticulture crop farmers saw an average monthly household income increase of approximately 32 percent from baseline to end-line. A farmer working with SOLID to produce these crops has on average been able to generate an average income of around LKR 21,500 per season from SOLID crops.” 20 As displayed in Graph 2, mini-survey data indicates that farmers experienced an increase in incomes across all sectors. According to the mini-survey, dairy farmers experienced the most significant average rise in income at 33.4 percent (28.6 percent for males and 38.1 percent for females), followed by poultry farmers at an average of 31.3 percent (25 percent for males and 37.5 percent for females), and finally horticulture farmers at an average of 28.7 percent (23 percent for males and 34.5 percent for females). Moreover, in FGDs, farmers expressed a high level of confidence in their income statements and had a good grasp of their associated costs of production and the profitability of activities due to active recordkeeping promoted by the project. In several cases, participants displayed their financial records or offered to retrieve them from their homes. A preliminary review of records showed that farmers captured costs correctly, and correctly calculated reconciliations to derive profit or loss. In the case of dairy farmers, milk collection invoices provided an extra level of recording that appeared to support their analyses, while poultry farmers and some horticultural farmers (for example, passion fruit) held receipts from buyers that verified their income calculations. In KIIs, IP staff explained that the promotion of financial recordkeeping comprised an integral element in the SOLID business skills training approach. For example, in the poultry sector, “Broiler Batch Ledgers” and “Coop Cards” provided convenient forms for record keeping.21 IP staff described these activities as essential to improving farmer analytical skills and, hence, the adoption of innovations. Background documents reveal that Sri Lanka has a significant unmet demand for milk and dairy products, although this demand has been declining in recent years. The country imports between 60 to 70 percent of its dairy consumption requirements from India, New Zealand and other countries22 . Companies such as CIC Holdings (CIC), MilkCo, and Nestlé source fresh milk from smallholder networks, making this an accessible market channel for commercially viable small-scale producers. In addition, these companies established floor prices, which were at times exceeded in order to ensure that producers generate a minimum level of income.23 Likewise, large-scale producers, such as Bairaha, CIC, Crysbro, and Prima, currently supply poultry meat to target communities.24 In KIIs, private sector staff stated that this leaves 18 Ibid 19 “Monitoring and Evaluation Plan for SOLID phase 2 (Dairy Development),” Chemonics. 20 “SOLID Annual Report: October 2015 – September 2016,” Chemonics, November 15, 2016. 21Dr. Sumith Atapattu, Andrew Baker, “SOLID Broiler Training Manual,” Chemonics. 22 See, for example “Sri Lanka: Dairy Sector Growth Opportunities,” FAO, 2010 and “Exporter Guide – Sri Lanka,” United States Department of Agriculture (USDA) Global Agricultural Information Network (GAIN), April 15, 2015. 23 “CIC Dairies, Milco in two-year deal to buy milk from Sri Lankan farmers,” Economy Next, September 7, 2017. 24 A.R. Wickramarachchi1, H. M. L. K. Herath, U. K. Jayasinghe-Mudalige, J. C. Edirisinghe1, J. M. M. Udugama1, L. D. M. N. Lokuge1 and W. Wijesuriya, “An Analysis of Price Behavior of Major Poultry Products in Sri Lanka,” Journal of 12 USAID.GOV Final Performance Evaluation for SOLID Activity unmet demand for fresh, locally produced poultry meat that can be met by smallholder broiler production. Although most agro-industry companies are generally reticent to become over-involved in upstream raw material production activities, a primary objective for these companies is securing supply chains. In KIIs with private sector staff, Nestlé managers stated that they are willing to engage dedicated field staff in providing technical assistance to their suppliers. In FGDs with dairy producers that supply Nestlé, farmers stated that they had frequently received such assistance. Farmers also stated that Nestlé was willing to assume the costs for maintenance of chilling stations. In KIIs, Nestlé staff stated that this will reduce the likelihood of equipment being idled by mechanical problems that users cannot afford to repair. In FGDs, all poultry farmers reported that transitioning from backyard to broiler production increased incomes and profitability. KIIs with GoSL staff indicated that the project introduced the concept of “minimum space,” i.e., that a technically sound minimum coop space is required to engage in commercial-scale broiler production, which facilitates the expansion of production to reach commercial channels that can effectively supply local markets. Markets for horticulture products comprise a broad spectrum, which KIIs with GoSL and private sector staff indicated includes the following: • Red onion, mushroom, and chili are sold into local markets. • Big onion is sold into local markets, replacing Indian imports (big onion is a new crop, planted on new land in Mannar District). • Chili seed is certified by the Departments of Agriculture and disseminated to other farmers in an effort to re-establish the chili sector. • Passion fruit is sold to Cargill’s or local markets in the case of market prices exceeding floor prices. In FGDs, a majority of horticulture farmers stated that markets for horticultural crops were more “formal” markets than they had previously supplied. Also, horticulture farmers agreed that, despite the negative impact of the current drought on production, these new market linkages provided higher remuneration than markets for previous products. In addition, these markets are accessible to individual farmers. Only an average of 3 percent (4 percent for males and 2 percent for females) of dairy farmer beneficiaries reported becoming a member of a farming organization (FO) over the course of the SOLID project, and only 1 percent (0 percent for males and 2 percent for females) of horticulture farmer beneficiaries joined FOs. On the other hand, 100 percent of poultry farmer beneficiaries in FGDs reported becoming a member of an FO over the course of the SOLID project. Dairy and horticultural producers indicated that access to new market linkages required increased organization due to the nature of supply chains operated by the private sector, which were willing to collect from individual farmers, while poultry farmers relied on their FO to achieve required scales. WOMEN’S ENGAGEMENT IN AGRICULTURE SOLID set a 20 percent target in terms of female project participation (Indicator 3: Proportion of female participants in USG-assisted programs designed to increase access to productive economic resources).25 25 “SOLID Annual Report: October 2015 – September 2016,” Chemonics, November 15, 2016. 13 USAID.GOV Final Performance Evaluation for SOLID Activity While a final annual report is still under review, reporting data indicates women were very engaged in the project; as noted, female participation at training or technical assistance demonstrations was between 40 to 45 percent, and female direct beneficiaries totaled 30 percent of all SOLID beneficiaries. According to mini-survey data as depicted in Graph 3, beneficiaries believed that women were as likely as men to benefit from project activities, with 80 percent of those surveyed (48 percent of males, 32 percent of females) strongly agreeing with this statement compared to just 2 percent overall that disagreed or strongly disagreed (Note: To enhance its legibility, this is not represented in the graph below). Disaggregated by sector, the beneficiaries surveyed believed that women were as likely as men to have benefitted from dairy and poultry activities, while horticulture activities were perceived as slightly less likely to benefit women as much as men. According to mini-survey data, overall income gains were higher for female beneficiaries surveyed (36.7 percent) compared to male beneficiaries (25.5 percent). It should be noted, however, that women’s incomes were lower than men’s at the start of SOLID. The difference in these income gains was largest in the poultry sector (37.5 percent as compared to 25 percent for males), followed by horticulture and, finally, dairy. Source: SOLID PE Mini-Survey Background research suggests that SOLID activities resulted in significant improvements in women’s engagement in agriculture and increased access to productive economic assets due to the activity’s focus on the intensive components of household farming systems, i.e., horticulture and poultry, which in traditional South and Southeast Asian farming systems are generally managed by women. This is opposed to the extensive component, i.e., staple grains, such as paddy. 26 In KIIs with GoSL staff, provincial Department of Animal Production and Health officials explained that even in the case of dairy farming, which is often traditionally managed by men due to integration into extensive staple production, the 26 See, for example, Bina Agarwal, “A Field of One's Own: Gender and Land Rights in South Asia,” Cambridge University Press, 1994, and Cheryl Doss, “The Role of Women in Agriculture,” Agricultural and Development Economics Division (ESA), Working Paper no. 11-02, Food and Agriculture Organization (FAO), March 2011. 0 10 20 30 40 50 60 70 Youth Adults Youth Adults Youth Adults Youth Adults Youth Adults Youth Adults MALE FEMALE MALE FEMALE MALE FEMALE Neither Agree Nor Disagree Agree Strongly Agree (#) Dairy Horticulture Poultry Total GRAPH 3: WOMEN WERE AS LIKELY AS MEN TO BENEFIT FROM SOLID ACTIVITIES 14 USAID.GOV Final Performance Evaluation for SOLID Activity GRAPH 4: SOLID ACTIVITY’S EFFECTIVENESS IN IMPROVING HOUSEHOLD NUTRITION 0 20 40 60 80 100 Disagree Neither Agree Nor Disagree Agree Strongly Agree Dairy Horticulture Poultry Total Source: SOLID PE Mini-Survey % intensification of cattle raising promoted by SOLID resulted in an increased role for females in managing cattle. In FGDs, female farmers indicated that increases in access to assets were largely tied to the expansion of production, i.e., increased herds (cattle), new sprinklers, (horticulture), and larger flocks (poultry). As a result, these female farmers indicated that they preferred activities that entailed such assets, which they perceived as increasing the share of household wealth they control. In addition, females stated a preference for crops for which they receive payment directly from buyers at or near their households (such as milk and passion fruit), which they view as advantageous given that income does not have to pass to them through males who deliver products to markets to receive payment. YOUTH ENGAGEMENT IN AGRICULTURE Analysis of mini-survey data showed that 60 percent of beneficiaries surveyed (73 percent of youth and 56 percent of adults) strongly agreed that youth were as likely as adults to benefit from SOLID activities, while none disagreed or strongly disagreed. In terms of sectors, 97 percent of beneficiaries surveyed agreed or strongly agreed that youth were as likely as adults to benefit from SOLID activities in the dairy sector, compared to 82 percent for horticulture, and 50 percent for poultry. In FGDs, farmers explained that poultry was not attractive to youth because it required their presence at home. In FGDs, participants expressed significant contradictions in their responses to questions around youth engagement in agriculture. For example, participants in a FGD in North Central Province stated that horticulture was not attractive to youth due to its perceived low status as a job, while members of the Youth Horticulture Initiative near Trincomalee described horticultural production as the most attractive sector for youth due to its “business orientation.” However, several general positive and negative factors related to youth engagement emerged. Positive factors included: (1) youth are attracted to activities that take up less time and (2) youth are attracted to activities that are more profitable and “business-like.” Negative factors included: (1) youth consider farming a “low status” occupation and (2) youth prefer to leave rural areas to find work, due to the improved social life, earning opportunities and better living standards they perceive in urban areas (a sentiment with which youth agreed). In response to the contradictions expressed, members of the Youth Horticulture Initiative offered the explanation that personal career preferences are highly subjective and subject to individual tastes. However, farming in an attractive region close to urban conveniences is likely to increase the attractiveness of agriculture as a career choice. Notably, participants in several FGDs with horticulture farmers stated that searching for technical resources online had served to engage youth in household farming systems by providing them with a role in which they could utilize specific skills. HOUSEHOLD NUTRITIONAL INTAKE According to the mini-survey, a majority of beneficiaries strongly agreed that SOLID activities were effective in helping them improve household nutrition (32 percent overall, compared to 2.3 percent overall that strongly disagreed). Poultry farmers felt strongest that the activities were effective in improving their household nutrition (100 percent agreed or strongly agreed) compared to 73 percent for 15 USAID.GOV Final Performance Evaluation for SOLID Activity dairy producers and 71 percent for horticulture producers. When horticulture and poultry farmers were asked in FGDs how SOLID activities had impacted their household nutrition, the most common response was that increased income helped improve their purchasing power for food and hence intake. However, in the case of dairy, farmers stated that the primary impact of the activities was that increased milk production led to increased home consumption. In contrast, virtually none of the farmers in any sector reported changing their nutritional behavior significantly, i.e., by increased food diversity, changes in dietary preferences and intake. According to farmers during site visits, SOLID’s nutrition activities in some areas appeared marginal. For example, in some areas, activities were limited to posters or booklet distribution. Likewise, in FGDs in Polanuruwa and Batticoloa, farmers indicated that, while they were grateful to receive a poster or booklet, they did not really understand how to use it. In KIIs with USAID, IP and Private Sector staff, respondents stated that nutrition activities were “half-hearted” or even “a distraction.” EVALUATION QUESTION ONE - CONCLUSIONS IMPROVED AGRICULTURAL OUTPUTS SOLID improved agricultural outputs in all three target regions evaluated and across genders in all three target sectors. This improvement was achieved by deploying effective techniques and technologies derived from commercial value chains, which facilitated increased production volumes in order to reach commercially viable scales or through facilitating the adoption of new commercial crops. Outcomes did not vary across the regions evaluated. However, climactic challenges—including drought across all three regions evaluated and torrential rain and flooding in Mannar—reduced the output of horticultural production. HOUSEHOLD INCOME SOLID significantly improved household income. These improvements resulted from improved outputs and demonstrated profitability, as well as demonstrable market linkages, in turn generating farmer investments into expansion of farming systems, resulting in significant improvements in household income. The promotion of farmer recordkeeping enhanced the adoption of SOLID skills and techniques and facilitated project reporting and evaluation. In addition, the focus on the intensive component of household farming systems led to higher income improvements for women than for men. WOMEN AND YOUTH ENGAGEMENT IN AGRICULTURE SOLID succeeded in engaging women in agriculture and in providing access to productive assets. These successes were largely due to the project’s selection of activities related to the intensive components of household farming systems. SOLID’s success in engaging youth in activities has been more ambiguous, however, when compared to the project’s gains made with women. While several general assumptions regarding youth engagement in agriculture can be deduced, youth engagement in agriculture is primarily subject to individual interests and preferences. These interests vary greatly and would have to be explored in more depth to understand if interventions of this nature are appropriate for this target audience. HOUSEHOLD NUTRITIONAL INTAKE SOLID successfully generated positive changes in household nutrition due to increased income generated by farming activities, as well as increased home consumption of milk. However, activities specifically related to nutritional behavior change were not sufficient to effect significant changes in behavior. 16 USAID.GOV Final Performance Evaluation for SOLID Activity EVALUATION QUESTION TWO - FINDINGS EQ2: What influence did SOLID have on the USG/GoSL relationship at the national and local levels? What does a productive relationship look like based on the SOLID experience? The initial design and start-up of the SOLID project took place against the backdrop of the administration of former Sri Lankan president Mahinda Rajapaksa (2005-15). Strained relations with the USG characterized the Rajapaksa administration, stemming from disputes over alleged human rights abuses that occurred during the final stages of the Sri Lankan civil war and from calls for a war crimes inquiry. 27 Furthermore, a prevailing high level of mistrust of non-governmental organization (NGO) activity placed significant restrictions on humanitarian and development projects. 28 KIIs with IP and USAID staff indicated that, even subsequent to project start-up, the SOLID project faced significant challenges in coordination with, and approval from, GoSL officials. In KIIs, IP staff described significant restrictions on operations, including checkpoints and limited entry into Northern Sri Lanka, strong pressure on donors to provide direct support to government agencies as opposed to working through IPs, and considerable official input into beneficiary selection, which threatened to undermine the integrity of project implementation. SOLID was initially assigned to the Ministry of Economic Development (MED) Project Monitoring and Management Unit (PMMU). In KIIs, USAID staff described facing a high level of pressure during the project planning stage to provide resources directly to the MED, which implemented the official GoSL Divi Naguma poverty reduction program, and to recruit unviable beneficiaries from its lists. In KIIs, USAID and IP staff stated that, following approval of the Chemonics SOW and project start-up, SOLID staff was able to coordinate with the provincial Departments of Agriculture and Departments of Animal Production and Health, ultimately undertaking a baseline study in 2015. However, project implementation continued to be substantially delayed due to slow approvals by the GoSL, the restrictions on movements, and continued pressure to transfer resources to government entities. Following the election of Maithripala Sirisena and the change of administration on January 9, 2015, the MED was abolished as a line ministry, and SOLID was shifted to the Ministry of National Policy Planning, Department of Economic Affairs, Child, Youth and Cultural Affairs. This move severed SOLID from the PMMU and any subsequent project reporting. Thus, PMMU is unaware of project achievements or resources. On the other hand, in KIIs, IP staff described the change in administration as ushering in a facilitating 27 See, for example, “WikiLeaks: Mahinda Rajapaksa 'Responsible for War Crimes’,” The Telegraph, November 2010, and Nilan Fernando, “Sri Lanka’s War Crimes Controversy,” New York Times, September 27, 2015. 28 See, for example, “More Warnings On NGO Subversion,” Daily News, Aug 20 2014. “Initially, the Ministry of Economic Development exerted pressure on us to include beneficiaries from the (official) Divi Neguma project, who did not meet SOLID requirements as viable producers.” KII, USAID staff “When you arrived at certain points in the north, vehicles were stopped, and often were not allowed through. We weren’t even able to visit some of the project sites at all.” KII, IP staff “When you arrived at certain points in the north, vehicles were stopped, and often were not allowed through. We weren’t even able to visit some of the project sites at all.” KII, IP staff “When you arrived at certain points in the North, vehicles were stopped, and often were not allowed through. We weren’t even able to visit some of the project sites at all.” KII, IP staff 17 USAID.GOV Final Performance Evaluation for SOLID Activity environment for operations, including lifting restrictions on movement, easing requests for direct support, and terminating interference in beneficiary selection. As a result, SOLID was able to implement planned activities with GoSL support. In KIIs with GoSL staff, officials at the Ministry of National Policy Planning indicated that SOLID activities complemented GoSL priorities in the agriculture and dairy sectors. Likewise, in KIIs with district Departments of Agriculture and Departments of Animal Production and Health staff, officials highlighted several activities to which SOLID contributed. Areas of collaboration and support cited by these officials include the following: • GoSL participation in the initial livelihoods assessment was limited to MED staff though they do not appear to have been highly represented at initial crop selection workshops based on participant lists in the SOLID annual report. However, Ministry of National Policy Planning officials indicated that SOLID identified several specific sectors and crops that aligned with GoSL policy priorities. For example, small onion and chili seed production responds to the country’s prevailing seed shortage and also promotes domestic production (both crops are currently imported from India). Likewise, increased domestic milk production reduces the country’s dependence on dairy imports. • SOLID coordinated with Ministry of Agriculture staff in identifying suitable agro-ecological zones for several horticulture crops. • SOLID assisted the Department of Agriculture and Department of Animal Production and Health to disseminate new varieties of seeds for several priority crops, including high-yielding chilis, and high-protein dairy fodder for silage production. SOLID also supported production of these crops by disseminating inputs (seeds) and productive assets such as sprinklers and pump motors. • SOLID helped introduce to farmers new activities that corresponded with Ministry of Agriculture objectives, particularly off-season big onion production and improved chili seed production, notably in Kahalla. • In several cases, SOLID engaged the Department of Agriculture and Department of Animal Production and Health staff as trainers, including in dairy activities. • SOLID trained an Agriculture Instructor from the Department of Agrarian Services on how to best use pesticides. To answer the question, “What does a productive relationship look like based on the SOLID experience?”, KIIs with GoSL staff produced limited input beyond saying that there is a need to better coordinate objectives and integrate resources. However, the responses also revealed a high level of interest in “assimilating” resources and effective approaches. Specifically, staff at the Ministry of National Policy Planning expressed an interest in adopting successful SOLID methodologies and associated training materials. Staff suggested that these resources would support the legitimacy of the agency. Likewise, at each agency interviewed, staff made a case as to why their particular entity was best placed to disseminate these resources. Based on reported SOLID results, GoSL staff at the Ministry of National Policy Planning stated that the agency has already allocated budgetary resources for the continuation of some activities introduced by the project. Specifically, this included a program to provide grants for 50 percent of the cost of grass choppers for silage, as well as for sprinklers for seed producers. 18 USAID.GOV Final Performance Evaluation for SOLID Activity On a related topic, in several FGDs with dairy farmers in the Northern Province, participants suggested that some public government services that could help enhance SOLID activities were not adequately available to beneficiaries. This especially included veterinary services; farmers cited artificial insemination and vaccinations (although these services are available to beneficiaries through private providers at cost). Follow-up with the Department of Animal Production and Health staff revealed that this problem exists across the country. EVALUATION QUESTION TWO - CONCLUSIONS What influence did SOLID have on the USG/GoSL relationship at the national and local levels? The relationship between SOLID and the GoSL was characterized more by coordination and support, with limited influence on relevant GoSL activities and approaches. This is unsurprising given the context of USG/GoSL relations prior to 2015. While the change in administration did result in a more collaborative relationship, this important change did not occur until the project’s midpoint, potentially limiting any gains. The dissolution of the MED following the change in administrations and the resulting severance of SOLID from the PMMU also resulted in a lack of official reporting at the national level. This made it difficult for some Ministries to grasp the project’s achievements and potential uses, potentially limiting SOLID’s influence even further. What does a productive relationship look like based on the SOLID experience? A productive relationship would entail understanding the goals and capacities of the GoSL’s relevant Ministries and identifying entry points for GoSL assistance and support (by that Ministry) at national and local levels. GoSL interest in assimilating resources and contributing to project impacts is likely to have a positive effect on spreading the impact of these resources. This also shows that the government is willing to coordinate objectives and resources to achieve a shared goal. EVALUATION QUESTION THREE - FINDINGS EQ3: Identify planned and unexpected results in relation to ensuring potential sustainability and effect on the local agricultural value chain through partnering with: OTHER USG PROJECTS/ACTIVITIES A review of USAID activities in Sri Lanka reveals that previous USAID-funded projects and activities in Sri Lanka include the BIZ+ project implemented by Land O’Lakes and VEGA. This project aimed to generate income, expand investment opportunities, and improve livelihoods, primarily through the establishment of private sector partnerships, as well as through the Reintegration and Stabilization in the East and North (RISEN) project that was implemented by Development Alternatives Inc. RISEN intended to improve personal and economic security in targeted communities in the East, providing support and opportunities for at-risk youth and building public confidence that core conflict issues are being addressed by the GoSL. In addition, USAID funded the Sustainable Intervention in Livelihoods in Kilinochchi, implemented by the Sri Lanka Center for Development Facilitation to promote dairy development and sustainable agriculture. Document review of the SOLID Year 1 Work Plan notes several areas where the project initially planned to coordinate with other USG-funded projects and activities (e.g., “USAID’s Biz+ project is supporting the dairy sector by providing grants to small dairy processors in locations where SOLID will work). SOLID has already taken steps to build on the 19 USAID.GOV Final Performance Evaluation for SOLID Activity institutional knowledge gained from previous programs, including hiring staff from USAID’s recently closed flagship project, RISEN.” 29 In practice, there appears to have been only very limited partnership with these programs. The only citation of partnership appears to be SOLID collaboration with BIZ+ to jointly hold a District Dairy Day in Thalawa, Anuradhapura District on September 8, 2016, described in the SOLID Year 2 Annual Report.30 KIIs with IP staff indicated that this was a result of poor overlap between project approaches and objectives (especially humanitarian assistance versus development), and a lack of “commercial orientation” in the case of development projects. In some cases, they suggested that the specificity of value chain requirements simply resulted in projects being incompatible with SOLID objectives. Likewise, dairy farmer FGD participants in all three evaluated regions claimed no experience with prior projects in the dairy sector, and KIIs with IP staff indicated there was no intentional beneficiary overlap. On the other hand, USAID staff indicated that they anticipate that SOLID will inform a United States Department of Agriculture (USDA) Food for Progress project related to dairy and soybean value chain development anticipated to start operations in Sri Lanka in fiscal year FY 2018. OTHER DONORS SOLID project documents note that, despite widespread donor interest in the country, the focus of most programs at the time of SOLID’s start was on humanitarian assistance to address post-conflict and tsunami-related reconstruction: “while donor presence in Sri Lanka has been strong over the past few years, many projects are now shifting away from humanitarian assistance and toward activities that lead to longer-term economic development.”31 However, these documents also suggest that this presence had developed a cadre of capable local partners that could assist SOLID implementation in the field, noting “the influx of donor assistance following conflict and natural disaster helped build an array of capable local implementing partners.” 32 Likewise, the initial livelihoods assessment, as well as initial project design documents and the Year 1 Work Plan, envisions working extensively through local partners to implement activities and further build local capacity. In addition, SOLID staff initially anticipated working relations with a wide variety of donors, including the Food and Agriculture Organization (FAO), the United Nations Office for the Coordination of Humanitarian Affairs, Japan International Cooperation Agency, AusAID, and the United Methodist Committee on Relief. However, in KIIs, IP staff stated that, in practice, SOLID encountered difficulties in identifying sufficiently skilled local IPs; thus, this approach was significantly scaled back during the first year of project implementation. Likewise, partnerships with other donors was limited. SOLID reporting data indicates only that FAO “made a request to reproduce the SOLID Nutrition Handbook with appropriate credit given to USAID. Accordingly, FAO printed and distributed 1,500 manuals among pre￾school teachers as part of improving nutrition among preschoolers.” 33 In KIIs, IP and USAID staff stated 29 “SOLID First Year Work Plan (with General Life-of-Project Work Plan) – Scope of Work from November 14, 2013 through November 13, 2016,” Chemonics, June 2014. 30 “SOLID Annual Report: October 2015 – September 2016,” Chemonics, November 15, 2016. 31 “SOLID First Year Work Plan (with General Life-of-Project Work Plan) – Scope of Work from November 14, 2013 through November 13, 2016,” Chemonics, June 2014. 32 Ibid 33 Ibid 20 USAID.GOV Final Performance Evaluation for SOLID Activity that the difficult operating environment prevailing during the initial year of the project meant that few donor programs compatible with SOLID were in progress; hence, effective partnership remained limited. On the other hand, in FGDs with horticulture farmers in the Northern Province, approximately 20 percent of beneficiaries stated that they had participated in previous FAO projects. However, specific crops differed from SOLID-promoted crops, and the focus was on food security, as opposed to commercial farming; as a result, there appears to have been little overlap. GOVERNMENT OF SRI LANKA As noted, the SOLID Year 1 Work Plan anticipated coordinating with the GoSL Divi Neguma program, which provides a range of support to small-holder farmers, including input supply and veterinary services, as do district-level programs. However, subsequent annual reports do not provide information on this topic. In KIIs, IP staff explained that Divi Neguma beneficiaries did not meet the criteria to participate in SOLID and were therefore not included in an intentional manner. In contrast, and as noted in a number of cases, SOLID collaborated with the Department of Agriculture and the Department of Animal Production and Health or contributed to some objectives with regard to improving domestic production of certain products on which Sri Lanka is dependent for imports (see Evaluation Question Two). In KIIs, GoSL and IP staff support for the introduction of silage to increase milk yield were likely to advance GoSL agency activities in this area, thereby expanding the effect of technology on the milk value chain. Likewise, SOLID support for onion and chili seed production will likely facilitate expanding GoSL activities around increasing available seeds for these target crops. INTERNATIONAL, NATIONAL AND LOCAL PRIVATE SECTOR FIRMS As noted in FGDs, dairy and horticulture farmers (especially passion fruit farmers) stated that they perceive a strong commitment on the part of private sector buyers to continue support for production and to preserve market linkages (see Evaluation Question One). In KIIs, Nestlé staff in particular expressed a commitment to investing in the maintenance of milk chilling units provided by SOLID and to continuing to promote and provide technical assistance to farmers on the use of high-protein sorghum fodder to improve milk production. EVALUATION QUESTION THREE - CONCLUSIONS OTHER USG PROJECTS/ACTIVITIES AND DONORS Although SOLID may inform a USDA FFP project, it is unclear if SOLID will influence the sustainability of local agricultural value chains in the long run. This is not entirely unexpected since the project had limited partnerships with previous USG projects due to varying objectives, and limited partnership with other donor projects and activities. It appears that the projects working in this space were focused on achieving their discrete project objectives and were not designed to work with other donors or programs in a deliberate or consistent manner, given the specificity of the projects’ objectives and goals. GOVERNMENT OF SRI LANKA SOLID supported some GoSL objectives which led to wider value chain impact. For example, SOLID support for the Department of Agriculture and Department of Animal Production and Health objectives in the dairy (silage) and horticulture (onion and chili seed) sectors is likely to broaden the effect on agricultural value chains through wider dissemination of these techniques and technologies to farmers. 21 USAID.GOV Final Performance Evaluation for SOLID Activity INTERNATIONAL, NATIONAL AND LOCAL PRIVATE SECTOR FIRMS SOLID’s partnership with the private sector significantly contributed to project sustainability. The private sector’s intentions to continue to support SOLID approaches and resources is likely to increase sustainability and to intensify the effect of SOLID activities on the dairy and horticultural value chains. Notably, the partnership with Nestlé is especially likely to foster sustainability and impact as the company continues to support maintenance of milk chilling equipment and silage. Likewise, the partnership with Cargill’s in horticulture (passion fruit) commercialization resulted in the improved sustainability of activities in this sector. EVALUATION QUESTION FOUR - FINDINGS EQ4: Are past beneficiaries currently implementing the skills and techniques promoted by SOLID? Did farmers involved in demonstration activities adopt the recommended practices? As illustrated in Graph 5, mini￾survey data show that 85.1 percent of beneficiaries surveyed (53.4 percent of males and 31.6 percent of females) strongly agreed or agreed that they are likely to continue implementing the techniques introduced by the SOLID project, compared to only 1.1 percent that either disagreed or strongly disagreed (0.7 percent of males and 0.4 percent of females). Ninety-two percent of respondents in the dairy and 100 percent of respondents in the poultry sector agreed or strongly agreed that they are likely to continue to implement the skills and techniques promoted by SOLID. Farmers identified the effectiveness in improving profitability as the key reason for continuing to implement skills and techniques in these sectors. Seventy-five percent of respondents in horticulture agreed or strongly agreed that they are likely to continue to implement the skills and techniques promoted by SOLID. Farmers cited climatic challenges (drought) as the key reason for discontinuing implementation of the skills and techniques promoted by SOLID. Likewise, in FGDs and KIIs with farmers, the vast majority of beneficiaries stated that they are currently implementing the skills and techniques promoted by SOLID and that farmers involved in demonstration activities have adopted recommended practices, with no discernable regional variation. With one exception, the only agriculture-related reason cited for not continuing to practice the skills and techniques promoted by SOLID was related to the adverse effects of weather. However, the vast majority of these farmers also indicated they were merely suspending activities until the return of more favorable weather conditions, i.e., the end of the current drought. Reasons not related to agriculture included plans to relocate or terminate farming. In most cases where farmers indicated they had suspended horticulture activities due to weather, they indicated they had already incurred start-up costs 0 5 10 15 20 25 30 Male Female Male Female Male Female Male Female Disagree Neither Agree Nor Disagree Agree Strongly Agree Dairy Horticulture Poultry GRAPH 5: LIKELINESS TO CONTINUE IMPLEMENTING TECHNIQUES LEARNED THROUGH SOLID Source: SOLID PE Mini-Survey % 22 USAID.GOV Final Performance Evaluation for SOLID Activity GRAPH 6: SOLID PRACTICES WERE EASY TO IMPLEMENT Source: SOLID PE Mini-Survey 0 5 10 15 20 25 30 Male Female Male Female Male Female Male Female Disagree Neither Agree Nor Disagree Agree Strongly Agree Dairy Horticulture Poultry % (e.g., stakes and lines for passion fruit) and so it made sense to continue when weather was favorable. Morunga tree farmers in Vavuniya, Northern Province, provided the sole exception to weather being an agriculture-related reason for not continuing to practice SOLID skills and techniques. As noted, these farmers are removing their trees since they were provided with seeds for a variety that was inconsistent with the prevailing market demand. In addition to the effectiveness of SOLID practices in improving incomes, one factor that may have supported the adoption of recommended practices may be ease of implementation. Seventy-three percent of surveyed beneficiaries strongly agreed or agreed that the practices introduced by the project were easy to implement, while only 2.3 percent either disagreed or strongly disagreed with this (24.7% neither agreed nor disagreed). Seventy-six percent of respondents in the dairy sector, 34 percent of respondents in horticulture and 100 percent of respondents in the poultry sector agreed or strongly agreed that practices promoted by the SOLID project were easy to implement. Graph 6 depicts the number of mini-survey respondents who found SOLID practices easy to implement. Further highlighting the attractiveness of SOLID activities, anecdotal experiences during data collection suggested that non-beneficiary farmers (e.g., neighbors, community members) adopted SOLID dairy techniques (such as silage), inspired by observing the enhanced performance of dairy sorghum-fed cattle. During several FGDs, non-beneficiary farmers attended due to their interest in dairy activities, and numerous FGD participants in all districts where the ET met with dairy farmers described teaching other community members the techniques they had learned through SOLID. This observation effect was particularly evident in Pandatharipu, in the Jaffna District of the Northern Province, where Sri Lankan president Maithripala Sirisena had visited a dairy beneficiary to present an award for excellence in dairy farming, provoking widespread interest among non-beneficiaries throughout the community. In a related comment, in FGDs and KIIs in Eastern and North Central Provinces, a number of horticulture farmers stated that they feel they have few information sources on new skills and techniques. Therefore, projects such as SOLID that introduce effective techniques are highly valued. When asked about access to online information about agricultural techniques, these farmers also explained that youth are more likely to utilize the Internet to access technical information, and often do so on behalf of their parents, who tend to have low Internet literacy and anticipate challenges in accessing relevant information in languages they can understand. According to one female adult horticulture farmer in the North Central Province, “It’s not that they don’t know about the Internet, it’s that they don’t speak English.” Most technical resources online are in English. Despite this perception, a web search reveals a great deal of relevant information available online in both Sinhala and Tamil, including YouTube videos on production techniques for most of the crops targeted by the SOLID project. 23 USAID.GOV Final Performance Evaluation for SOLID Activity EVALUATION QUESTION FOUR - CONCLUSIONS SOLID succeeded in promoting new skills and techniques to farmers, and most farmers involved in the demonstration activities have adopted recommended practices. Even in cases (primarily in horticulture) where farmers indicated that they were not currently implementing the skills and techniques promoted by SOLID, farmers indicated that activities were only suspended until more favorable weather conditions warranted resumption. Interestingly, non-beneficiary farmers also reported adopting SOLID techniques, partially due to the GoSL and the President’s recognition of a dairy beneficiary. It is plausible, then, that GoSL promotion of SOLID’s techniques could promote uptake even further in certain areas. Some older farmers in the Eastern and North Central Provinces perceived challenges in accessing information on new techniques and technologies that can improve their farming systems. This was largely due to limited or no experience accessing the Internet and difficulty accessing information in local languages. However, young people can act as intermediaries, connecting older household members to the available wealth of online information. RECOMMENDATIONS Recommendation Relevant EQ(s) Estimated Time Frame Livelihoods developent is a multi-disciplenary effort, which relies on sector-specific specialized expertise in various technical areas. In planning future livelihoods activites in Sri Lanka and other countries, implementers should adopt a two-step approach to livelihoods assessment to inform project design: 1. Identify strategic technical areas of focus during the initial design phase (e.g., agriculture, access to finance, small and medium enterprises) in order to identify relevant sectors to the context (e.g., post-conflict reconstruction, disaster recovery) which are most likely to have the greatest impact on target beneficiary groups 2. Subsequently, apply specialized expertize to analyze specific activities, sourced from technical specialists with commerical experinece in targeted sectors (e.g., vocational training, agriculture, livestock, finance, health). In addition, in the case of value chain development, in order to identify effective techniques and technologies, implementors should assess the prevailing level of development of target value chains vis-à-vis more developed comparable commercial value chains (for example, in similar or relevent countries) and focus on the adoption of commercial practices and technologies drawn from these examples (for example, silage). EQ 1-4 On-going In agricultural development programs in Sri Lanka and other countries, both USAID and implementing partners should include record keeping (e.g., crop budgets: cost of production forecasts and reconciliation) as an integral activity to (1) demonstrate profitability of EQ 1, 3 and 4 On-going 24 USAID.GOV Final Performance Evaluation for SOLID Activity Recommendation Relevant EQ(s) Estimated Time Frame new technologies and inputs, (2) develop analytical skills of producers to evaluate new technologies and inputs and promote innovation, and (3) facilitate project M&E data collection through the agregation of farmer records. In line with best practices, crop budgets should detail costs for all inputs, including capitalization of land and labor costs (including family labor) in order to evaluate opportunity costs. Crops budgets should also include both planning and reconciliation tables in order to evaluate deviances from planned budgets. (See Figure 1: Illustrative Crop Budget, on the following page.) Given prevailing gender roles regarding the management of the various components of family farming systems in the region, USAID and implementing partners in Sri Lanka and other similar countries in South and Southeast Asia should focus on intensive (as opposed to extensive) components of household farming systems in order to promote women’s engagement in agriculture. These intensive components usually include horticulture and poultry, as opposed to staple grain production. EQ 1 On-going Given Nestlé’s established track record of active support for USG and other donor development initiatives in various value chains, USAID should explore increasing partnership with Nestlé as a potential development alliance partner, both in Sri Lanka and in other countries where the company is active. EQ 1-3 Medium-term USAID and IPs should include a web presence (e.g., tutorial videos and technical resources) in future projects in Sri Lanka to (1) engage youth in activities to facilitate transfer of knowledge to older farmers who may not feel adequately skilled to access online information that exists and (2) increase the scale of impact of similar projects on selected value chains. USAID should take care not to duplicate information already available online in Sinhala and Tamil. EQ 1-4 Medium-term Given GoSL interest in “assimilating” resources and responsibility for positive impact, implementing partners and the GoSL should improve coordination to increase the positive impact of activities on value chains. Examples of improved coordination could include the following: greater coordination in conducting assessments, selecting target sectors for program implementation, and developing methodologies and exchanges, as well as increasing engagement of GoSL staff in activities (e.g., as trainers through TOT and through secondment of project staff). EQ 2-3 Short-term 25 USAID.GOV Final Performance Evaluation for SOLID Activity Source: Absolute Options LLC CROP BUDGET Name: Male Female (circle one) Location: Crop: Planting Date: Reconciliation Date: EXPENSES Unit Cost per Unit Number of Units Total Cost per Unit Number of Units Total Land Labor Seeds: Fertilizer 1: Fertilizer 2: Fertilizer 3: Tools 1: Tools 2: Tools 3: Other: Other: Other: TOTAL EXPENSES INCOME Sales 1: Sales 2: Sales 3: TOTAL INCOME PROFIT/LOSS FIGURE 1: ILLUSTRATIVE CROP BUDGET 26 USAID.GOV Final Performance Evaluation for SOLID Activity ANNEX 1: SCOPE OF WORK SECTION C - STATEMENT OF WORK Title: Final Performance Evaluation of Supporting Opportunities for Livelihoods Development (SOLID) Activity C.1 PURPOSE OF THE EVALUATION SOLID is a pilot activity that works in partnership with the Government of Sri Lanka, local government, and the private sector to increase economic opportunities for vulnerable families in focused provinces in Sri Lanka. The SOLID Activity ends in August 30, 2017. USAID Sri Lanka will use the findings of the evaluation to document best practices, lessons learned and recommendations for enhancing agriculture and/or economic development interventions in Sri Lanka. USAID/Sri Lanka intends the evaluation be an effective learning tool that can: 1. Be used by USAID/Sri Lanka to assess the effectiveness of the SOLID program and in particular, help identify lessons learned; and 2. Be used by USAID, the implementing partner Chemonics International Inc., Government of Sri Lanka entities, and other donor organizations in their future planning of similar agriculture programs aimed at accelerating livelihood opportunities, economic growth and reducing poverty in a post conflict region. C.2 SUMMARY INFORMATION of SOLID ACTIVITY Activity Name Supporting Opportunities for Livelihoods Development (SOLID) Implementer Chemonics International, Inc. Contract # and Task Order # GS-23F-0127P AID-OAA-M-12-00008 Total Estimated Cost (TEC) $10,785,000 Life of Activity November 13, 2013 – August 30, 2017 Active Geographic Regions Northern, Eastern, North Central and Uva Provinces Development Objective(s) (DOs) DO 2: Increased and More Equitable Economic Growth in the North, East, and Surrounding Conflict Affected Areas1. USAID Office Office of Economic Growth C.3 BACKGROUND Sri Lanka has experienced rapid economic growth, achieving an average of 7 to 8 percent per year since the end of the 26-year civil war, which ended in 2009. However, that growth has not been equitable island-wide. Most of the economic growth has occurred in the Western Province, which generates nearly 44 percent of the Gross Domestic Product (GDP). In the regions of the country most directly affected by the long conflict, economic growth has been much slower, with the Eastern and Northern Provinces contributing only 6.3 percent and 4 percent to GDP respectively. The Western Province is represented by the Sinhalese majority, while the Eastern and Northern Provinces are primarily comprised of the minority ethnic groups, Tamils and Muslims. This economic disparity is stark and has been, and will continue to be, a motivator of dissonance, which could threaten lasting peace in the 27 USAID.GOV Final Performance Evaluation for SOLID Activity country. SOLID is a pilot activity under the Asia and the Middle East Economic Growth Best Practices (AMEG) Project. The main goal of the project is to strengthen the livelihoods of targeted households. The four objectives of the activity are: • Production practices of small-holder farmers improved: • Market access for small-holder farmers increased: • Knowledge of good agri-business/organizational practices increased: and • Household diets improved. A. Program Description The SOLID Activity is a buy-in task order under the AMEG Project. Its purpose is to increase household incomes and enhance the livelihoods of rural residents in Northern, North Central, Eastern, North Western, and Uva Provinces of Sri Lanka. The SOLID Activity achieves its goals by assisting farmers to increase agricultural production, reduce production costs, access markets, and improve household diets. SOLID’s approach is to introduce better production methods and product handling techniques; link farmers to markets; build relationships between farmers and buyers; enhance farm production capabilities by making available materials and equipment to increase production possibilities; and to improve household nutrition through education. SOLID’s interventions target farmers growing horticultural products, milk producers, and small-scale poultry operations. SOLID assisted 3,757 beneficiary households as of September 2016. The sector and geographical breakdown of beneficiaries are given in the table below: Beneficiary Breakdown by Sectors Beneficiary Breakdown by Region Horticulture 2,564 Northern Province 2,295 Dairy 945 Eastern Province 814 Poultry 248 North Central Province 648 Total 3,757 Total 3,757 B. Development Hypothesis and Results Framework The development hypothesis is: By increasing financial capital (household-level income), physical capital (productive assets), social capital (access to markets), and human capital (knowledge and nutrition), SOLID will strengthen target groups’ livelihoods and contribute to increased and more equitable, sustainable economic growth. This framework is represented graphically as follows: 28 USAID.GOV Final Performance Evaluation for SOLID Activity C. Program Modifications On September 24, 2013, USAID Sri Lanka approved committing up to $8,000,000 to the SOLID Activity as a two year pilot activity under the AMEG Project, managed by the Asia and Middle East Bureau in Washington D.C. and implemented by Chemonics International, Inc. Due to delays in obtaining activity approval from the Government of Sri Lanka (GSL) and additional delays in securing Activity registration under Sri Lanka's Office of Company Registration, in September 2014 an extension of twelve months to the period of performance was allowed and the TEC was increased to $9,285,000. This resulted in a new award completion date of November 13, 2016. In September 2016, the period of performance was extended from November 13, 2016 to September 14, 2017 with an increase in budget of $1,500,000, thus increasing the total estimated cost to $10,785,000. However, due to budgetary constraints and AMEG closeout requirements the SOLID Task Order will end on August 30, 2017. D. Project Summary SOLID’s approach to implementation is collaborative, involving private businesses, farmer or producer organizations that represent farmers, and agencies of the GSL working to increase rural incomes and farm production. The SOLID activity operates based on market demand, where advantageous interventions are possible, and it is sure that activities undertaken are practical and sustainable. SOLID provides training and technical assistance where market opportunities exist and private companies are willing to engage with farmers. SOLID is working in areas with vulnerable populations, where private businesses are willing to engage farmers and farming communities to supply additional products. SOLID’s original target was to assist 3,000 vulnerable households during the life of the Activity; an additional 1000 households were added during the extension period. 29 USAID.GOV Final Performance Evaluation for SOLID Activity Outcome 1: Production practices of smallholder farmers improved. SOLID assists farmers to produce agricultural goods ranging from big onions to milk. SOLID trained farmers on good agricultural practices and production of quality goods to meet market demands. Through training, technical assistance, and limited direct assistance, these smallholder farmers aim to achieve increases in productivity and gain additional income. Outcome 2: Market access for smallholder farmers increased. Each of SOLID’s interventions begins with an examination of market linkages and expected returns. In undertaking this work, SOLID partners with dairy companies, buyers and retailers of food products, input suppliers, as well as smaller organizations. SOLID links the farmers to these private sector companies. Outcome 3: Knowledge of good agri-business/organizational practices increased. SOLID commences its interventions working with individual farmers by providing technical assistance, training, and limited production inputs to initiate new farming activities. SOLID activities encourage the farmers to work as a group to benefit from collective bargaining. Outcome 4: Household diets improved. Activities under SOLID’s nutrition component were implemented in four stages: Stage 1: Needs based awareness on nutrition together with capacity building on a multi sectoral approach; Stage 2: Awareness among community leaders; Stage 3: Training SOLID beneficiaries on nutrition behavior change; and Stage 4: Training of SOLID beneficiaries in the neighborhood through selected households E. Documents for Review A variety of SOLID-related documents, including but not limited to the following, will be available and provided upon award to facilitate the desk review: • SOLID initial assessment report • Annual work plans • Current M&E Plan • Quarterly reports C.4 EVALUATION QUESTIONS 1. To what extent did the intended outcomes in the bulleted list below occur over the course of SOLID? Did outcomes differ by region, gender of beneficiary, or sector (dairy, poultry and horticulture)? • Improved agricultural outputs; • Household income; • Women’s engagement in agriculture and access to productive economic assets • Youth engagement in agriculture and • Household nutritional intake 2. What influence did the SOLID have on the USG/GSL relationship at the national and local levels? What does a productive relationship look like based on the SOLID experience? 3. Identify planned and unexpected results in relation to ensuring potential sustainability and effect on the local agricultural value chain through partnering with: 30 USAID.GOV Final Performance Evaluation for SOLID Activity • Other USG projects/activities • Other donors • Government of Sri Lanka • International, national, and local private sector firms 4. Are past beneficiaries currently implementing the skills and techniques promoted by SOLID? Did farmers involved in demonstration activities adopt the recommended practices? When evaluating the performance and effect of the SOLID Activity, the evaluators should also include the positive and negative impact of external factors, such as changes in economic, political, and financial conditions. C.5 EVALUATION DESIGN AND METHODOLOGY This final performance evaluation is intended to answer the evaluation questions presented above. The conceptual approach to answer these questions should focus on qualitative and quantitative data collection methods. This approach should include, but not limited to the following: desk study, key informant interviews, surveys, focus group discussions, and consultations with relevant stakeholders. The evaluation team should comprise of independent external consultants who have a sound knowledge of agriculture, livelihood, economic development and, the Sri Lankan economic development context. The evaluators’ expertise and input in the proposal and during the evaluation design phase is anticipated. The team should have a mix of Sri Lankan and international consultants and collectively have fluent tri￾lingual skills in English, Tamil, and Sinhala. The independent external evaluation team is expected to work in conjunction with USAID/Sri Lanka’s Program Office and the Office of Economic Growth to plan and implement the required evaluation. The consultants are expected to provide significant overall leadership and direction, exercise a degree of autonomy, and have the final responsibility for conducting the evaluation and completing evaluation deliverables. The evaluation team will be required to evaluate this project in a timely manner. Data requirements, collection methods, and required analyses will be determined collaboratively with USAID/Sri Lanka, under the direction of an independent evaluation team leader. Details on final data collection methods and instruments, key informants or respondents, and analytical framework(s) will be approved by USAID/Sri Lanka as part of the initial work plan approval. The methodological approach should include a reasonable sample of beneficiaries to be interviewed taking into consideration appropriate representation from geographical regions, sectors, ethnicity, and gender. Data, where applicable, must be disaggregated by sex, ethnicity, and location. The Mission is interested in gender issues, especially achievements or observations of the evaluation team during the course of the evaluation, in addition to those specified in the evaluation questions. The evaluation team may propose additional specific sub-question(s) related to gender issues in the work plan as appropriate. As summarized below, the data collection and analysis process will comprise of three phases. The Mission expects the evaluation team to present strong quantitative and qualitative analysis, within data limitations, that clearly addresses questions. All key and sub-questions stated in section IV above must be addressed, to the extent practical, in all three phases. The desk study and internal consultations (see more details below) will support planning for external data collection in the field. • Literature Review: The evaluation team shall review at least existing documents and information listed in section III E above, and work with USAID/Sri Lanka to acquire additional documents and information as needed, and prioritize primary data collection where gaps remain. Other literatures that provide 31 USAID.GOV Final Performance Evaluation for SOLID Activity knowledge on external factors, such as changes in economic and political conditions shall also be reviewed and utilized. • Internal Consultations: The evaluation team shall meet (to the extent possible) with the key stakeholders: USAID/DCHA-PPM and USAID/ASIA in Washington DC as the Project COR is based in Washington DC. In Sri Lanka, the evaluation team will meet the Activity Manager (AM) of the SOLID Activity, members of the SOLID team, the Economic Growth Office leadership, the Program Office, the SOLID evaluation manager, Office of Financial Management, and Office of Acquisition and Assistance. External interviews, surveys, and focus group discussions: The evaluation team will conduct in-person interviews, surveys, and/or focus group discussions with SOLID implementing partners, collaborating partners, selected beneficiaries, donors with similar projects, and other key stakeholders to allow for a range of perspectives and give depth to the evaluation. C.6 FINAL REPORT FORMAT The evaluation final report not exceeding 25 pages (excluding report annexes) should include an executive summary; introduction; background of the local context and the Activity being evaluated; the main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations; and lessons learned (if applicable) as described here. The report should be formatted according to the evaluation report template. The executive summary should be 3–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods and limitations, findings, conclusions, recommendations, and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.). • The annexes to the report shall include: • The Evaluation SOW; • As required, any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; • All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; • Sources of information, properly identified and listed; and • Disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of interest. In accordance with AIDAR 752.7005, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within 30 calendar days of final approval of the formatted report.” C.7 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per the USAID Evaluation Policy and USAID ADS 201, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report.2 • The evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not, and why. • The Evaluation report should be readily understood, should identify key points clearly, distinctly, and succinctly. • The Evaluation report will be structured and will lay out 1) findings, 2) recommendations, and 3) lessons learned. 32 USAID.GOV Final Performance Evaluation for SOLID Activity • The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. • Evaluation reports should adequately address all evaluation questions included in the statement of work, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. • The evaluation methodology shall be explained in detail and all sources of information properly identified. If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. • Findings should be specific, concise, and supported by strong quantitative and/or qualitative evidence. Recommendations need to be supported by a specific set of findings. Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. C.8 OTHER REQUIREMENTS All raw datasets (quantitative data and coded qualitative data) collected by the evaluation team must be provided in machine-readable, non-proprietary formats as required by USAID’s Open Data policy and must be submitted to the Development Data Library (DDL) at https://www.usaid.gov/data (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the Activity or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the Contracting Officer. 33 USAID.GOV Final Performance Evaluation for SOLID Activity ANNEX 2: EVALUATION TEAM Team Leader: Mr. David Rinck has more than 20 years of experience in monitoring and evaluation of economic growth initiatives, specializing in food security, agriculture and agro-enterprise development, market systems and economic policy analysis. With excellent technical capabilities, he has successfully designed, implemented and managed evaluations of agricultural development projects as both Team Leader and as a specialized team member. In his current role as the Managing Director of Absolute Options LLC (AO), Mr. Rinck has provided technical direction to over a dozen evaluations and agricultural value chain/market systems assessments, such as the Performance Evaluation of the USAID Solutions for African Food Enterprises (SAFE) Program in Kenya, Malawi and Zambia; Mid-Term Evaluation of the Inter-American Development Bank (IDB) Creating Alliances in Cacao for Improved Access and Organization (CACAO) Project in Haiti; and, Mid-Term Evaluation of Phase 2 of the United States Department of Agriculture (USDA) Cashew Value Chain Enhancement Project in Senegal and Gambia, to name a few. Before establishing Absolute Options LLC, Mr. Rinck served as the Chief of Party (COP) on the U.S. Department of State Results Oriented Commercial-Organization Capacity Development (ROCCD) Project in Tunisia. In this role, he managed a team of local and international staff to improve policy advocacy capacity and promote agricultural trade and investment. Previously, he served as a Regional Food for Peace Officer for USAID/East Africa, where Mr. Rinck supported food and agricultural market development and food security assistance programs. Mr. Rinck also worked with Catholic Relief Services (CRS) in Kenya, Zimbabwe and Benin to support development assistance programs in agriculture. He has excellent written and verbal communication skills, and has authored several publications on economic development. Mr. Rinck holds an M.A. in Social Sciences from the University of Chicago, and a B.S. in Agricultural Economics from the University of California at Davis. He is fluent in English, French, Spanish and Portuguese, and has a working knowledge of Arabic. Mid-Level Livelihood Specialist: Ms. Vichithrani (Vichi) Gunawardena has over 13 years of professional experience in food security and livelihoods needs assessment, plus market analysis and value chain development. She started her career working for Oxfam Great Britain (GB) as a Program Officer focusing on identification of livelihood development needs for tsunami-affected communities in southern Sri Lanka. In 2010, she worked for the World Bank as a Rural Development Specialist, providing technical support on livelihood needs assessments, value chain-based market analysis, and producer groups to establish value chain-based partnerships, youth skill development and women empowerment in community-driven development projects. Ms. Gunawardena recently worked for Save the Children International in Yemen as a Food Security and Livelihood Advisor. She has provided high-level work experience and commitment to participatory approaches, gender equality and reducing discrimination. She has experience in policy analysis and strategic planning, as well as extensive experience in planning, implementation and monitoring of livelihood development projects. She has also experience in evaluation of rural development projects with a special focus on livelihood development components, for example, when she worked for World Bank in 2012 to evaluate the Agriculture Global Practice Unit. Ms. Gunawardena holds both Bachelor and Master degree in Agriculture. She is a Sri Lankan native speaker who is proficient in Sinhalese and English. Mid-Level Agriculture Specialist: Mr. Fuard Marikar has over 40 years of experience in planning, strategy development, implementation, research, and evaluation of natural resource, agricultural and livelihoods projects across Sri Lanka. Previously engaged as an evaluation specialist and team leader, he has significant experience in evaluation design, data collection, economics, monitoring and surveying. Mr. 34 USAID.GOV Final Performance Evaluation for SOLID Activity Marikar began his career at the Ministry of Planning and Economic Affairs before moving to the Ministry of Lands and Land Development and Mahaweli Development and then becoming a full-time consultant. His technical areas of experience include water and sanitation, livestock management, sustainable agriculture, subsistence farming and SME development. He brings a wealth of knowledge in financial economics, project management, and analysis of large-scale agricultural projects. He has built strong relationships with government and private entities, as well as with NGOs and CBOs. He holds a PhD in natural Resource Economics from Colorado State University, a Master of Arts in Agricultural Economics from Stanford University and a Bachelor of Science in Agriculture from the University of Ceylon, Sri Lanka. He is a native Sri Lankan native who is proficient in Sinhalese and Tamil. Research Assistant: Mr. Zarouk Zihan has 14 years of planning, designing, supervising and implementing programmes and projects in a variety of environments, particularly in post crisis scenarios. He has extensive experience in progressively responsible positions with NGOs and the United Nations Development Program in Sri Lanka and Bosnia and Herzegovina, including as a field project specialist and programme officer overseeing socio-economic recovery & development, sustainable livelihood, disaster management and participatory rural engagement activities. He holds a Masters of Development Studies from the University of Colombo, Sri Lanka. Logistician: Ms. Sahunthalai Muthurajah has more than 4 years of experience in administrative on project management with both local and international NGOs. Since year 2011, after she graduated her Bachelor’s degree in Social Work subject, she started her career with a Project Coordinator for International Institute of Development Training (IIDT). She worked as a focal point of a multi-year project implemented by PADEM (France) for Improvement of Living and Economic Condition for Plantation Workers in Sri Lanka. The project covered 17 estates from Galle, Ratnapura and Nuwreliya Districts. Moreover, Ms. Muthurajah participated in a psychosocial training for trainers, which is a community based training course for improve her related field works. She has a very good command in Tamil, Sinhala, and English languages. She worked as a translator as required by projects. Ms. Muthurajah also holds a Master of Development Studies and LLB (honors). Team Language Abilities: The ET will collectively be able to communicate in English, Sinhalese, and Tamil. All team members have a good command of English. The Livelihood Specialist is also a native Sinhalese speaker. The Interpreter and Logistician both know Tamil and Sinhalese as well as English, as will the Research Assistant. This will ensure the team will be able to communicate clearly with partners and beneficiaries for data collection as well as submit clear written deliverables to USAID/Sri Lanka. 35 USAID.GOV Final Performance Evaluation for SOLID Activity ANNEX 3: DATA COLLECTION SCHEDULE Data collection took place between July 17 and August 14. July 7-22; KIIs in Colombo. Site visits took place between July 23 and August 9. Region District Location Date Northern Jaffna Pandatharipu 7/24/17 Northern Jaffna Achchuveli 7/24/17 Northern Killinochchi Kulunthupilavu 7/25/17 Northern Killinochchi Vattakachchi 7/25/17 Northern Vavuniya Thavasikulam 7/26/17 NorthernNorthern Vavuniya Sannasiparanthan 7/27/17 Northern Mullaitivu Muthayankaddu 7/27/17 Northern Vavuniya Nedunkeni 7/27/17 Northern Mannar Erukkalampitty 7/28/17 Northern Mannar Andankulam 7/28/17 North Central Anuradhapura Samagipura 7/31/17 North Central Anuradhapura Parawahagama 8/1/17 North Central Anuradhapura Kahalla 8/1/17 Eastern Trincomalee Thalgaswewa 8/2/17 Eastern Trincomalee Kinniya 8/2/17 Eastern Trincomalee Upuvali 8/3/17 Eastern Trincomalee Sampur 8/3/17 Eastern Batticaloa Vallavally 8/4/17 Eastern Batticaloa Ollikkulum 8/4/17 North Central Polonnaruwa Medirigiriya 8/7/17 North Central Polonnaruwa Katukeliyawa 8/8/17 FIGURE 11: DATA COLLECTION MAP FIGURE 11: DATA COLLECTION MAP FIGURE 11: DATA COLLECTION MAP TABLE #3: PLAN OF MANEUVER TABLE #3: PLAN OF MANEUVER PLAN OF MANEUVER 36 USAID.GOV Final Performance Evaluation for SOLID Activity ANNEX 4: REFERENCES Agarwal, Bina. A Field of One's Own: Gender and Land Rights in South Asia. Cambridge University Press, 1994. Atapattu, Dr. Sumith, and Andrew Baker, SOLID Broiler Training Manual. Chemonics. Ball, Donald M., C.S. Hoveland, G.D. Lace Field. Southern Forages: Modern Concepts for Forage Crop Management. International Plant Nutrition Institute, 1996. “CIC Dairies, Milco in two-year deal to buy milk from Sri Lankan farmers.” Economy Next, September 7, 2017. Doss, Cheryl. “The Role of Women in Agriculture.” Agricultural and Development Economics Division (ESA) Working Paper no. 11-02, Food and Agriculture Organization (FAO), March 2011. “Drought, Floods Slash Sri Lanka's Rice Production, Threaten Food Security - U.N.” Reuters, June 22, 2014. Exporter Guide – Sri Lanka. United States Department of Agriculture (USDA) Global Agricultural Information Network (GAIN), April 15, 2015. Fernando, Nilan. “Sri Lanka’s War Crimes Controversy.” New York Times, September 27, 2015. Howell, Jim. For the Love of Land: Global Case Studies of Grazing in Nature's Image. Booksurge Publishing, March 20th, 2009. King, Gary, Robert Keohane, and Sydney Verba. Designing Social Inquiry: Scientific Inference in Qualitative Research. Princeton University Press, 2016. Midmore, David. “Principles of Tropical Horticulture.” CABI, 2015. Monitoring and Evaluation Plan for SOLID phase 2 (Dairy Development). Chemonics. “More Warnings On NGO Subversion.” Daily News, Aug 20 2014. National Youth Policy of Sri Lanka. Ministry of Youth Affairs and Skills Development, 2014. Savory, Allan, Jody Butterfield. Holistic Management, Third Edition: A Commonsense Revolution to Restore Our Environment (3rd Edition). Island Press, 2016. Scope of Work – Sri Lanka Supporting Opportunities in Livelihoods Development Pilot Activity. Chemonics, November 1, 2013. SOLID Activity Annual Report – October 2014 – September 2015. Chemonics, revised December 7, 2015. SOLID Annual Report: October 2015 – September 2016. Chemonics, November 15, 2016. 37 USAID.GOV Final Performance Evaluation for SOLID Activity SOLID First Year Work Plan (with General Life-of-Project Work Plan) – Scope of Work from November 14, 2013 through November 13, 2016. Chemonics, June 2014. Sri Lanka: Dairy Sector Growth Opportunities. FAO, 2010. Supporting Livelihoods for Vulnerable Populations in Sri Lanka - Assessment of Findings and Opportunities. AMEG and USAID, March 29, 2013. Wickramarachchi1, A.R., H. M. L. K. Herath, U. K. Jayasinghe-Mudalige, J. C. Edirisinghe1, J. M. M. Udugama, L. D. Lokuge1W. Wijesuriya. “An Analysis of Price Behavior of Major Poultry Products in Sri Lanka.” Journal of Agricultural Sciences, Vol. 12, No. 2, May 2017. “WikiLeaks: Mahinda Rajapaksa 'Responsible for War Crimes’.” The Telegraph, November 2010. 38 USAID.GOV Final Performance Evaluation for SOLID Activity ANNEX 5: KEY INFORMANT INTERVIEWS AGENCY / COMPANY TITLE DATE Chemonics Backstop 7/15/17 Chemonics Deputy Team Leader 7/18/17 USAID Economic Advisor 7/19/17 Ministry of National Policy Planning Additional Secretary (Development) 7/19/17 Ministry of National Policy Planning Advisory Secretary (Planning Div.) 7/19/17 Chemonics Chief of Party (CoP) 7/19/17 USAID Budget Specialist, M&E Focal Point 7/20/17 USAID Senior Program Mgt. Specialist 7/20/17 Chemonics M&E Manager 7/20/17 Chemonics Dairy Technical Adviser 7/20/17 Dept. of Animal Production and Health Provincial Director 7/24/17 Nestle Field Executive Officer 7/24/17 Farmer Dairy 7/24/17 Farmer Horticulture (Big Onion) 7/25/17 Chemonics (former) Field Assistant - Killinochchi District 7/25/17 Dept. of Animal Production and Health Deputy Director 7/26/17 Chemonics Reg. Coordinator - Northern Region 7/26/17 Nestle Field Extension Officer 7/26/17 Vavuniya Fruit Growers Cooperative Field Officer 7/27/17 Farmer Dairy 7/27/17 Farmer Horticulture (Morunga) 7/27/17 Department of Agriculture Deputy Provincial Director 7/28/17 Nestle Field Extension Officer 7/28/17 Farmer Dairy 7/31/17 Department of Agriculture Agriculture Instructor 7/31/17 Farmer Horticulture (Passion Fruit) 7/31/17 Cargill's Field Officer 7/31/17 Farmer Horticulture (Chili Seed) 8/1/17 Department of Agriculture Agriculture Instructor 8/1/17 CIC Dairy Development Officer 8/2/17 Department of Agriculture Director of Planning 8/2/17 Trinco Broiler Prod. and Mgt. Ltd. President 8/2/17 USAID Team Leader - EG, Middle East Bureau 8/3/17 USAID Director of Economic Dev. and Energy 8/3/17 Chemonics Technical Officer, Dairy 8/4/17 Farmer Poultry 8/4/17 Ministry of Agriculture Additional Secretary of Technology 8/9/17 Chemonics Team Leader (former CoP AMEG) 8/10/17 Cargill's General Manager 8/11/17 HJS Condiments Pvt Ltd Managing Director 8/12/17 39 USAID.GOV Final Performance Evaluation for SOLID Activity ANNEX 6: FOCUS GROUP DISCUSSIONS Region/ District Location Sector Participants Date Northern Jaffna Pandatharipu Dairy 10 7/24/17 Jaffna Achchuveli Horticulture (Red Onion) 9 7/24/17 Killinochchi Kulunthupilavu Horticulture (Red Onion) 8 7/25/17 Killinochchi Vattakachchi Dairy 10 7/25/17 Vavuniya Thavasikulam Dairy 13 7/26/17 Vavuniya Sannasiparanthan Horticulture (Morunga) 6 7/27/17 Mullaitivu Muthayankaddu Dairy 7 7/27/17 Vavuniya Nedunkeni Horticulture (Passion Fruit) 8 7/27/17 Mannar Erukkalampitty Horticulture (Big Onion) 7 7/28/17 Mannar Andankulam Dairy 10 7/28/17 N. Central Anuradhapura Samagipura Dairy 8 7/31/17 Anuradhapura Parawahagama Horticulture (Chili Seed) 14 8/1/17 Anuradhapura Kahalla Horticulture (Chili Seed) 13 8/1/17 Polonnaruwa Medirigiriya Horticulture (Mushroom) 9 8/7/17 Polonnaruwa Katukeliyawa Dairy 13 8/8/17 Eastern Trincomalee Thalgaswewa Dairy 5 8/2/17 Trincomalee Kinniya Poultry 3 8/2/17 Trincomalee Upuvali Horticulture (Chili and Red Onion) 7 8/3/17 Trincomalee Sampur Horticulture (Chili and Up-Country Vegetables) 7 8/3/17 Batticoloa Vallavally Dairy 7 8/4/17 Batticoloa Ollikkulum Poultry 11 8/4/17 Total 185 FOCUS GROUP DISCUSSIONS 40 USAID.GOV Final Performance Evaluation for SOLID Activity ANNEX 7: MINI-SURVEY PARTICIPANTS Participants Region/ District Location Sector Total Males Females Youth Northern Jaffna Pandatharipu Dairy 10 9 1 1 Jaffna Achchuveli Horticulture 9 4 5 2 Killinochchi Kulunthupilavu Horticulture 8 5 3 3 Killinochchi Vattakachchi Dairy 10 5 5 1 Vavuniya Thavasikulam Dairy 13 10 3 3 Vavuniya Sannasiparanthan Horticulture 6 4 2 2 Mullaitivu Muthayankaddu Dairy 5 3 2 0 Vavuniya Nedunkeni Horticulture 8 5 3 0 Mannar Erukkalampitty Horticulture 7 4 3 0 Mannar Andankulam Dairy 10 4 6 0 N. Central Anuradhapura Samagipura Dairy 0 0 0 0 Anuradhapura Parawahagama Horticulture 14 6 8 3 Anuradhapura Kahalla Horticulture 13 9 4 1 Polonnaruwa Medirigiriya Horticulture 9 0 9 0 Polonnaruwa Katukeliyawa Dairy 13 11 2 2 Eastern Trincomalee Thalgaswewa Dairy 7 4 3 0 Trincomalee Kinniya Poultry 3 2 1 0 Trincomalee Upuvali Horticulture 8 8 0 5 Trincomalee Sampur Horticulture 7 6 1 5 Batticoloa Vallavally Dairy 7 7 0 2 Batticoloa Ollikkulum Poultry 7 2 5 0 Total 174 108 66 30 Percent (%) 100% 62% 38% 17% MINI-SURVEY PARTICIPANTS 41 USAID.GOV Final Performance Evaluation for SOLID Activity ANNEX 8: DATA COLLECTION PROTOCOLS 1. KII Guide – (circle one) USAID and other Donors, GoSL, Private Sector Companies, and Beneficiaries Interview date and location: Interviewer: [Where appropriate] Agency or Company: [Where appropriate] Title(s): [Beneficiaries] [circle type: dairy, poultry, horticulture (note specific horticultural crop)] Name(s): Sex: Consent Statement: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called Social Impact, a company that is based in the United States. Our team is in Sri Lanka to conduct a study about the SOLID Activity, which was funded by USAID. We would like to conduct a brief interview today to learn about your perspectives on project performance. This information will be used in report for USAID that will be publicly available. It is important to understand that while we would like your help in this study, you do not have to take part if you do not want to, and you do not have to answer any questions if you do not feel comfortable answering. If you decide to take part, your responses will be kept strictly confidential. This means that your name will not be mentioned anywhere in the report, and will not be provided to anyone, including at your agency. Any personal information we collect today will be stored in a secure computer file. The objective of this research is to improve the performance of projects like SOLID. The information may be used by other organizations as well. The interview is expected to take about 60 minutes. Do you have any questions? You may ask questions at any time. If you have questions or concerns about the research after we leave today, you can contact [insert ET name and contact information]. By saying “yes,” and participating in this study, you are indicating that you have heard this consent statement, had an opportunity to ask any questions about your participation and voluntarily consent to participate. Will you participate in this interview? You may answer yes or no. ◻ Yes, I will participate ◻ No, I will not participate 42 USAID.GOV Final Performance Evaluation for SOLID Activity Background Data (Select relevant questions) B.1a. [GoSL staff] What is the structure and role of your agency? B.1b. [Private Sector] Please provide an overview of your operations in Sri Lanka. B.2. What are the major challenges and opportunities in the region(s) where you work? [Probe for ways in which these may be unique to the region, or may vary by stakeholder group i.e. youth, women, etc.] B.3. What is your experience, if any, with other USAID, GoSL or other donor projects related to sectors addressed by SOLID? B.4. What input, if any, did you or your agency/company provide into the design or implementation of the SOLID project? EQ1: To what extent did the intended outcomes in the bulleted list below occur over the course of SOLID? Did outcomes differ by region, gender of beneficiary, or sector (dairy, poultry and horticulture): Improved agricultural outputs; Household income; Women’s engagement in agriculture and access to productive economic assets; Youth engagement in agriculture and; Household nutritional intake 1.1. To what extent, if any, do you believe the SOLID activity has achieved its intended outcomes? [Ask open-ended then probe as needed for examples along the areas below to understand if outcomes have or have not been achieved] • What were the changes related to agricultural outputs? • What were the changes related to household income? • What were the changes related to household nutritional intake? 1.2. What factors have contributed to these results? How have they contributed to results? [Ask open￾ended, then probe for 1-2 examples as relevant amongst the areas below] • Specific training • Providing inputs • Introducing new technology • Improving production practices • Increasing market access • Increasing knowledge of good agribusiness/ organizational practices • Improving access to finance 1.3 How if at all have outcomes varied by: • Region? • Between former conflict and non-conflict zones? • Farming type? 43 USAID.GOV Final Performance Evaluation for SOLID Activity 1.3a. What factors are behind any variation(s)? [Probe: economic, social, climatic, or political events] 1.4. How are youth more engaged in agriculture as a result of SOLID activities, if at all? 1.4a. What factors have contributed to (or detracted from) these results for youth, and how? 1.5. To what extent, if any, have SOLID outcomes varied for women? [Probe for examples about women’s engagement in agriculture and access to productive economic assets] 1.5a. What factors have contributed to (or detracted from) these results for women, and how? 1.6. How have outcomes differed for men/women and youth? 1.7. How, if at all, could the activities under SOLID have been improved? [Probe for examples, including by stakeholder group] EQ2: What influence did SOLID have on the USG/GSL relationship at the national and local levels? What does a productive relationship look like based on the SOLID experience? 2.1. How, if at all, did SOLID project staff work with the GoSL? What about with other donors (IOs, DFIs)? 2.1a. How would you describe this relationship? [Probe for success and challenges of these working relationships] 2.2. How if at all has the SOLID activity influenced the way the GoSL works with the USG? What examples illustrate this influence? 2.3. How could the SOLID Activity better support the relationship between the USG and the GoSL? EQ3: Identify planned and unexpected results in relation to ensuring potential sustainability and effect on the local agricultural value chain through partnering with: Other USG projects/activities; Other donors; Government of Sri Lanka, and: International, national, and local private sector firms 3.1. How, if at all, have activities under SOLID sustainably impacted agricultural value chains through its partnerships? [Probe for relevant 1-2 examples below] • Other USG projects/activities • Other donors • Government of Sri Lanka • International, national, and local private sector firms 3.2. What factors are behind the sustainability of any improvements or lack of improvements? Please provide specific examples. 44 USAID.GOV Final Performance Evaluation for SOLID Activity 3.3. Do you believe changes in income, if any, that resulted from SOLID will be sustainable? If so, why? 3.3a. Do you believe that increases in sales will be sustainable? If so, why? 3.4. What, if any, are the unexpected results on agricultural value chains that have resulted from the SOLID Activity? EQ4: Are past beneficiaries currently implementing the skills and techniques promoted by SOLID? Did farmers involved in demonstration activities adopt the recommended practices? 4.1. How, if at all, have beneficiaries implemented the skills and techniques introduced by the SOLID Activity? Please provide examples describing the specific beneficiary, skill and/or technique. [Probe for specific mention of farmers involved in demonstration activities] 4.1a. What information is available that documents the uptake of these skills/techniques etc. by beneficiaries? [Probe for any environmental/situational challenges with implementation as well] 4.5. How likely are beneficiaries (you) to continue utilizing the skills and techniques introduced by the SOLID Activity? Why or why not? 4.6. What factors have impeded implementing these skills? 4.6a How could these factors have been overcome? 4.7. Have beneficiaries (you) accessed credit or loans to finance the activities related to SOLID that they implement? If not, why not? 45 USAID.GOV Final Performance Evaluation for SOLID Activity 2. FGD Guide - Project Beneficiaries Interview date and location: Interviewer: Province/District: Total Participants (number): Production Type (circle and number): horticulture: poultry: dairy: Youth (number): Sex (number): Males: Females: Consent Statement: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called Social Impact, a company that is based in the United States. Our team is in Sri Lanka to conduct a study about the SOLID Activity We would like to conduct a discussion with you today to learn about assistance and support you may have received through the SOLID project. It is important to understand that while we would like your help in this study, you do not have to take part if you do not want to, and you do not have to answer any questions if you do not feel comfortable doing so. If you decide to take part, your responses will be kept strictly confidential. This means that your name will not be mentioned anywhere in the report, and will not be provided to anyone, including Chemonics or anyone in your community. Any personal information we collect today will be stored in a secure computer file. The objective of this research is to improve the performance of projects like SOLID. The information may be used by other organizations as well. It is very important for you to know that this study will NOT determine if you will receive any services now or in the future. The way you answer our questions will NOT determine if you will receive any kind of government or NGO assistance. There is no direct benefit to you for your participation in this study. The purpose is only to help us improve the services of projects like this one or of the donors. The discussion is expected to take about 1.5-2 hours. You may ask questions at any time, or excuse yourself at any point if you no longer want to participate. If you have questions or concerns about the research after we leave today, you can contact [insert ET name and contact information]. Are there any questions? By saying “yes,” and participating in this study, you are indicating that you have heard this consent statement, had an opportunity to ask any questions about your participation and voluntarily consent to participate. Please raise your hand if you are willing to participate in this discussion. 46 USAID.GOV Final Performance Evaluation for SOLID Activity Background Data B.1. What type of farming [poultry/dairy/horticulture (note specific crop)] do you practice? B.2. What are the major opportunities/challenges for production in the region where you work? B.3. What experience have you had with other external projects (explain)? B.4. Are you members in any farmer organizations? If so, which ones? For how long? What service(s) does the organization provide? How useful is this organization? EQ1: To what extent did the intended outcomes in the bulleted list below occur over the course of SOLID? Did outcomes differ by region, gender of beneficiary, or sector (dairy, poultry and horticulture): Improved agricultural outputs; Household income; Women’s engagement in agriculture and access to productive economic assets; Youth engagement in agriculture and; Household nutritional intake 1.1 How, if at all, did SOLID impact your agricultural production and income? [Probe for changes or results that were observed (positive, negative or no change)] • For example, were there changes in HH income? • What about nutritional intake? • Were these changes due to SOLID or another intervention? 1.2 What factors were behind any changes in production? [Capture examples from the list below as possible] • Specific training • Providing inputs • Introducing new technology • Improving production practices • Increasing market access • Increasing knowledge of good agribusiness/ organizational practices • Improving access to finance 1.3. Were there any external factors that impacted these result (financial, political etc.)? How did these factors impact results? 1.4. Have you noticed any differences in the roles of women at the HH level as a result of SOLID? 1.4a. What about for youth? If yes, in what ways? If not, why? 1.5. If so, what does this look like in reality? [Probe for relevant examples related to access to assets, social status etc.] 1.5a. Are youth now more engaged in agriculture? Why or why not? If so, how? 47 USAID.GOV Final Performance Evaluation for SOLID Activity 1.6. How, if at all, have SOLID activities changed the way you eat in your homes? Do you eat new foods as a result of SOLID activities? EQ3: Identify planned and unexpected results in relation to ensuring potential sustainability and effect on the local agricultural value chain through partnering with: Other USG projects/activities; Other donors; Government of Sri Lanka, and; International, national, and local private sector firms 3.1. What have been the major changes to your farming that have resulted from working with SOLID? 3.2. What more needs to be done to improve your production and your income? 3.3. Do you believe changes in income you may have experienced, if any, that resulted from SOLID will continue into the future? If so, why? 3.3a. Do you believe that increases in sales will continue into the future? If so, why? 3.4. What, if any, are the unexpected results your production that have resulted from the SOLID Activity? EQ4: Are past beneficiaries currently implementing the skills and techniques promoted by SOLID? Did farmers involved in demonstration activities adopt the recommended practices? 4.1 Has SOLID introduced any training, inputs or new technology to you? If yes, can you give examples of how you are using these new approaches? If no, why not? 4.1a. If yes, how likely are you to continue utilizing the training, inputs, and new technology introduced by SOLID? 4.1b. If no, was there a new approach that you wanted to learn but did not? 4.2. How, if at all, could the training, inputs, and new technology introduced by SOLID have been more effective? 4.3. Are there any challenges in using the training, inputs, and new technology provided by SOLID? If so, why is this? 4.3a. Do men and women face different challenges in this respect? 4.3b. Do youth face unique challenges in this respect? 4.3c How could these challenges be overcome? 4.4. What additional training do you need to improve your production and your income? 48 USAID.GOV Final Performance Evaluation for SOLID Activity 3. Mini Survey – Project Beneficiaries Survey date and location: Enumerator: Province/District: Production Type (circle): horticulture poultry dairy Youth (age 18-25) (yes or no): Gender (male or female): Consent Statement: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called Social Impact, a company that is based in the United States. Our team is in Sri Lanka to conduct a study about the SOLID Activity We would like to conduct a brief survey today to learn about the assistance and support you or your organization received through SOLID. This study seeks to understand how SOLID services and support improved your farming and nutrition. In addition to this survey, our team will be speaking with many people in Sri Lanka to understand their experiences as well. It is important to understand that while we would like your help in this study, you do not have to take part if you do not want to, and you do not have to answer any questions if you do not feel comfortable doing so. If you decide to take part, your responses will be kept strictly confidential. This means that your name will not be mentioned anywhere in the report, and will not be provided to anyone, including Chemonics or anyone in your community. Any personal information we collect today will be stored in a secure computer file. The objective of this research is to improve the performance of projects like SOLID to better serve your community. The information may be used by other organizations as well. It is very important for you to know that this study will NOT determine if you will receive any services now or in the future. The way you answer our questions will NOT determine if you will receive any kind of government or NGO assistance. There is no direct benefit to you or your organization for your participation in this study. The purpose is only to help us improve the services of projects like this one or of the donors. The survey is expected to take about 60 minutes. Do you have any questions? You may ask questions at any time. If you have questions or concerns about the research after we leave today, you can contact [insert ET name and contact information]. By saying “yes,” and participating in this study, you are indicating that you have heard this consent statement, had an opportunity to ask any questions about your participation and voluntarily consent to participate. Will you participate in this survey? You may answer yes or no. ◻ Yes, I will participate ◻ No, I will not participate 49 USAID.GOV Final Performance Evaluation for SOLID Activity Background Data (ensure quality check on responses) Crops Previous to Solid Currently Are you a member of a farmer organization (yes/no)? Farm area (HA) Number of crops (#) Use of pesticides (yes/no) Importance of crops in home consumption (1-10) Change in income from crops due to SOLID (to nearest 10%) N/A Comments on crops Poultry Are you a member of a poultry organization (yes/no)? Number of poultry (#) Importance of poultry in home consumption (1-10) Feed source (free range/own production/purchase) Change in income from poultry due to SOLID (%) N/A Comments on poultry Dairy Are you a member of a dairy organization (yes/no)? Number of dairy cattle (#) Fodder source (free range/own production/purchase) Importance of dairy in home consumption (1-10) Change in income from dairy due to SOLID N/A Comments on dairy Did you access new finance through SOLID support? YES NO Opinions on SOLID (select one column) Questions Strongly Disagree Disagree Neither Agree Nor Disagree Agree Strongly Agree SOLID activities addressed my priority concerns regarding farming/poultry/dairy SOLID activities addressed my priorities regarding nutrition SOLID activities were effective in improving my farming/poultry/dairy income SOLID activities were effective in improving my farming family’s nutrition SOLID Activities were easy to implement I am likely to continue implementing 50 USAID.GOV Final Performance Evaluation for SOLID Activity techniques I learned through SOLID Youth were as likely as older persons to benefit from SOLID activities Women were as likely as men to benefit from SOLID activities SOLID Techniques (yes/no and ranking 1-5 where “1” is lowest and “5” is highest) Techniques Did you learn this technique through SOLID support (yes/no)? Is this a new technique for you (yes/no)? Did you receive sufficient support from SOLID to practice technique (1-5)? How likely are you to continue to practice this technique (1-5)? How important is this technique to your household income (1- 5)? How important is this technique to your household nutrition (1- 5)? 1. 1. Fodder sorghum used as cattle feed 2. 2. Silage production 3. 4. 3. Production of Total Mixed Ration (TMR) cattle feed 5. 4. Big onion seedling production using raised beds and rain shelters 6. 5. Field transplanting of big onion seedlings 7. 6. Sprinkler irrigation systems for water conservation 8. 7. True seed production 9. 8. New cultivation methods for growing value-added small gherkin 10. 9. Oyster mushroom production tech 11. 10. Broiler coop design 12. 11. Use of rain shelters 13. 12. Broiler management practices 14. 15. 13. Onion cultivation using true seeds Additional Comment 51 USAID.GOV Final Performance Evaluation for SOLID Activity ANNEX 9: PROJECT BACKGROUND Project Background AMEG and USAID jointly implemented a livelihoods assessment in post-conflict regions of Sri Lanka from March 4 to 22, 2013 to “identify constraints to the livelihoods of vulnerable populations in the North and select districts in the East, including North and South Trincomalee and West Batticaloa, and inform future activities to transition USAID development assistance to targeted, sustainable livelihoods development”.34 This assessment, determined that opportunities in vocational training, as well as in the dairy, horticulture and poultry sectors would be the most viable in strengthening the livelihoods of targeted populations. Based on this initial assessment (eliminating vocational training), Chemonics subsequently prepared a SO) for an activity that would reach an estimated 3,000 households with “replicable, scalable, and sustainable” pilot activities, which USAID approved on November 1, 2013.35 The SOW defined a two-year “base period” (November 15, 2013 to November 14, 2015) with a six￾month “option” (November 15, 2015 to May 14, 2016), dependent on an extension of the AMEG contract for a corresponding period, and which was eventually funded to August 2017. KIIs with USAID and IP staff indicate that SOLID was initially designed as a “buy-in” to the AMEG project in order to rapidly and flexibly implement a scalable activity to improve the livelihoods of vulnerable populations in the selected areas. As a pilot project taking an incremental approach, in an initial annual work plan, Chemonics proposed an “illustrative” initial list of indicators “likely to be included in the monitoring and evaluation (M&E) Plan”, employing standard “F-indicators” to the degree possible, as opposed to proposing a full Performance Monitoring Plan (PMP) or Results Framework prior to project start-up. 36 The initial annual work plan also included the results of a commodity selection workshop, which convened specialists to discuss and decide on a final list of horticultural commodities and interventions that would have the highest potential livelihoods impact in target districts based on a detailed selection process, as well as value chain maps and financial projections developed through two subsequent analyses by consultants with extensive commercial experience in the selected value chains. The first annual report included the fully developed Results Framework build around four intermediate results (IR) (see Annex 10: SOLID Results Framework).37 This document also reported 2,145 farmers engaged in dairy and seven horticultural activities in eight districts, and the start-up of several crosscutting activities in gender and environmental protection (safe pesticide use). The second annual work plan anticipated reaching more than 3,200 households through the addition of 1,078 households in new interventions, including a youth farmer program in Trincomalee District. 38 The report also described new water conservation activities to counter water-related constraints, the 34 “Supporting Livelihoods for Vulnerable Populations in Sri Lanka - Assessment of Findings and Opportunities,” AMEG and USAID, March 29, 2013. 35 “Scope of Work – Sri Lanka Supporting Opportunities in Livelihoods Development Pilot Activity,” Chemonics, November 1, 2013. 36 “SOLID First Year Work Plan (with General Life-of-Project Work Plan) – Scope of Work from November 14, 2013 through November 13, 2016,” Chemonics, June 2014. 37 “SOLID Activity Annual Report – October 2014 – September 2015,” Chemonics, revised December 7, 2015. 38 “SOLID Second Year Work Plan,” Chemonics, September 30, 2015. 52 USAID.GOV Final Performance Evaluation for SOLID Activity completion of training in poultry, and the anticipated scale-up of nutrition activities launched in June 2015. An annex to the Year Two Work Plan recognizes climatic events as producing significant disruptions to horticultural production, estimating that only 793 of 1,467 horticultural beneficiaries, or 54 percent, were able to complete activities during Year 1 due to weather-related losses.39 The second annual report describes the introduction of 13 new technologies over the life of the activity to improve productivity, conserve resources (especially water), reduce pesticide use, and save labor.40 It also reports assisting more than 3,700 farming households to increase income by more than 10 percent with female participation at training or technical assistance demonstrations between 40 to 45 percent and female direct beneficiaries totaling 30 percent overall. At the time of writing, a final project report was under review. 39 “SOLID Second Year Work Plan: Annex 1- Crop Loss Due to Bad Weather: Rate and Revenue Loss in Year One Interventions,” Chemonics, September 30, 2015. 40 Ibid 53 USAID.GOV Final Performance Evaluation for SOLID Activity ANNEX 10: SOLID ACTIVITIES BY PROVINCE Source: “SOLID Annual Report: October 2015 – September 2016,” Chemonics, November 15, 2016. 6 5 2 1 4 9 3 7 8 1 54 USAID.GOV Final Performance Evaluation for SOLID Activity ANNEX 11: SOLID RESULTS FRAMEWORK Figure 1: SOLID Results Framework Source: “SOLID Activity Annual Report – October 2014 – September 2015,” Chemonics, revised December 7, 2015. 55 USAID.GOV Final Performance Evaluation for SOLID Activity ANNEX 12: EVALUATION DESIGN MATRIX Evaluation Questions (EQs) Data Sources Data Collection Methods Data Analysis Methods 1. To what extent did the intended outcomes in the bulleted list below occur over the course of SOLID? Did outcomes differ by region, gender of beneficiary, or sector (dairy, poultry and horticulture)? • Improved agricultural outputs; • Household income; • Women’s engagement in agriculture and access to productive economic assets • Youth engagement in agriculture • Household nutritional intake • Project documents • USAID staff • Project Staff • Provincial/District GoSL Staff • Project beneficiaries • Document Review • KIIs-Project staff (CII and IPs) • KIIs-Provincial/District GoSL Staff • FGDs-Project Beneficiaries • FGDs-Project Beneficiaries – Youth • FGDs-Project Beneficiaries– Women • Mini-Survey– Project Beneficiaries Content Analysis – For identifying SOLID program successes (or lack of successes) Trend Analysis – For assessing progress against program indicators Gender Analysis – All data collected through its KIIs, FGDs and observations will be disaggregated by sex and analyzed for its effects on female beneficiaries of the program. 2. Is there evidence that SOLID activities influenced relationships between the USG and the GoSL (national and local), and if so, how? How do stakeholders view a productive relationship? • Project documents • USAID staff • Project Staff • Provincial/District GoSL Staff • Document Review • KIIs – USAID Staff • KIIs – National GoSL Staff • KIIs – Project staff (CII and IPs) • KIIs – Provincial/District GoSL Staff Content Analysis – For identifying SOLID program successes (or lack of successes) Trend Analysis – For assessing progress against program indicators 3. Identify planned and unexpected results in relation to ensuring potential sustainability and effect on the local agricultural value chain through partnering with: • Other USG projects/activities • Other donors • Government of Sri Lanka • International, national, and local private • Project documents • Project Staff • Provincial/District GoSL Staff • Project Beneficiaries • Project Beneficiaries – Youth • Project Beneficiaries – Women • Document Review • KIIs-Project staff (CII and IPs) • KIIs-Provincial/District GoSL Staff • FGDs-Project Beneficiaries • FGDs-Project Beneficiaries – Youth • FGDs-Project Beneficiaries– Women • Mini-Survey– Project Beneficiaries Content Analysis – For identifying SOLID program successes (or lack of successes) Trend Analysis – For assessing progress against program indicators Gender Analysis – All data collected through its KIIs, FGDs and observations will be disaggregated by gender and analyzed for its effects on female beneficiaries of the program. 56 USAID.GOV Final Performance Evaluation for SOLID Activity Evaluation Questions (EQs) Data Sources Data Collection Methods Data Analysis Methods sector firms 4. Are past beneficiaries currently implementing the skills and techniques promoted by SOLID? Did farmers involved in demonstration activities adopt the recommended practices? • Project documents • Provincial/District GoSL Staff • Project beneficiaries • Project beneficiaries – Youth • Project beneficiaries – Women • Document Review • KIIs-Project staff (CII and IPs) • KIIs-Provincial/District GoSL Staff • FGDs-Project Beneficiaries • FGDs-Project Beneficiaries – Youth • FGDs-Project Beneficiaries– Women • Mini-Survey– Project Beneficiaries Content Analysis – For identifying SOLID program successes (or lack of successes) Trend Analysis – For assessing progress against program indicators Gender Analysis – All data collected through its KIIs, FGDs and observations will be disaggregated by gender and analyzed for its effects on female beneficiaries of the program. 57 USAID.GOV Final Performance Evaluation for SOLID Activity ANNEX 13: EVALUATION TEAM CONFLICT OF INTEREST FORMS 58 USAID.GOV Final Performance Evaluation for SOLID Activity 59 USAID.GOV Final Performance Evaluation for SOLID Activity 60 USAID.GOV Final Performance Evaluation for SOLID Activity 61 USAID.GOV Final Performance Evaluation for SOLID Activity 62 USAID.GOV Final Performance Evaluation for SOLID Activity 1 USAID.GOV Final Performance Evaluation for SOLID Activity