November 2018 This publication was produced at the request of the United States Agency for International Development. It was prepared by independently by Matteo Borzoni (Team Leader) with the support of Joanna Khater from Social Impact, Inc. Note: Limited redactions have been made to this version of the report in accordance with the principled exceptions to the presumption in favor of openness established in OMB Bulletin 12-01, “Guidance on Collection of U.S. Foreign Assistance Data.” LEBANON INDUSTRY VALUE CHAIN DEVELOPMENT (LIVCD) FINAL PERFORMANCE EVALUATION FINAL REPORT Performance Management and Support Program for Lebanon (PMSPL II) Lebanon Industry Value Chain Development (LIVCD) Final Performance Evaluation Cover Photo: Workers sorting apples at the cold storage facility of Liban Village, Lebanon (Matteo Borzoni, Evaluation Team Leader/ Social Impact) Performance Management and Support Program for Lebanon (PMSPL II) November 2018 This document was submitted by Social Impact, Inc., to the United States Agency for International Development under USAID Contract/Order No. AID-268-C-15-00001. This document is available printed or online. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information is available from: Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ii | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 ABSTRACT Implemented between September 2012 and January 2019, the USAID-funded Lebanon Industry Value Chain Development (LIVCD) project sought to develop inclusive and competitive value chains in rural areas of the country to help increase the income of the population there, including micro, small, and medium enterprises. The evaluation examines the project’s relevance, efficiency and cost-effectiveness, effectiveness, sustainability, and additionality. It provides conclusions and recommendations to inform future activities and follow-on actions. The methods employed included a document review, 64 key￾informant interviews, 28 focus group discussions (with 221 individuals), and direct observations. LIVCD’s performance varied according to the targeted value chain. Interventions with avocado and olive farmers were the most cost-effective, while those with apple and cherry farmers were the least cost￾effective. Significant contextual challenges hindered LIVCD’s performance. However, the evaluation found evidence of improved incomes and competitiveness for avocado, olive, and grape farmers in addition to processed-food companies. There was limited or no income improvement for LIVCD-supported apple and cherry farmers. The total value of loans raised was considerable. However, access to finance improved for 102 beneficiaries only and was rather the result of investments leveraged by grants and technical assistance than that of activities aimed at promoting access to finance. Additionality was evident when new technologies or techniques were introduced. The project design included aspects related to ensuring sustainability. However, without further improvements in the quality of the produce, the interventions’ sustainability is at risk, especially in the apple value chain. USAID/LEBANON LIVCD FINAL EVALUATION REPORT | iii NOVEMBER 2018 Table of Contents ABSTRACT .................................................................................................................................... i ACRONYMS.................................................................................................................................. v EXECUTIVE SUMMARY.............................................................................................................. 1 I. Introduction........................................................................................................................... 7 II. The Development Problem and USAID’s Response ......................................................... 7 The Development Problem........................................................................................................................................ 7 USAID’s Response........................................................................................................................................................ 8 III. Evaluation Purpose, Users, and Questions ....................................................................... 10 Evaluation Purpose and Users................................................................................................................................. 10 Evaluation Questions................................................................................................................................................. 10 EVALUATION TEAM AND SUPPORT STRUCTURE ..................................................................................... 11 IV. Evaluation Methodology ..................................................................................................... 12 Data Collection Methods.......................................................................................................................................... 12 Document Review................................................................................................................................................. 12 Key Informant Interviews.................................................................................................................................... 13 Focus Group Discussions (FGDs) ..................................................................................................................... 13 Observations........................................................................................................................................................... 14 Data Analysis................................................................................................................................................................ 14 Data Analysis Processes....................................................................................................................................... 15 Data Triangulation................................................................................................................................................. 15 Data Analysis Methods......................................................................................................................................... 15 Biases and Other Limitations................................................................................................................................... 16 V. Findings and Conclusions.................................................................................................... 17 Evaluation Question 1 ............................................................................................................................................... 17 Findings..................................................................................................................................................................... 17 Conclusions ............................................................................................................................................................. 19 Evaluation Question 2 ............................................................................................................................................... 19 Findings..................................................................................................................................................................... 19 Conclusions ............................................................................................................................................................. 21 Evaluation Question 3 ............................................................................................................................................... 22 Findings..................................................................................................................................................................... 22 Conclusions ............................................................................................................................................................. 31 Evaluation Question 4 ............................................................................................................................................... 32 iv | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 Findings..................................................................................................................................................................... 32 Conclusions ............................................................................................................................................................. 34 Evaluation Question 5 ............................................................................................................................................... 35 Findings..................................................................................................................................................................... 35 Conclusions ............................................................................................................................................................. 37 VI. Recommendations .............................................................................................................. 38 VII. Annexes................................................................................................................................ 41 Annex A: Evaluation Statement of Work ............................................................................................................. 41 Annex B: Data Collection Protocols ..................................................................................................................... 42 Key Informant Interviews.................................................................................................................................... 42 Focus Group Discussions.................................................................................................................................... 43 Annex C: Documents Reviewed............................................................................................................................. 44 Annex D: Evaluation Respondents......................................................................................................................... 48 Key Informants....................................................................................................................................................... 48 FGD Respondents................................................................................................................................................. 49 Annex F: Evaluation Design Matrix ........................................................................................................................ 56 FIGURES AND TABLES Figure 1: Number of Beneficiaries by Value Chain and by Sex............................................................................. 17 Table 1: Key Informants by Type and Gender.......................................................................................................... 13 Table 2: FGD Groups by Value Chains...................................................................................................................... 14 Table 3: Costs and Benefits by Value Chain.............................................................................................................. 20 Table 4: Grant Budget by Value Chain ....................................................................................................................... 20 Table 5: Loans Awarded ................................................................................................................................................ 28 Table 6: Distribution of Grants by Processing Time............................................................................................... 30 Table 7: Number of Grants per Year......................................................................................................................... 30 USAID/LEBANON LIVCD FINAL EVALUATION REPORT | v NOVEMBER 2018 ACRONYMS AUB American University of Beirut BBS beneficiary-based survey BCR benefit-cost ratio CCIAT Chamber of Commerce Industry and Agriculture of Tripoli and North Lebanon CCIAZ Chamber of Commerce, Industry and Agriculture of Zahle CDCS Country Development Cooperation Strategy CLA Collaboration, Learning and Adaptation COP chief of party DAI Development Alternatives, Inc. DEC Development Experience Clearinghouse DO development objective DO2 second development objective EoI expression of interest EQ evaluation question ESS electrostatic spraying system ET evaluation team FG focus group FGD focus group discussion FtF Feed the Future FY fiscal year GDP gross domestic product GM gross margin GNI gross national income GOL Government of Lebanon Ha hectare ILO International Labor Organization IP implementing partner IPTT Indicators Performance Tracking Table IR intermediate result KI key informant KII key informant interview LIBNOR Lebanese Standards Institution LIVCD Lebanon Industrial Value Chain Development M&E monitoring and evaluation MFI microfinance institution MoA Ministry of Agriculture MoT Ministry of Tourism MSMEs micro-, small, and medium-size enterprises MTE midterm evaluation NGO nongovernmental organization PMP performance management plan PMSPL Performance Monitoring and Support Lebanon RfP request for proposals SI Social Impact SoW scope of work TA technical assistance TL team leader vi | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 UNDP United Nations Development Program USAID United States Agency for International Development USEK Holy Spirit University of Kaslik USG United States government VC value chain USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 1 NOVEMBER 2018 EXECUTIVE SUMMARY INTRODUCTION Executed by Development Alternatives, Inc. (DAI) from September 2012 to January 2019, the United States Agency for International Development (USAID)’s Lebanon Industry Value Chain Development (LIVCD) project is a $46.2-million activity that seeks to develop inclusive and competitive value chains to raise the incomes of the rural population in Lebanon, including micro-, small, and medium-size enterprises (MSMEs). Lebanon is an upper-middle-income country that has been experiencing a deep economic crisis since early 2012. Lebanese products and services suffer from a lack of competitiveness, largely due to the negative economic effects of 15 years of war and civil strife. Traditional markets and value chains have been disrupted, and private and public sector investment has stagnated. The impact on rural areas has been particularly acute as it has been exacerbated by an absence of public sector services such as agricultural extension. LIVCD aims “to develop functioning, competitive value chains to increase incomes of the rural population, including MSMEs.” To achieve its objective, LIVCD has collaborated with private actors within the selected value chains by providing technical assistance (TA), equipment, grants, and access to investment for value chain financing to overcome constraints to competitiveness and inclusiveness. The project targeted eight value chains: i) pome fruits, ii) stone fruits (cherries and avocados), iii) table grapes, iv) olive oil, v) rural basket of products (including honey, eggs, pine nuts, and herbs), vi) processed foods, vii) rural tourism and handicrafts, and viii) floriculture. In 2014, floriculture and handicrafts were excluded from LIVCD’s portfolio. This final evaluation documents LIVCD’s critical successes and shortcomings to provide the USAID Mission in Lebanon, the U.S. government (USG), and the wider donor community with usable lessons learned for adaptation under the USAID Collaborating, Learning, and Adapting (CLA) framework. Through five evaluation questions (EQs), this evaluation examines the project’s relevance, efficiency and cost￾effectiveness, effectiveness, and sustainability and additionality. It then provides conclusions and recommendations to inform future activities and follow-on actions. The key user is the economic growth team at USAID/Lebanon—specifically, the economic growth office, the program office, and others at the Mission’s discretion. The evaluation’s results will be used by USAID/Lebanon in its annual portfolio review. Using a theory-based, utilization-focused approach, the evaluation team (ET) employed a qualitative design consisting of the following data collection methods: document review, key informant interviews (KIIs), focus group discussions (FGDs), and observations. The ET also reviewed and used quantitative data (indicators and survey data) collected by the project. The team followed a purposive sampling approach for KIIs and FGDs and conducted a total of 64 KIIs (with 71 individuals) and 28 FGDs (with 221 individuals). Data triangulation was used to cross-check the results using several analysis methods, including content, contribution, comparative, and gender analysis, to provide evidence for the evaluation’s findings and conclusions. FINDINGS AND CONCLUSIONS EQ 1: To what extent were the value chain approach and LIVCD’s objectives and performance indicators appropriate for meeting the Country Development Coordination Strategy (CDCS) development objective (DO)? The value chain (VC) approach allowed the project to address all stages required to bring an agricultural product to the final consumer, allowing the project’s scope to target a wide range of economic actors. 2 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 The VC approach also allowed the project to design cost-reduction and quality-improvement interventions with the aim of creating a more competitive product in terms of cost and quality. Although processed foods and rural tourism are not value chains, they were included in the project because of their high potential to target women, unlike the male-dominated agriculture sector. In addition, the processed foods sector had a high potential to reduce poverty through supporting companies that create jobs and strengthening interventions for women’s cooperatives. CDCS Development Objective 2 (DO2) is “Inclusive economic growth enhanced.” Comparing LIVCD’s DO, goal, and intermediate results (IRs) with those of the CDCS shows that the project’s DO is well aligned with the CDCS. However, of the five LIVCD DO performance indicators, the indicator formulated as “number of jobs impacted” is very generic and not informative because it does not reflect job creation. In addition, although LIVCD activities aimed to strengthen business associations and policy advocacy— areas explicitly covered by the CDCS—these activities are not reflected by any of the project’s indicators. In conclusion, the VC approach aligned with DO2 since it was appropriate to improve the competitiveness of Lebanese products and enhance economic growth. The VC approach also made the interventions more inclusive by enabling LIVCD to target a wide range of economic actors. By adding rural tourism and processed foods to its portfolio, the project also enabled a greater inclusiveness of women. The project’s performance indicators were overall in line with the CDCS; however, the lack of an indicator reflecting job creation prevented a full appreciation of the project’s inclusiveness and contribution to the CDCS DO. EQ 2: To what extent were the selected interventions cost-effective? The ET calculated the benefit-cost ratio (BCR) using the LIVCD farmer survey’s results. The BCR was high for those interventions with avocado farmers (6.74) and olive farmers (3.01). For avocado, this was due to the training, which increased the yield. For olive oil, the high BCR was a result of the introduction of mechanical harvesters, which reduced harvesting costs. The BCR was low for cherry (0.11), mainly because of lower yields, and negative for apple (-0.46), mainly because of a lower sale price resulting from the devaluation of the Egyptian pound and limited export market opportunities apart from Egypt. The project asked grantees to contribute to the cost of grants. The grantees’ contribution was higher in the avocado value chain (160 percent of the awarded amount) and in the grape value chain (180 percent), while it was lower for honey (63 percent), rural tourism (65 percent), and processed foods (82 percent). However, the project TA inspired significant investments among beneficiary companies of the processed foods value chain. Also, the processed foods companies visited by the ET reported increases in both volumes produced and sales. For the cherry value chain, the grantees’ contribution was higher than the awarded amount; however, the incremental gross margin (GM) of cherry farmers was lower than the project’s direct cost. Rural tourism value chain beneficiaries reported that their jobs were not impacted by trainings but rather by the number of tourists visiting the target areas. Tourist numbers rose in the targeted areas, thus increasing the incomes of guesthouse owners, restaurants, and guides. However, since many other non￾LIVCD-funded interventions targeted the same areas, it is questionable to attribute the increase of tourists in target areas to LIVCD efforts. In conclusion, interventions in the avocado and olive oil value chains were more cost effective than those in the other agricultural value chains. Interventions in the cherry and apple value chains were not cost effective. Grantee contributions in the avocado and grape value chains added significantly to the interventions’ cost￾effectiveness. In the processed foods value chain, the grantee contribution was lower than the grant￾awarded amount. However, the TA-inspired investments among some of the beneficiary companies USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 3 NOVEMBER 2018 enlarged their production capacities. In addition, the fact that the beneficiary companies in the processed foods value chain increased sales and volume produced suggests that interventions in this value chain were also cost effective. EQ 3: To what extent have LIVCD’s interventions resulted in meeting the objectives and key performance indicators (KPIs) of the initial and extended contracts? Were the selected interventions and entry points in each value chain strategic in leading to the intended results? What internal and external factors contributed to the achievement (or nonachievement) of the project’s results? All project targets were achieved, with some exceptions for the GM indicators. The LIVCD farmer survey showed that changes in the GM per hectare have reached the 10 percent target increase for avocado, grape, and olive oil. The GM per hectare rose for cherry and honey, but the 10 percent increase was not met. The GM per hectare fell for apples. The target for women’s participation was achieved (23 percent of all beneficiaries were women). The FGDs revealed that the location and time of all training sessions conducted were suitable for women. However, women’s participation was low in all six agricultural value chains because agriculture is male-dominated and production is perceived as too tough for women. Interventions were devised to enhance farmers’ individual productivity in value chains. However, all categories of economic operators reported significant contextual challenges, which include: a small and fragmented land ownership structure, the farmers’ old age, the closure of land border crossings, the dubious quality of pesticides sold by input providers, the unregulated import of olive oil, and the Ministry of Agriculture (MoA)’s weak capacity to provide TA and the laboratory analyses needed for exports (Tripoli’s Chamber of Commerce provides all the required laboratory analyses to export fruits, but the fruit exporters we visited were using only MoA’s lab). With these conditions, it is very “difficult to beat the system” with training, TA, and the introduction of modern equipment. However, both the processed foods sector and the olive oil value chain showed some evidence of greater competitiveness and higher incomes. This was achieved by introducing technological innovations and developing market linkages for some processed food companies. Incomes in the avocado and grape value chains also rose. The FGDs and the LIVCD survey’s results for the GM suggest that apple value chain competitiveness did not improve, and that project benefits were lower for cherry value chain beneficiaries. The rising number of tourists visiting the areas targeted by the rural tourism value chain interventions generated higher incomes there. However, except for one village, the attribution of tourists increase to the project is questionable. The total value of loans raised by LIVCD beneficiaries is considerable ($18.8 million). However, the loans were awarded to a very small number of beneficiaries (102). The project developed feasibility studies for 27 beneficiaries, which resulted in 12 awarded loans. The fact that only one beneficiary company used the LIVCD feasibility study to raise loans suggests that the loans awarded to companies (40 percent of the total value of loans) resulted more from the investments leveraged by grants and TA than from the activities aimed at promoting access to finance. The combined use of TA and grants allowed the project to tailor interventions to the investment capacities of beneficiaries and to their risk aversion profile. The project partnered with cooperatives, private companies, and the Ministry of Tourism (MoT). Targeting cooperatives allowed LIVCD to reach a high number of farmers at a low cost. Targeting cooperatives was also strategic to establish service centers that deliver pruning services and manage mechanical harvesters, but not to improve access to output markets. The targeted private companies that provide agricultural production and post-harvest services were instrumental in developing the chosen value chains. MoT’s involvement was strategic to develop the 4 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 National Strategy on Rural Tourism. However, the fact that the political consensus so created was not used to develop the needed regulation was a missed opportunity. The beneficiary organizations of processed-foods interventions were also strategic to create employment opportunities for women (and some people with disabilities) and to provide a market outlet for minor quantities of low-quality grapes and apples. EQ 4: To what extent can LIVCD’s interventions be considered sustainable? The project design had several aspects related to ensuring sustainability, such as requesting the grantees to contribute to grant costs and an intervention strategy focused on the private sector. For the apple value chain, private sector involvement occurred during the final phase when a new large factory was established in Bcharre through LIVCD support. However, the factory foresees using just 30 percent of its capacity in 2018–19 because there are not enough quality apples in the region. Other activities were implemented in the same area, such as supporting an apple farmers’ cooperative and funding a small apple juice factory and an agro-input seller that also provides TA services to farmers. For apple farmers, continued access to TA is facilitated by the fact that Bcharre’s six municipalities are currently employing agronomists to provide technical advice to farmers. The municipality of Baskinta is also employing an agronomist to follow up with apple farmers. Specialized teams offering pruning services were formed following the training provided by LIVCD and the equipment delivered. For avocado farmers, TA in the South is provided by the experts that partnered with LIVCD throughout the project’s life. In the North the expert who was employed by the grantee left Lebanon. The ET found contrasting evidence on TA opportunities for avocado farmers, but in Akkar (only) an EU-funded project is providing TA to the same LIVCD beneficiaries. FGDs with avocado farmers revealed that avocados can be easily sold. LIVCD has also confirmed the growing demand for avocados. In the grape value chains, two of the three exporters supported by the project have agronomists to provide TA to farmers. They have also expressed their willingness to buy grapes from the farmers to whom they were linked once the new plantations increase production (in 2019). However, they also reported that they would not engage in a similar project again since following up with smallholders is too complicated and expensive. The third supported exporter is no longer in the grape business in Lebanon. LIVCD provided TA to grape farmers either directly or through the personnel employed through grants awarded to CCIAZ. However, the CCIAZ grape expert left three months before the end of the grant (given the short remaining period no replacement took place) and the EU grape project implemented through CCIAZ targets only part of the LIVCD grape beneficiaries. In honey, the use of TA services supported by the project depends largely on the distance from TA providers. Whereas four FGDs with beekeepers reported they were using TA and production services, the focus group in Baalbek-Hermel reported to have never used any TA service. In addition, the artificial insemination center for queen bees supported by the project is contested by academics. In rural tourism, the local project initiatives are expected to be maintained by the grantees. The increase in the number of tourists visiting the target rural areas suggests that the rise in tourism income motivates a continuation of the services delivered. An important activity promoted by the project was the development of the National Rural Tourism Strategy; however, the strategy is no longer on the policy agenda since the minister of tourism changed. The steering committee responsible for the implementation of the strategy is no longer active. An unplanned effect of the project was the establishment of four start-up companies that, at the time of the evaluation, were providing TA services to beekeepers and avocado farmers. They also maintain the roasting equipment and optical sorters delivered to food processing companies or cooperatives. In conclusion, the project design included aspects related to ensuring sustainability. For the apple value USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 5 NOVEMBER 2018 chain, this also included concentrating interventions in a single region. However, without clear improvements in produce quality, the sustainability of interventions is at risk. Continued access to TA is provided by two of the three grape exporters for 30 grape farmers only, and CCIAZ’s capacity to provide TA to grape farmers is limited. Access to TA is easier for those avocado farmers in the South, while ET found conflicting evidence of available TA opportunities for avocado farmers in Akkar. Interventions in the honey value chain resulted in some groups having easier access to TA and production services and others having none. For rural tourism, while local interventions have good prospects of sustainability, the development of the National Strategy did not prove to be sustainable. EQ 5: Were the positive results spread across all beneficiaries or mainly to few ones only? And to what extent did the interventions demonstrate additionality? LIVCD survey results showed that four categories of beneficiary farmers—grape, avocado, olive, and honey producers—experienced benefits that can be attributed to the project, while two categories— apple and cherry farmers, which constitute 17 percent of the beneficiary farmers—did not. Beneficiaries of the interventions in the processed foods and rural tourism sectors were not included in the LIVCD farmer survey. Five beneficiary companies (out of six interviewed by the ET) of the processed foods interventions reported an increase in the number of employees, sales, and volumes produced as a result of LIVCD support. Four companies also reported an increase in the number of farmers from whom they were buying produce (one company had just installed the equipment funded by USAID). The LIVCD impact study shows that incomes rose for the beneficiaries of 14 out of 18 rural tourism interventions. Four interventions did not deliver results, with no benefits accruing to beneficiaries. The project reached the target for women’s participation, but women’s participation was low in all agricultural value chains. A gender analysis was not systematically included in the initial value chain assessment studies. Also, the fact that LIVCD’s staff could not recall the Resource Guide for Gender Integration in Value Chains developed by PMPSL II for the Mission indicated that the resource guide was not used. Interventions promoting avocado cultivation were additional because avocado was introduced in new areas and because farmers reported having learned new techniques. Grape farmers also reported that the LIVCD contribution was essential to establish new plantations and install drip irrigation systems. Beekeepers reported mixed considerations on the additionality of beehive distribution, with some reporting that no hives would have been bought without LIVCD’s support and others stating that hives are affordable and would have been bought without support. Beneficiaries of the technological innovations introduced by LIVCD stated that their investments would not have been possible without this assistance. These investments included mechanical harvesters, cucumber optical sorters, tank fermenters for pickles, freekeh roasters, and a beeswax recycling center. Mixed considerations were reported by the companies that received grants for post-harvest services, with two of them stating that without LIVCD’s assistance there would have been no (or very different) investments, and two others reporting that the investments were already planned but that LIVCD had accelerated the process. Replications occurred for the mechanical harvesters for olives, the cucumber optical sorters, and the freekeh roasters. In conclusion, benefits were spread across all beneficiaries except for apple and cherry farmers, who are a minority of the beneficiary farmers. However, across the six agricultural value chains, more benefits accrued to men than to women because the percentage of male farmers is much higher than that of female farmers. This is mainly explained by the fact that agriculture in Lebanon is a male-dominated environment. Interventions in the processed foods sector created jobs and increased the number of farmers supplying the companies and the quantity of raw material bought. The additionality of grants and TA was evident 6 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 for interventions introducing technologies hitherto unknown to beneficiaries. Grants for avocado and grape farmers can be considered additional. Additionality was less evident for grants awarded to well￾established companies and that did not introduce innovations. Also, hive distribution was not completely additional. RECOMMENDATIONS 1. Continue using grants to introduce innovations among risk-adverse beneficiaries and provide TA to well-established companies. 2. Prioritize interventions aimed at developing the Bcharre apple value chain to make Bcharre a model to be showcased and replicated. 3. Consider providing support to develop regulations with the purpose of promoting an enabling environment for private sector development projects. 4. Continue targeting both private companies to improve access to international markets and cooperatives to establish service centers for input provision and TA. 5. In future private sector development agriculture-related activities, consider capturing the income generated, employment created, and changes in the enabling environment. 6. Seek assistance from external and independent experts on the relevance and sustainability of risky and contested grant proposals when dealing with interventions with high scientific content. 7. Consider further interventions in the cherry value chain. 8. Consider developing interventions to promote avocado exports. 9. Ensure that agriculture project contractors apply a gender integration strategy in all project aspects. 10. Consider ways to support the Ministry of Agriculture in key policy areas. I. INTRODUCTION Executed by Development Alternatives, Inc. (DAI) from September 2012 to January 2019, the United States Agency for International Development (USAID)/Lebanon Industry Value Chain Project (LIVCD) is a $46.2 million activity that sought to develop inclusive and competitive value chains to raise the incomes of the rural population, including micro-, small, and medium-size enterprises (MSMEs). This report details the results of a final performance evaluation of LIVCD, conducted by Social Impact, Inc. (SI) for USAID/Lebanon as part of the Performance Monitoring and Support Lebanon (PMSPL II) project. II. THE DEVELOPMENT PROBLEM AND USAID’S RESPONSE THE DEVELOPMENT PROBLEM The gross national income per capita in Lebanon in 2017 was $13,040,1 ranking Lebanon as an upper￾middle-income country. Poverty in Lebanon was estimated at 27 percent as of 2011.2 According to a 2004–05 United Nations Development Program (UNDP) study—which continues to be used by policymakers to design poverty reduction policies and programs)3—more than 20 percent of households engaged in agriculture are below the poverty line.4 (In 2016, UNDP conducted a new rapid poverty assessment in Lebanon, but results are not yet available.) Gross domestic product (GDP) growth was 2.8 percent in 2012 and constantly declined through 2015, reaching 0.8 percent. In 2016 and 2017, GDP growth was 2 percent.5 The contribution to the GDP of the value added produced by the agriculture sector was 3.7 percent in 2012, 4.1 percent in 2014, and 3.5 percent in 2016 and 2017. According to International Labor Organization (ILO) estimates, the percentage of employment in agriculture was 3.2 from 2014 to 2017. 5 The 2010 LIVCD request for proposals (RfP) emphasizes that many Lebanese products and services suffer from a lack of competitiveness, due largely to the negative economic effects of 15 years of war and civil strife. Traditional markets and value chains have been disrupted, and private and public sector investment has stagnated. The impact on rural areas has been particularly acute as it has been exacerbated by an absence of public sector services such as agricultural extension. Many rural areas have as a result become increasingly marginalized and economically isolated. When the project was designed, the LIVCD RfP identified the following gaps constraining the capacity of rural value chain actors to compete in international or even domestic markets: i) the lack of extension services; ii) the absence of municipal revenue to fund market infrastructure; iii) the limited access to credit in many rural areas; iv) the lack of market intelligence to inform decision making and preserve capital; v) 1 Current prices, WB Atlas method. [Online] Available from: http://databank.worldbank.org/data/home. 2 UNDP. (2016). Rapid poverty assessment for Lebanon in 2016. Press release. [Online] Available from: http://www.lb.undp.org/content/lebanon/en/home/Response_to_the_Syrian_Crisis/successstories/Rapid-Poverty￾Assessment-in-Lebanon-for-2016.html 3 Nuktrety N. & Al Jamal S. (2016). Poverty Inequality and Social Protection in Lebanon. American University of Beirut and Oxfam. [Online] Available from: http://website.aub.edu.lb/ifi/publications/Documents/research_reports/20160426_poverty_inequality.pdf 4 UNDP (2008). Poverty, Growth, and Income Distribution in Lebanon. UNDP. [Online] Available from: http://www.undp.org/content/dam/lebanon/docs/Poverty/Publications/Poverty,%20Growth%20and%20Income%20D istribution%20in%20Lebanon.pdf 5 World Bank. [Online] Available from: Available from: http://databank.worldbank.org/data/home. 8 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 the lack of transparency in value chain market information and transactions; vi) the high production costs, vii) the high transaction costs; viii) the general absence of appropriate post-harvest handling, causing large crop losses; and ix) the lack of agricultural product sorting, grading, packaging, and cold storage facilities in some areas. Two types of agriculture are predominant in Lebanon: commercial agriculture, which uses modern production and post-harvest techniques for commodities destined for export and domestic markets, and family agriculture, which is devoted to preserving cultural and family heritage. The agriculture sector is male dominated. The ILO employment estimates from 2016 show that4.2 percent of men were employed in the agriculture sector, while just 0.045 percent of women employment was in that sector. However, this gender gap explains the male dominance in the agriculture sector only to a very limited extent. In 2016, males over 15 constituted 71 percent of the labor force, whereas women over 15 constituted 23 percent.6 Moreover, the LIVCD midterm evaluation analysis showed that women are paid less than men in agriculture. For example, in harvesting work women’s daily wage is 65 percent lower than that of men. Food exports represented 14.9 percent of total merchandise exports in 2010 and 29.9 percent in 2016. 5 These figures indicate the increasing importance of the agro-food sector among exports. Overall, agricultural production depends largely on the immigrant workforce (mainly Syrian), especially for casual labor. The midterm evaluation reports that in the LIVCD target crops, the average number of non-Lebanese workers during harvest operations per farm is more than double the number of Lebanese workers. USAID’S RESPONSE The LIVCD project’s RfP emphasizes that the economic growth the country has been witnessing since 2007 has benefited many sectors of the national economy. However, rural areas, which depend primarily on agriculture and, to some extent, tourism, did not benefit much. They continued to suffer from limited private sector investment and access to credit. The RfP also stresses that a lack of adequate value chain finance is cited in several reports as an important factor limiting famers’ ability to participate in high value chain production. Lebanon has significant natural resources and advantages, including sizeable expanses of fertile soils, a multitude of micro-climates, the capacity to produce early season fruits and vegetables, a close proximity to Gulf and European markets, abundant natural landscapes and heritage, ecological diversity, a multitude of religious and historical sites, a liberal business environment, a large entrepreneurial population, and a large Lebanese diaspora. Lebanon has a potential competitive advantage in fresh produce (fruits and vegetables) and agro￾processing. The comparatively underdeveloped state of Lebanon’s ecological, historical, and religious assets suggests that additional opportunities to create rural wealth exist in rural tourism, including agro￾tourism. USAID’s LIVCD project was originally designed as a five-year project to respond to the above-mentioned sector problems. USAID signed the LIVCD contract (No. AID-268-C-12-00001) with DAI in September 2012 for an initial value of $41,682,272 until September 30, 2017. A contract modification was made on June 23rd, 2017, extending LIVCD by 16 months until January 2019 and increasing the budget by $4.5 million, thus totaling $46,182,272. 6 ILO estimate. [Online] Available from: http://databank.worldbank.org/data/home. USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 9 NOVEMBER 2018 The LIVCD project was designed to partner with the private sector and to apply the value chain approach to enhance inclusive economic growth, increase private sector competitiveness, and improve the incomes of the rural population. No theory of change or development hypothesis was used to design the LIVCD project. A development hypothesis was later added during the development of the LIVCD monitoring and evaluation plan. The hypothesis states that “If actors, including micro, small, and medium enterprises (MSMEs) in the target value chains increase investment into their businesses to increase productivity, make new business linkages, and identify and enter new markets, then the value chain will be more competitive and MSME incomes will rise.” The project’s purpose is “to develop functioning and competitive value chains to increase the incomes of the rural population, including MSMEs.” A functioning value chain is defined in the contract as a competitive and inclusive value chain. Competitiveness is thus measured by the increase in sales and productivity and the improvement of product quality, and inclusiveness by the number of value chain participants, including MSMEs, farmers, and other organizations receiving assistance. To achieve its objective, LIVCD collaborates with actors within the selected value chains by providing technical assistance (TA), equipment, grants, and access to investment for value chain financing to overcome constraints to competitiveness and inclusiveness. The LIVCD activities are designed to achieve the following four LIVCD intermediate results (IRs): • IR1: Increased access to markets in target value chains. • IR2: Increased business linkages in target value chains. • IR3: Increased productivity in target value chains. • IR4: Constraints to lending and investment reduced in target value chains. According to the contract modification signed in 2017, it is expected that by the end of the project the following results will be achieved: • A minimum of seven functioning competitive value chains. • At least 17,000 MSMEs, including farmers and other organizations receiving business development services. • At least 12,500 small and medium-size commercial growers benefiting from the dissemination of improved production and post-harvest technologies, and MSMEs who have applied improved technologies or management practices. • At least 1,250 MSMEs, including farmers, reached by LIVCD to apply for value chain finance. • At least $30 million of new private sector investment leveraged by LIVCD, including investment by farmers and other value chain actors. • An increase of at least 5,000 businesses, farmers, or micro-enterprises benefiting from horizontal and vertical linkages. • At least 150 new strengthened export markets, niche markets, or distribution channels for selected value chain products. The LIVCD contract scope of work (SoW) specifies that the contractor is expected to undertake tasks under the following three components: 1) Conduct a complete assessment of the prospects for rural value chains in Lebanon based on secondary and primary market research, analysis of the current environment for value chains in 10 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 Lebanon, and a reconciliation of market intelligence with the capacity of Lebanon’s rural sector to select the value chains for LIVCD intervention. 2) Undertake value chain upgrading activities to increase the competitiveness of selected Lebanese value chains that have the potential to raise incomes of the rural population, supply key markets, and increase export growth and/or the number of rural tourists. 3) Assist in accessing value chain finance for producers and other value chain actors in the rural economy. In May 2013, eight value chains were approved for inclusion in the project: i) pome fruits, ii) stone fruits (cherries and avocados), iii) grapes, iv) olive oil, v) rural basket of products (including honey, eggs, pine nuts, and herbs), vi) processed foods, vii) rural tourism and handicrafts, and viii) floriculture. In May 2014 LIVCD, with the approval of USAID, eliminated floriculture from LIVCD’s portfolio. The project was implemented in all governorates of Lebanon. III. EVALUATION PURPOSE, USERS, AND QUESTIONS EVALUATION PURPOSE AND USERS This final evaluation documents LIVCD’s critical successes and shortcomings to provide the Mission, the U.S. government (USG), and the wider donor community with usable lessons learned for adaptation under the United States Agency for International Development (USAID) Collaborating, Learning, and Adapting (CLA) framework. The evaluation examines the project’s relevance, efficiency and cost-effectiveness, effectiveness, and sustainability and additionality. It then provides conclusions and recommendations to inform future activities and follow-on actions. The key user is the economic growth team at USAID/Lebanon—specifically, the economic growth office, the program office, and others at the Mission’s discretion. The evaluation results will be used by USAID/Lebanon in its annual portfolio review and by the LIVCD implementer, DAI, to inform future program implementation. The Government of Lebanon and key donors might also use the evaluation recommendations to design new interventions in the sector. EVALUATION QUESTIONS The LIVCD final evaluation scope of work contained eight illustrative evaluation questions (EQs), exceeding the recommendations provided by the relevant USAID How-To Note.7 Moreover, some of the EQs were not researchable because they asked what the result could be if an alternative approach were implemented, with no actual implementation of the envisaged alternative approach.8 The evaluation team worked with USAID to validate and improve the EQs. In agreement with USAID/Lebanon, the original set of questions was reformulated into the following five questions: 7 USAID. (2016). How-To Note: Evaluation Statement of Work. Program Cycle Guidance. [Online] Available from: https://usaidlearninglab.org/library/how-note-evaluation-statement-work 8 An example is “Was the project able to achieve more value for money than alternative approaches (achieve the same results at a lower cost)? If yes, what would have been the alternatives?” USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 11 NOVEMBER 2018 Relevance 1. To what extent were the value chain approach and LIVCD’s objectives and performance indicators appropriate for meeting the Country Development Cooperation Strategy (CDCS) development objective? Efficiency and cost-effectiveness 2. To what extent were the selected interventions cost effective? Effectiveness 3. To what extent have LIVCD’s interventions resulted in meeting the objectives and key performance indicators (KPIs) of the initial and extended contract? a. Were the selected interventions and entry points in each value chain strategic in leading to the intended results? b. What internal and external factors contributed to the achievement (or nonachievement) of the project’s results? Sustainability and additionality 4. To what extent can LIVCD’s interventions be considered sustainable? 5. Were the positive results spread across all beneficiaries or only a few? And to what extent did the interventions demonstrate additionality?9 EVALUATION TEAM AND SUPPORT STRUCTURE SI’s Performance Monitoring and Support Lebanon (PMSPL) II fielded a two-person evaluation team possessing a complementary mix of evaluation, value chain analysis, agricultural, and methodological knowledge. The team leader (TL), Matteo Borzoni, is an expert in value chain development and agro￾economics. He has a strong interdisciplinary background and 15 years of experience as a consultant, researcher, and project manager for a wide range of donors, including USAID. Since 2002, Dr. Borzoni has been involved with implementations and evaluations of value chain development projects in Latin America, the Middle East, Africa, and Asia. Joanna Khater, the local evaluation expert, has conducted evaluations and monitoring assignments for USAID since 2013. Both the TL and the local expert have participated in the midterm evaluation. Harvey Herr, PMSPL II’s chief of party (COP); Carine Khoury, PMSPL II’s deputy COP; and Najwa Andraos, PSMPL II’s gender specialist, provided advice and ad hoc inputs throughout the evaluation. 9 USAID requires grants and other interventions, especially to private sector entities, to demonstrate "additionality," or evidence that similar outcomes targeted by the grant or intervention would not have been realized in the absence of the USAID intervention. Simply citing the likely benefits and outcomes attributed to the grant/intervention is not sufficient justification, as it does not meet the additionality requirement. Evidence of additionality is normally provided by offering evidence that the purposes or proceeds of the grant/intervention: 1) could not, or likely would not, have been financed or purchased by the proposed beneficiary without USAID support; 2) serve a risk-reducing purpose, catalyzing the beneficiary to make an investment in a mostly unfamiliar technology with locally unproven benefits; 3) produce a demonstration effect, meaning that the success of the grant will induce significant numbers of others to copy the intervention, thus magnifying its impact; 4) allows the beneficiary to access finance that they otherwise could not have obtained; 5) crowds in outside resources/investment by offering a missing component to a transaction that induces other parties to participate and dedicate resources to the activity; 6) adds a noncommercial but important development component to a larger activity. 12 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 PMSPL II subcontracted InfoPro, a Lebanese research firm, to conduct focus group discussions (FGDs) and provide support in qualitative analysis. The firm consists of researchers with experience using focus groups in different sectors. Each focus group was facilitated by a moderator, who was supported by a note taker. InfoPro also contracted Dr. Ali Chalak, an associate professor from the Faculty of Agriculture at the American University of Beirut (AUB), to analyze the FGD results. IV. EVALUATION METHODOLOGY The evaluation team took a theory-based, utilization-focused approach. Theory-based evaluation focuses on providing an in-depth analysis of a program’s underlying logic and causal linkages. It recognizes that a multitude of factors and interactions influence a project’s impact and looks to identify causal factors judged to be most critical to a project’s overall success. A utilization-focused approach is based on the principle that evaluations should be judged on their usefulness to their intended users; therefore, they should be planned and conducted in ways that enhance the likely utilization of both the findings and the process itself to inform decisions. This final evaluation employed a qualitative design consisting of the following data collection methods: document review, key informant interviews (KIIs), FGDs, and observations, described below. A structured survey (with quantitative data) was used for the midterm evaluation. In addition, LIVCD conducted a large survey (with a sample size of 907 farmers) to estimate the gross margins of beneficiaries. The final evaluation used the survey conducted at the time of the midterm evaluation and the results data made available by the project (including the LIVCD survey conducted to estimate gross margins). In this way, the findings obtained by the qualitative design of this evaluation were complemented by a quantitative analysis of already available datasets and project indicators. Interventions were assessed in all value chains except rural baskets and floriculture. The latter was eliminated from LIVCD’s portfolio at the beginning of the project. The rural baskets value chain was not assessed upon request from USAID, given that interventions under this category were mainly about livelihood diversification rather than development of competitive agricultural value chains. DATA COLLECTION METHODS Document Review The ET benefited from a relatively rich project document set. Before beginning fieldwork, the team reviewed the evaluation’s SoW; LIVCD’s project description, results framework, and monitoring and evaluation (M&E) plan; project studies (market studies, initial value chain assessments, impact assessments); annual and quarterly reports; and project contract documents and workplans. These documents provided a useful overview of the diversity of project activities. In addition, the ET used the farmer survey conducted by LIVCD (which compares the 2017–2018 and 2013–2014 agricultural seasons) to analyze cost-effectiveness (EQ2) and inform EQ3 and EQ5. The ET also reviewed a prioritized list of non-project and secondary documents (Annex C), mainly about projects in Lebanon focused on agriculture, agricultural value chains, agribusiness, and rural tourism. The desk review presented a critical opportunity for the team to identify, select, and refine the list of interviewees and site selection and develop draft data protocols and interview guides. USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 13 NOVEMBER 2018 Key Informant Interviews The ET used a purposeful sampling approach10 to identify KII candidates. Although the selection of key informants (KIs) was informed by the literature review, the ET also applied other criteria, including the KIs’ relative positions of authority within their respective organizations/communities, the degree to which they were beneficiaries of LIVCD support, and the value of the responses they were likely to provide to the evaluation effort. KII protocols (Annex B) consisted of interview topics and questions derived from the EQs, as well as from the ET’s document review, discussions with USAID and LIVCD staff, and the latter’s evaluation design knowledge. The ET conducted the KIIs at the locations selected by (and therefore convenient for) the interviewees. A total of 64 KIIs (including one-on-one and group interviews)11 were conducted with 71 individuals. The KIs for this evaluation are disaggregated below in Table 1 by respondent type and gender. Table 1: Key Informants by Type and Gender Key Informant Type Male Female TOTAL USAID 1 - 1 LIVCD 4 10 14 Representatives of beneficiary organizations of TA and grants 31 5 36 Individual beneficiaries 5 2 7 Bank 1 - 1 Other 10 2 12 Total 52 19 71 Focus Group Discussions (FGDs) The LIVCD objective clearly states that the project’s ultimate purpose is to increase income. Among all categories of beneficiaries included in the beneficiary list, focus groups (FGs) were organized with those that could raise their income by participating in the project. More specifically, FGDs were conducted with farmers only, thus excluding beneficiaries of the rural tourism and processed foods interventions. Activities in these sectors were tailored to the needs of beneficiary organizations. In addition, there is no single category of beneficiary in the processed foods and rural tourism sectors. The ET believes that FGDs are not the right tool to collect information about the project performance for the processed foods and rural tourism sectors, since beneficiaries and interventions are too heterogeneous (information from the beneficiaries of these two sectors was collected only through KIIs with individual representatives of the beneficiary organizations). For this final evaluation, InfoPro, a Lebanese research firm, was contracted to conduct FGDs with farmers. Each FGD was composed of farmers from the same value chain. InfoPro was instructed to ensure that women, men, and youth were represented in each FGD and engaged equally in the discussion. Based on previous experience in the country, the ET determined that 10 Purposeful sampling is a nonprobability sampling technique that relies on the researcher’s judgment to select the analysis units. It is widely used in qualitative research for the identification and selection of information-rich cases. This involves identifying and selecting individuals or groups of individuals that are especially knowledgeable about or experienced in a phenomenon of interest. 11 Group interviews included two or more respondents but followed the KII protocols. 14 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 participation of both men and women in the same FGD would not inhibit free and open participation of all, since no sensitive issues were discussed. InfoPro and the ET made every effort to remove any obstacles that would prevent women from taking part in the KIIs and FGDs (including avoiding conducting KIIs and FGDs after 5:00 pm and on Sundays and holding meetings in locations where women feel comfortable). FGD protocols (Annex B) consisted of questions that were derived from the EQs, the ET’s document review, and discussions with LIVCD staff. All FGDs followed a common format and lasted between one and two hours. InfoPro conducted a total of 28 FGDs with groups across all value chains, as detailed in Table 2 below. The FGDs included a total of 221 participants (197 males, 24 females) and had an average of 8 participants each (ranging from 5 to 12 individuals). Using the information in the LIVCD beneficiary list, participants were selected based on the value chain and on the grantees or companies they were linked to as reported in the LIVCD database. Table 2: FGD Groups by Value Chains Value chain Number of FGDs Apple 5 Avocado 4 Cherry 4 Grape 5 Honey 5 Olive oil 5 Total 28 Despite all the efforts made by InfoPro to invite women, the number of female participants was low, reflecting the fact that in Lebanon the agriculture sector is mainly male dominated (see Section II) and that female farmers were a small minority of LIVCD beneficiaries. LIVCD had a relatively larger number of female beneficiaries in the rural tourism and processed foods sectors (see Section V). However, the number of female beneficiaries was much lower in the six agricultural value chains for which FGDs were held. To properly capture the project’s effect on the beneficiaries that could raise their income as a result of the project’s activities, the beneficiary sampling frame was filtered to retain only farmers, thus excluding nonfarmer beneficiaries (both the midterm evaluation and the survey planning and sampling document of the LIVCD farmer survey noted that several nonfarmer beneficiaries were included in the beneficiary list). As a result, the number of potential female farmers to be called to participate in the FGDs was further reduced. Observations Because KIIs were also conducted with farmers across the country (who were grantee representatives), the ET requested permission to observe demo plots and equipment delivered by the project. They took notes and photos and used these observations to confirm the details shared during the KIIs and the information found in the project’s documents. DATA ANALYSIS The ET used data triangulation to cross-check results and different analysis methods to provide evidence for the evaluation’s findings and conclusions. USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 15 NOVEMBER 2018 Data Analysis Processes The ET took notes during the KIIs, FGDs, and observation visits and shared reflections on each data collection activity with other team members within two days of the activity. During these sessions, called internal debriefs, the team discussed the collected evidence, patterns, and discrepancies, which helped answer the evaluation questions. After completing fieldwork, the team cleaned and shared electronic summaries of the interview notes. During analysis, data were disaggregated by sex and value chain, when possible, to capture any differences across these categories. The team entered the preliminary findings and conclusions into a matrix that organized the analyses and recommendations by question. The matrix served as a basis for the preliminary findings’ presentation to USAID on October 5, as well as for the evaluation report. Data Triangulation The ET used analytical triangulation approaches to develop findings and come up with conclusions. Triangulation enabled the ET to cross-verify and cross-validate findings to determine the program’s overall effectiveness. In particular, the ET used methodological triangulation to develop parallel protocols with the same or similar questions across KIIs and FGDs. This enabled greater data triangulation because each method addressed subsets of the same evaluation questions, and their findings were validated or refuted by other techniques (findings that were refuted by other techniques were not retained). Methodological triangulation also enabled the ET to strengthen the potential linkages and accuracy of the data if the results obtained through one method were less conclusive than those obtained from another. Data Analysis Methods The ET employed several data analysis methods to identify key findings from the collected data and draw conclusions and make recommendations on sustaining LIVCD’s positive results. Analysis methods included the following: 1. Content Analysis – The ET conducted content analysis through intensive review of the collected KII and FGD data to identify and highlight notable examples of LIVCD successes (or lack thereof) that contributed to (or inhibited) LIVCD contributions to the identified objectives. The findings were triangulated with project documents/data and observation results. 2. Contribution Analysis – Contribution analysis is an approach for assessing and inferring causality in program evaluations. It was conducted using FGD and KII data, focusing on questions that asked respondents about the changes they had observed since the beginning of their engagement with LIVCD. 3. Comparative Analysis – The ET also conducted comparisons of LIVCD results across stakeholder categories, value chains, and gender groups to assess convergence or divergence in perspectives (triangulation). 4. Gender Analysis – Gender analysis included disaggregation of data by gender and analyzing the effects of the project on men and women. Consistent with USAID’s evaluation policy and recognizing that integration’s effects and the project’s success might vary according to gender, SI applied a gender perspective and ensured that research activities included female participants. Therefore, the FGDs and KIIs included questions to assess the extent to which women beneficiaries were involved in the development of the target value chains. 16 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 5. Quantitative analysis – The ET used the LIVCD farmer survey results and dataset and the LIVCD cost data to assess the cost-effectiveness of LIVCD’s interventions by value chain and triangulate the findings from the FGDs and KIIs. BIASES AND OTHER LIMITATIONS USAID/Lebanon and other donors have been assisting Lebanon’s agriculture sector for many years. The multiplicity of donors and implementers actively working in this sector makes it challenging to assess attribution. The evaluation therefore focused less on attribution and more on LIVCD’s implementation, approach, and contributions. During fieldwork and data analysis, the ET was aware of several limitations and bias risks. The team was especially keen on mitigating the below-listed risks. • Quality of the LIVCD farmer survey: The ET used the LIVCD farmer survey to analyze cost￾effectiveness and triangulate and complement the findings for EQ2 and EQ5. Data were collected in September 2018 by using LIVCD staff as enumerators. In addition, the farmer survey conducted by LIVCD is a beneficiary-based survey (BBS), and its design consists of a before-and-after comparison. This design means that the project performance is evaluated by comparing outcome variables (e.g., gross margin) before LIVCD with the same outcome variable after LIVCD for the same group of farmers. No control group is used. Consequently, any difference in the outcome variable could be the result of positive or negative spontaneous dynamics or other causes that have nothing to do with the project (e.g., rainfall, price changes at an international level, higher or lower production due to alternate bearing) 12 rather than the result of project interventions. Finally, the LIVCD questionnaire asked farmers to recall production costs and revenues for both the 2017–2018 (i.e., the endline) and the 2013–2014 (i.e., the baseline) agricultural seasons. Recall error is a possibility when farmers are asked to report information dating back four years. To mitigate these limitations, the ET triangulated the results of the LIVCD farmer survey with the findings derived from other sources, such as the midterm evaluation survey, the KIIs, and secondary sources. • Recall Bias: Recall bias is a common evaluation problem. LIVCD beneficiaries, in certain instances, responded to questions posed by the ET with answers that blended their experiences into a composite memory. Respondents who participated in more than one LIVCD activity or in similar activities conducted by other implementers were not always able to distinguish LIVCD activities (trainings, events, meetings) as separate activities. The ET mitigated this risk by a) asking the question to multiple group members, b) asking questions using various terms for “training,” and c) conducting as many KIIs and FGDs as necessary within the available evaluation period to triangulate responses and increase the validity of the evaluation findings. • Response Bias: Response bias is the risk of key informants being motivated to provide responses that would be considered socially desirable or influential in obtaining donor support. Response bias is also connected to cultural and social norms and impacted by gender and social ranking. This risk was identified during the FGDs. If the first person who speaks in an FGD is the most senior in the group, other participants might take their cues from this person and only echo his/her responses. The research firm that conducted the FGDs effectively probed the participants on programmatic challenges and limitations (asking specific questions about desired alternative courses of action), thus minimizing this bias. • Selection Bias: Selection bias is an inherent risk whereby implementers help to facilitate contact with project beneficiaries. The ET and the research firm that conducted the FGDs selected 12 Alternate bearing refers to the tendency of an entire tree to produce a greater than average crop one year, and a lower than average crop the following year. USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 17 NOVEMBER 2018 respondents from the complete sampling frame of LIVCD’s beneficiaries to ensure balanced responses. V. FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1 To what extent were the value chain approach and LIVCD’s objectives and performance indicators appropriate for meeting the CDCS development objective? Findings The 2014–2018 Country Development Cooperation Strategy (CDCS) has two development objectives (DOs). The first DO is about the public sector’s capacity and is not relevant for LIVCD. The second (DO2) is “Inclusive economic growth enhanced.” The LIVCD RfP identifies Lebanese products’ lack of competitiveness as a limiting factor for economic growth. This determination is confirmed by the value chain assessment studies conducted at the beginning of the project. In principle, the value chain approach addresses all the stages required to bring an agricultural product to the final consumer, leaving the project enough scope to target a wide range of economic actors and allowing the project to design cost reduction or quality improvement interventions for different categories of beneficiaries, with the ultimate aim of creating a more competitive product in terms of cost and quality. LIVCD targeted eight value chains, including processed foods and rural tourism. As noted in the midterm evaluation, processed foods and rural tourism are not value chains per se, but sectors. As explained in Section II, the agriculture sector is mainly male dominated, whereas the processed foods and rural tourism sectors were reported by LIVCD’s staff to have the highest potential to target women. Figure 1 depicts the number of beneficiaries by value chain in August 2018. Olive oil, rural tourism, and processed foods presented the highest number of female beneficiaries. The FGDs with olive farmers revealed an important role of women in olive production and managing cooperatives. However, the olive oil value chain also included people that took 163 1,160 95 1,323 187 1,375 512 4,206 1,358 5,517 216 1,895 880 670 377 860 932 1,030 0 2,000 4,000 6,000 N of Beneficiaries Avocado Cherry Grape Honey Olive Oil Pome Fruits Processed Food Rural Basket Rural Tourism Source: LIVCD Monitoring system Female Male Figure 1: Number of Beneficiaries by Value Chain and by Sex 18 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 part in oil tasting events, which were conducted by targeting associations of women who were not farmers. In addition, the midterm evaluation survey showed that only 22 percent of olive farmers stated that farming is their main source of income. This finding was confirmed by an academic, who reported that in Lebanon olive orchard owners inherited the land but do not often farm it. Rather, the tenant is in charge of farming and marketing the produce. So it is possible that the female beneficiaries reported in high numbers in the olive oil value chain are not actually farmers. In addition, the LIVCD staff reported that the processed foods sector also has a high potential for poverty reduction by supporting companies that create jobs and strengthening women cooperatives. The LIVCD DO, formulated as “to develop functional competitive value chains to increase incomes of the rural population including MSMES,” had five indicators. Indicator 4, formulated as “number of jobs impacted,”13 did not require LIVCD to report on the number of new jobs created but rather allowed LIVCD to count farmers and beneficiaries of rural tourism and processed foods whose jobs were assumed to have improved after an intervention, which also included a training in production techniques. The LIVCD M&E plan compares the CDCS DO2, IRs, and sub-IRs with the LIVCD DO, goal, and IRs (with their indicators). The CDCS has two IRs14 and five sub-IRs.15 All IRs and sub-IRs under the DO2 are covered by LIVCD’s indicators and goals with the following exceptions: • Even though LIVCD had no indicators covering CDCS sub-IR 2.1.1 (strengthened business association services and policy advocacy), the project conducted activities in the area covered by this sub-IR, like organizing round tables on apples, developing the National Rural Tourism Strategy, and supporting the Lebanese Standards Institutions to develop new standards for honey. However, there were no indicators to reflect policy advocacy activities or contribution to changes in sector regulation and policy aspects. • LIVCD had no indicators for sub-IR 2.2.1 (strengthened microfinance association and member institutions). However, activities on microfinance were already covered by another USAID-funded project.16 In addition, microfinance institutions (MFIs) provide only short-term financing, whereas the targeted agricultural value chains mainly need long-term financing to establish new orchards. So sub-IR 2.2.1 is not relevant to LIVCD. Before the midterm evaluation, LIVCD was simply counting the number of people who took part in training sessions, which also included a presentation on loan products delivered by a bank credit officer (such presentations were generally included before or after training sessions on agricultural technical aspects). LIVCD added two indicators to properly cover the performance of the project for the access to finance area, which is covered by IR 2.2 of the CDCS. Those two indicators count the number of farmers or enterprises that successfully accessed loans and the value of loans issued as a result of U.S. government assistance. 13 The other four indicators are i) number of MSMEs, including farmers, and other organizations receiving business￾development services from USG-assisted sources; ii) value of new private sector investment leveraged by USG assistance; iii) value of investment made by farmers and other value chain actors leveraged by LIVCD implementation; iv) proportion of female participants in USG-assisted programs designed to increase access to productive economic resources (assets, credit, income or employment). 14 IR2.1 is “Improved private sector competitiveness” and IR2.2 is “Increased access to finance, especially for business start-ups and women entrepreneurs.” 15 Sub-IR 2.1.1 is “Strengthened business association services and policy advocacy”; Sub-IR 2.1.2 is “Increased business and trade linkages”; Sub-IR 2.1.3 is “Increased workforce development, linked to job opportunities.” Sub-IR 2.2.1 is “Strengthened microfinance association and member institutions,” and Sub-IR 2.2.2 is “Increased early-stage investment financing for new and existing firms.” 16 Livelihoods and Inclusive Finance Expansion (LIFE) project. USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 19 NOVEMBER 2018 Conclusions The value chain approach was in line with the CDCS DO2 because it was appropriate to improve the competitiveness of Lebanese products and ultimately enhance economic growth. The value chain approach also enabled LIVCD to target a wide range of economic actors, thus making the interventions more inclusive. Adding the rural tourism and processed food sectors to the target value chains enabled LIVCD to include more women—important because the agriculture sector in Lebanon is mainly male dominated. LIVCD showed adaptive capacity by adding two indicators (in the second half of the project’s duration) to reflect the project’s performance on access to finance, which is part of the CDCS. However, a full appreciation of the project’s contribution to the CDCS was limited by the lack of indicators reflecting the number of jobs created and by changes in the business association services, regulatory aspects, and policy advocacy. More specifically, indicators that reflect the number of jobs created would have provided information on the activity’s inclusive character, which is clearly embedded in the definition of the CDCS DO2, while indicators that reflect changes in regulatory aspects, policy, and business services would have informed USAID on the project’s effect on the development of an enabling environment for economic growth. EVALUATION QUESTION 2 To what extent were the selected interventions cost effective? Findings The LIVCD farmer survey reports the average gross margin (GM) per hectare (or per hive for honey production) and the total area planted by LIVCD beneficiaries (or the total number of hives for beekeepers) who were already producing the target crops in 2013–2014. The ET calculated the incremental GM per crop by subtracting the GM per hectare (and per hive) of 2017–2018’s agricultural season (the endline) from the GM per hectare (and per hive) of 2013–2014 (the baseline). The result was multiplied by the total cultivated area (calculated by the LIVCD survey) to obtain the total incremental GM of each agricultural value chain. The benefit-cost ratio (BCR) was obtained by dividing the total incremental GM by the total expenditures (including TA and grants) provided by LIVCD management. Results for the GM calculation are reported in Table 3 along with the BCR. Management costs and costs that could not be attributed to a single value chain are not included in the table. Given that the sampling frame of the LIVCD survey excluded farmers who established new orchards through project grants, the cost of the grants to establish new orchards was subtracted from the costs covered by the project for the calculation of the BCR (and is not included in the table).17 In this way, the BCR so calculated does not take into account the benefits and costs of new orchards. 17 These included three grants to establish avocado orchards, three grants to establish demo orchards for grape farmers linked to exporters, and part of two grants that funded drip irrigation systems for grape farmers and that were awarded to CCIAZ. More specifically, for the two CCIAZ grants the cost was reduced proportionally to the number of farmers that established new orchards with the drip irrigation funded by LIVCD (27 farmers out of 119 for the first grant and 22 out of 49 for the second grant). 20 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 Table 3: Costs and Benefits by Value Chain GM / Ha-hive at the baseline ($/Ha-hive) GM / Ha-hive at the end-line ($/Ha-hive) Cultivated area (Ha) Total expenditures ($)18 BCR Apple 4,920 3,455 747 2,392,925 -0.46 Avocado 10,153 18,822 502 646,132 6.74 Cherry 4,842 5,153 438 1,272,749 0.11 Grape 7,691 9,124 1,221 1,390,964 1.26 Honey 167 175 71,448 2,739,828 1.80 Olive 1,332 2,600 5,077 2,137,693 3.01 Table 3 shows that interventions with avocado farmers had the highest BCR, followed by interventions with olive farmers. As explained under the findings for EQ3, for avocado farmers this was due to the productivity increases caused by training and grafting, whereas for olive farmers the main reason was the reduction of production costs thanks to the introduction of mechanical harvesters. The BCR is higher than one for honey and grapes. The findings for EQ3 show that for honey the increase in GM was caused by an increase in the number of hives (rather than by an increase in the GM per hive). The lowest BCR is for interventions with apple and cherry farmers. It is negative for apple farmers because of the devaluation of the Egyptian pound (which caused a dramatic reduction of the farm gate price in Lebanon) and because improvements in yields were limited (see findings for EQ3). The budget allocation for grants was $10.2 million. At the end of July 2018, LIVCD had spent (or committed) 99.9 percent of this amount. Table 4 reports the awarded grant amount by value chain and the corresponding total grantee contribution. As seen in Table 4, cost-share contribution was proportionally higher in the grape value chain (180 percent of the awarded amount) and the avocado value chain (160 percent of the awarded amount). In the processed foods value chain, cost-share contribution was lower than the awarded amount. However, the ET visited two companies (out of six interviewed) that made significant investments ($6.2 million and $0.7 million) to expand their production capacities and that were inspired by the project TA. Also, as reported under the findings for EQ3, the processed foods companies visited by the ET reported increases in both the volume produced and sales. Table 4: Grant Budget by Value Chain19 Awarded Cost-share Cost-share / awarded Apple 1,289,815 1,157,991 90% Avocado 270,229 431,530 160% Cherry 429,715 561,327 131% Grape 1,015,721 1,823,750 180% Honey 1,775,301 1,047,154 59% 18 The figure is the sum of the awarded grant amount (thus excluding cost-share) and TA costs. For stone fruits, the TA component provided by LIVCD did not distinguish between TA costs for cherry and avocado (i.e., the two crops included in the stone fruit value chain). The TA costs of these two value chains were estimated by proportionally allocating the TA costs of the stone fruit value chain to the total grant value awarded by LIVCD for the cherry and avocado value chains (which was known). 19 The table excludes three grants that covered more than one value chain. USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 21 NOVEMBER 2018 Olive 1,243,953 780,193 63% Processed food 1,907,609 1,571,455 82% Rural Tourism 1,516,709 984,453 65% Rural Basket 529,450 377,173 71% In the rural tourism sector, the guesthouse owners interviewed by the ET reported that their work was not impacted by the training sessions they had attended. However, they also reported that their income rose as a result of the higher number of tourists staying in their facilities. The LIVCD impact assessment study reports figures that clearly show how the number of tourists rose in the target areas. LIVCD may have contributed to this trend since the grants consisted of increasing the products and services offered in the target areas. However, the same study indicates that other interventions were funded by other donors in the same area. Conclusions Cost-effectiveness was analyzed by using the BCR for the agricultural value chains that were covered by the farmer survey and comparing the value of the grantees’ contribution with the amount awarded by the project. Interventions in the avocado and olive oil value chains were more cost effective than those in the other value chains. For the avocado value chain, the yield increase attributed to the project improved the profitability of avocado, so the incremental gross margin is much higher than the costs covered by the project. For the olive oil value chain, the distribution of mechanical harvesters significantly reduced production costs (see EQ3) and since the machinery was shared by more farmers, costs were reduced per farmer (and per quantity of harvested olives). Interventions in the cherry and apple value chains were not cost effective since the incremental gross margin was lower than the project’s direct costs. Farmer beneficiaries of the grape and avocado value chains significantly contributed to the investment cost required to establish new orchards by paying more than the amount awarded by the project. The grantee contribution was higher than the awarded amount in the cherry value chain as well. However, the increase in the incremental gross margin here was lower than the direct costs covered by the project. Cost-share was required only for the grant component. In the processed foods sector, TA inspired some beneficiary companies to take up new investments to enlarge their production capacities. In addition, the fact that all visited companies reported increases in sales suggests that the interventions were cost effective in this value chain also. Tourist numbers rose in the areas targeted by the project, which suggests the interventions helped enhance rural tourism. However, an assessment of the cost-effectiveness of the LIVCD rural tourism interventions is not possible since many other donors were promoting similar interventions. Also, it is not possible to assess the cost-effectiveness of new orchards since the farmers who established new orchards were not included in the farmer survey. 22 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 EVALUATION QUESTION 3 To what extent have LIVCD’s interventions resulted in meeting the objectives and key performance indicators (KPIs) of the initial and extended contract? a. Were the selected interventions and entry points in each value chain strategic in leading to the intended results? b. What internal and external factors contributed to the achievement (or nonachievement) of the project’s results? Findings The LIVCD Indicators Performance Tracking Table (IPTT) shows that that by the end of June 2018 all project targets had been achieved with some exceptions for the gross margin indicators. The farmer survey results show that the change in the GM per hectare reached the 10 percent target increase for avocado, grape, and olive oil. The GM per hectare increased for cherry and honey but failed to reach the 10 percent target. Moreover, the GM per hectare decreased for apple. The target for women’s participation was achieved (23 percent of total beneficiaries were women). However, women’s participation was low in all agricultural value chains (see findings for EQ1 and EQ5). The FGDs revealed that the location and time of the training sessions conducted were suitable for women. They also revealed that agricultural operations are mainly conducted by men. As one female grape producer put it, “If my son were in the country, he would have been part of the project, not me!” The main reason is that agricultural operations are considered excessively tough and tiring for women. As a result, women managing orchards would end up employing laborers, thus reducing profits. Another reason reported in the FGDs is that agricultural work is considered either inappropriate or dangerous for women since it requires spending several hours in the field with male workers. A KI from academia explained that there’s a social stigma in Lebanon attached to women conducting hard labor in agriculture. However, the important role of Syrian women laborers in Lebanon’s agriculture sector shows that agricultural work is in principle feasible by women as well. LIVCD Objective 1: Increased Competitiveness of Lebanese Value Chains LIVCD partnered with different actors depending on the target value chain. Rural Tourism In the rural tourism sector, LIVCD’s main partners were the Ministry of Tourism (MoT) for the development of the National Rural Tourism Strategy and nongovernmental organizations (NGOs) for grant implementation. The development of the Rural Tourism Strategy was reported by NGOs to have created an important momentum that put rural tourism at the center of the tourism development policy. The strategy’s development was clearly supported by the minister. However, this political endorsement was not used to develop (and pass) the needed regulation for the sector. Indeed, interviews with sector operators revealed that the lack of an updated regulation for rural tourist guides and the presence of unprofessional tour organizers are considered limiting factors for the sector’s development. The LIVCD impact assessment study for rural tourism mentions that the high number of nonregistered tour organizers causes an unfair competition for formally registered companies. The LIVCD impact assessment study for rural tourism provides details on 18 different project interventions in rural tourism (including the development of the national strategy). An analysis of the single rural tourism interventions with KIs revealed that 14 interventions were implemented as planned, with 10 of them delivering the expected results. USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 23 NOVEMBER 2018 The study reports figures on the number of tourists and income generated for nine interventions. For all these interventions, there was an increase in income for beneficiaries (guesthouses, restaurants, hotels, hostels, guides) and in the number of tourists. However, except for Hadath El Jebbhe, the study attributed the increase in the number of tourists in the target areas to LIVCD only to a limited extent since other projects were being implemented by municipalities or other donors. The case of Hadath El Jebbeh village is different since no rural tourism activity was taking place there before LIVCD. As a result of the project’s activities, a small rural tourism industry emerged in the village, with 11 new shops opening, as reported by a KI from the village. Processed Foods Interventions in the processed foods value chain provided TA and grants to individual companies and processing cooperatives and were tailored to each beneficiary organization’s needs. All six organizations visited by the ET reported satisfaction with the TA and grants provided. TA consisted of one-to-one interactions. Five of the six visited companies reported an increase in sales or in the volume of raw material processed as well as an increase in the number of employees (the sixth company was an apple juice manufacturer that had just completed the installation of the processing equipment supported by LIVCD). In addition, one intervention strengthened the production capacity of an organization employing people with disabilities. LIVCD also supported freekeh20 production by delivering roasting equipment to women cooperatives. The midterm evaluation and a KII conducted with a grantee during the final evaluation both reported an increase in sales and production capacities for the interviewed cooperatives. Because of the introduction of a more efficient roasting equipment for freekeh production, women producing freekeh are now renting land to cultivate wheat (the raw material for freekeh) and are employing men to do the labor, according to LIVCD staff. Avocado Avocado is a relatively new crop in Lebanon. The midterm evaluation has already noted that technical knowledge was limited in avocado and that avocado cultivation is concentrated mainly in the South. LIVCD developed a technical curriculum that was used by training providers. All four FGDs with avocado farmers unanimously reported satisfaction with the training sessions attended, their usefulness, and the follow-up they received. More specifically, the farmers found most useful the training they received on harvesting time, pruning, and frost-resistant varieties in cold areas. Interventions in the South focused on increasing productivity in avocado orchards, while in Mount Lebanon and Akkar they focused on expanding avocado cultivation into mountain areas where there hadn’t been any orchards. Two of the four FGDs were with farmers that had planted avocado for the first time. They could not report on the profitability of this effort since the avocado tree requires three to four years to produce the first yield and three more years to reach its full production capacity. However, they expected avocado farming to be profitable for them since prices are considered high. The FGDs with farmers that already had avocado orchards revealed that their orchards are now more profitable and that the new grafting techniques they now use on old trees contributed to increasing their production (they learned these techniques in the training they received). The survey conducted at the time of the midterm evaluation revealed that 58 percent of avocado farmers are applying new agricultural practices for the first time. In 20 A cereal product made from green durum wheat that is roasted and rubbed to create its distinguished flavor. 24 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 addition, beneficiary farmers who were already producing avocado reported that avocados can be easily sold since demand is high and that the selling price is profitable for them. All these results are confirmed by the LIVCD farmer survey, which reports that the GM per hectare increased by 85 percent21 (the increase is statistically significant) from the 2013–2014 agricultural season to the 2017–2018 season. The survey also shows that the quantity produced rose by 35 percent.21 Apple Apple is the most important crop in value of production in Lebanon after potato and tomato.22 The LIVCD project facilitated the establishment of seven service centers by targeting four cooperatives, two NGOs, and one private company selling agricultural inputs and services. The service centers were supposed to provide TA to farmers, facilitate market access for products, and organize the purchase of fertilizers and pesticides. At the time of the final evaluation, one of the four cooperatives was already selling apples to a supermarket in Lebanon and exporting them as well. However, this cooperative did not attribute its success in marketing apples to LIVCD assistance. All other service centers provided TA services (e.g., pruning and spraying) but not marketing services. KIIs with service centers revealed a big demand for pruning services but a smaller one for spraying services. The FGs provided mixed evidence on the training sessions’ usefulness, with three out of five FGs saying they were indeed useful, especially for irrigation management and pruning, and two saying they were not, with farmers arguing that the training did not teach them anything new and that there was not enough practical implementation. The midterm evaluation survey showed that 17 percent of LIVCD’s apple farmers are now applying new techniques as a result of the training they received. When interviewed, the trainers reported that one-to-one training was more effective than group training. The KIs also reported that one main problem in the apple value chain was the presence of 25-year-old orchards that produce varieties that are no longer in demand and an excessive use of pesticides of doubtful quality. The limited availability of quality apples was perceived by the interviewed exporters as the greatest challenge to expanding exports beyond the traditional market, which is Egypt. They emphasized how important exporting to better-paying countries was for the sustainability of the apple value chain. This is because the November 2016 devaluation of the Egyptian pound dramatically reduced the selling price of apples in Lebanon. The LIVCD farmer survey showed that the unit price of apples fell by 21 percent from baseline to endline. Overall, the five FGs reported no increase in their profits in the past couple of years, since the price of apples paid by traders fell and the techniques they learned at the LIVCD training sessions raised the costs of pesticides. This finding is confirmed by the LIVCD farmer survey, which reports that the GM per hectare fell by 30 percent23 and the cost of pest management rose by 21 percent, 23 while the change in total production was positive (9.5 percent) but not statistically significant. Cherry Cherry and apple production occur largely in the same areas. Cherry farmers were described by LIVCD staff as apple farmers who also cultivate cherries. LIVCD staff also reported that the cherry value chain is the most problematic because agricultural practices are still outdated and apple farmers consider cherry a marginal crop. The midterm evaluation survey revealed that only 12 percent of the interviewed cherry farmers applied new agricultural techniques that they were unaware of before the training. In three of the four FGs with cherry farmers, participants expressed satisfaction with the training they received, while in the fourth farmers complained that the training did not offer them any new knowledge or instructions on what fertilizers to use. Two of the four FGs reported that they received regular follow-up after the 21 The difference in the means of production cycle is statistically significant. 22 Source: FAOSTAT. 23 The difference in the means of production cycle is statistically significant. USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 25 NOVEMBER 2018 training, while the other two (Aarsal and Baabda) reported no or limited follow-up. In two of the FGs, cherry farmers also expressed concerns over the quality of seedlings available in Lebanon. LIVCD supported private sector TA provision. The FG in Bcharre revealed great satisfaction with the TA they received from a small company supported by LIVCD in the apple and cherry value chains, while the Aarsal and Baabda FGs reported the absence of a reference company to turn to when TA is needed. The FGs also reported that the buyers supported by LIVCD did not provide a valid market outlet either because the quality demanded by one of the companies supported by LIVCD was high or because the price offered by another company supported by the project was lower than the price they could get elsewhere. Overall, cherry was perceived by farmers to be profitable since the local market price is considered high. In this regard, the LIVCD farmer survey shows that GM per hectare has risen by 6 percent23 and that the quantity produced has fallen by 11 percent, but such a reduction is not statistically significant. The survey also shows that the price has risen by 30 percent. Because FGDs reported no benefits from the project on market aspects, and one buyer reported an increase in the price of cherry at the national level, the price increase cannot be attributed to the project. The 2010 Agricultural Census revealed that 65 percent of cherry-cultivated areas are in Baalbeck￾Hermel,24The KIs specified that the greatest cherry-producing area was Aarsal, but that part of Lebanon could not be reached by the project until its final phase for security reasons. One of the cherry traders supported by LIVCD reported to have tried to buy cherries from Aarsal but they could not find enough quantity with the right quality and finally ended up buying cherries from other areas. The lack of TA services and follow-up from the LIVCD staff, as reported by the Aarsal FG, was also due to security reasons. Grape For the grape value chain, the project supported two cooperatives to distribute electrostatic spraying systems (ESS) and partnered with three grape exporters to identify 50 smallholders to establish new vineyards and plant the varieties demanded by exporters on 70 hectares. During the project’s final phase, LIVCD also gave two grants to the Chamber of Commerce, Industry, and Agriculture of Zahle (CCIAZ) to establish new drip irrigation systems. The farmers’ cooperatives’ representative reported to have used the ESS and consequently reduced the quantity and cost of the pesticides used. Records provided by LIVCD’s staff, which were developed using sale receipts from the two cooperatives that received the ESS, show an increase in unit price and quantity sold. The two grants awarded to the CCIAZ enabled 169 farmers to establish new drip irrigation systems on both existing and new orchards. The FGs reported that it was still too early to assess how much the drip irrigation system increased their profitability. However, farmers from one FG complained about the quality of this system, saying they had to continuously follow up with the tube-selling company to replace cracked tubes. During FGDs, grape farmers reported great satisfaction with the TA and follow-up provided by the LIVCD value chain manager for grape. The training was also generally reported to have been beneficial, albeit with some exceptions. For example, the farmers complained that the training on irrigation management was 24 Source: Ministère de l’Agriculture e FAO (2012) Résultats Globaux du Module de Base du Recensement de l’Agriculture 2010. Projet « Observatoire Libanais de Développement Agricole » de la Coopération Italienne. [On line]. Available from: http://www.agriculture.gov.lb/html/RESULTATS_RECENCEMENT_AGRICULTURE_2010/RAPPORT_RESULTATS _GLOBAUX_DU_RECENCEMENT_2010.pdf 26 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 inadequate for their field conditions (due to the high cost of extracting water from artisanal wells). In one FG, the farmers complained that they received very different instructions from LIVCD and CCIAZ experts on the needed quantity of irrigation water. The five FGs revealed that follow-up was provided by LIVCD after training through phone calls, field visits, and a WhatsApp group. However, the way this follow-up was provided was not uniform, with farmers from three FGs reporting having received limited follow-up, and farmers from the other two saying they had received excellent and constant follow-up. The farmer survey shows that the sale value per farmer has risen by 12.4 percent and the cost of pest management has fallen by 11 percent. These positive changes compensated the average reduction of the unit price per famer by 13 percent, so, the increase in GM per hectare was 19 percent.25 Farmers expect that the newly planted vineyards will be profitable, especially those planted with export￾quality varieties. As one farmer put it, “We have a better-quality produce now and we are able to grow new varieties that are in high demand. Therefore, we are able to secure better prices.” However, it is too early to assess the actual profitability of these vineyards, since the vines had not yet reached their full maturity at the time of the final evaluation. One common problem reported was the sale of defective pesticides from input providers, which reduced profits. During the FG, the grape farmers reported selling to the Ferzol market, to middlemen, and directly to Beirut and Saida. Of the three exporters with whom the project partnered, two reported to have provided TA to farmers and to have started buying grapes from a smaller number of farmers at the same price paid by the wholesale market. They also plan to buy more grapes as soon as the new vineyards increase the produced quantity. The third exporter is no longer in the grape business. One of the farmers who was linked to this exporter reported he was now selling his grapes to the wholesale market through middlemen. The grape value chain had the lowest rate of female beneficiaries (5 percent) when the midterm evaluation was performed, but at the time of the final evaluation that rate had risen to 12 percent. This increase was due to the two grants awarded to CCIAZ, which involved a higher percentage of women than did previous interventions in the grape value chain. However, CCIAZ’s role in selecting beneficiaries is not very clear, since the FGDs revealed that the farmers who applied for the drip irrigation system came to know about this possibility through word of mouth from municipalities or other farmers. The LIVCD staff reported to have suggested the initial list of beneficiaries to CCIAZ, whereas CCIAZ reported to have used their internal database to select beneficiaries. The FGDs also revealed that the project with CCIAZ was not well advertised. In this regard, one farmer reported, “I found out about this project by chance at the end of the season. When I went to apply, they told me that I was late and that they were no longer accepting applications. In the end, I was able to contact the head of the CCIAZ. He is the one who helped me. He also told me that there were a lot of funds but that no one was applying. I told him that no one was aware of the project due to the lack of advertising.” Olive With a total cover of 60,300 hectares, olive is the most important crop in Lebanon in terms of land use.26 LIVCD supported farmers cooperatives by organizing training sessions and distributing mechanical harvesters and other equipment (for spraying, pruning, and plowing). Three of the five FGs reported satisfaction with the training while the other two complained that trainings did not teach them anything new. The survey conducted for the midterm evaluation shows that 37 percent of beneficiary olive farmers were applying techniques they were not familiar with before the LIVCD support. All five FGs with olive farmers reported little or no follow-up after the training. 25 All differences in means reported in this text are statistically significant (unless specified differently). 26 FAOSTAT. USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 27 NOVEMBER 2018 FGDs revealed that the mechanical harvesters were largely appreciated by the farmers as they enabled them to reduce production costs. The LIVCD survey reveals that the farmers reduced the costs of labor for harvesting by 40 percent and that the total produced costs were reduced by 21 percent.23 However, FG participants also reported that the distributed mechanical harvesters were not enough to satisfy their needs. Not all FG participants said that they managed to use them, and some complained about the excessive waiting time to get access to these machines. The Ministry of Agriculture (MoA) estimates that there are 170,000 olive farmers in Lebanon,27 while LIVCD reported that the mechanical harvesters it distributed were used by approximately 2,500 farmers—i.e., 1.5 percent of the total number of farmers. LIVCD supported five mills with TA and new equipment and FG participants reported satisfaction with the services these mills provided. The project delivered equipment to the Holy Spirit University of Kaslik (USEK) to detect frauds and plant pathogens and supported the Chamber of Commerce, Industry, and Agriculture of Tripoli (CCIAT) to establish a laboratory for organoleptic and chemical analyses to classify olive oil. CCIAZ’s laboratory was also equipped to conduct full testing services for olive oil. There are no specific findings on the use of laboratory equipment delivered by the project since they had just been installed when this evaluation was taking place. In three of the five FGs with olive farmers, the participants complained about the lack of market outlets. Farmers said they still had unsold oil from the previous season and that they often end up giving it away as a gift to their friends and relatives. However, the LIVCD survey reports that farmers increased the quantity sold by 15 percent and that the combined effect of this increase in the quantity sold and production costs increased the GM per hectare by 95 percent.23 One major problem reported by both FG participants and interviewed mills is the import of low-quality, cheap oil from Syria. The project organized olive oil tasting events to raise awareness of oil quality, but no studies were conducted to assess how much these activities increased people’s olive oil consumption or willingness to pay more for quality oil. Two participants in one FG reported that they produce olive oil for personal use and not for sale. In this regard, the survey conducted for the midterm evaluation revealed that only 22 percent of the LIVCD farmer beneficiaries in the olive oil value chain reported farming as their main source of income. Honey Interventions in the honey value chain occurred during the original contract duration only, since USAID instructed DAI to exclude this value chain from the LIVCD portfolio during the extension period (which started on October 1, 2017). A main finding from the midterm evaluation was that only 2 percent of LIVCD beneficiary beekeepers reported that beekeeping was their main source of income, a fact confirmed by two KIs from academia who reported that beekeeping in Lebanon was an activity conducted mainly by hobbyists. LIVCD targeted cooperatives to distribute beehives and organize training sessions. In total, the project partnered with 31 cooperatives to distribute 5,870 beehives to 1,497 beekeepers (who bought 1,967 beehives as cost-share in addition to those received by LIVCD).28 FGDs revealed that training was useful for beginners but not very useful for experienced honey producers: three FGs reported that training was beneficial because they had limited experience in beekeeping while the other two said it was not useful. Participants of one FGD also reported that trainers lacked the expertise and that the knowledge offered was not relevant to their needs. The survey conducted for the midterm evaluation showed that 21 percent of beneficiary beekeepers (42 out of 198 beekeepers included in the sample) had never produced honey before 2013, when the project started. FGDs also revealed that post-training 27 Cited by IDAL. (2017). Olive Oil Industry in Lebanon. 2017 Factsheet. 28 Source: communication from LIVCD staff. 28 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 follow-up varied according to the cooperatives involved and by geography, with four FGs reporting regular follow-up either by LIVCD experts or by the president of the local cooperatives and one FG reporting no follow-up. The LIVCD survey shows that the GM per hive rose by 5 percent, the average number of beehives per beekeepers by 35 percent, and honey sales by 15 percent.29 The midterm evaluation already noted that during the project’s initial phase, staff had limited expertise in beekeeping at the time of the distribution of low-quality beehives. One FG participant said, “The donation given by USAID is a joke…; all bees that we received from USAID are now dead because the queen bees were three years old.” The midterm evaluation also noted that the LIVCD staff gained experience during the project life and that remedial actions were taken to improve the quality of procured beehives. The project supported one cooperative (out of the 31 cooperatives it partnered with for beehive distribution) to establish new commercial linkages through organized marketing courses. LIVCD also supported commercial buyers to increase their sales; however, the FG participants that were introduced to them refused to conclude sales of honey with the supported buyers because of the low price offered. The honey value chain’s impact assessment report emphasizes that small beekeepers (with no more than 50 hives) directly sell to consumers through their personal network of friends and relatives at an average price of $25-30/kg of honey. Larger beekeepers sell the production of approximately 20 hives directly to consumers and the rest to commercial buyers for $10-13/kg. The project also supported the development of complementary TA services, like a wax extraction center. FGs reported to have used the wax extraction center with satisfaction since it enabled beekeepers to cut production costs and avoid spreading diseases (which easily happens when the old wax is dumped). LIVCD Objective 2: Increased Access to Value Chain Finance The LIVCD IPTT shows that at the time of the final evaluation, 102 project beneficiaries (12 companies and 90 individual farmers) were awarded loans for a total value of $18.8 million. Table 5 reports the number of loans by category of loan recipients and by value chain. An analysis of loan distribution30 reveals that the first 10 largest loan recipients were awarded loans worth $8.7 million, which is 46 percent of the total loan value. Also, the total value of loans awarded to the 12 beneficiary companies was $7.5 million (40 percent of the total). The total value of loans awarded to the 90 beneficiary farmers was $11.2 million (60 percent of the total). Among agricultural value chains, the highest number of loans and the highest loan value were awarded to avocado and grape producers. Table 5: Loans Awarded Beneficiary type Total loan value ($) N. of loans Agr. Producers 11,299,166 90 Avocado 4,011,600 24 Cherry 150,000 1 Grape 4,275,733 31 Honey 274,000 11 Olive oil 335,000 3 Pome fruits 2,252,833 20 Companies /processors 7,232,702 11 Avocado 230,000 2 29 All differences in means reported in this text are statistically significant (unless specified differently). 30 Source of data: LIVCD M&E system. USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 29 NOVEMBER 2018 Honey 600,000 1 Olive oil 60,000 1 Processed food 5,692,702 6 Fruit exporter 650,000 1 Retailer 252,667 1 Total 18,784,535 102 Source: LIVCD M&E system LIVCD developed feasibility studies for loan applications (upon beneficiaries’ requests) until 2016, when USAID instructed DAI to interrupt this activity. LIVCD records show 27 beneficiaries of this activity and 12 loans awarded. Of the 12 awarded loans, one was to a company, one to a cooperative and the rest to farmers. The number of loan beneficiaries (102) is higher than the number of loans awarded upon submission of the feasibility study developed by LIVCD (12) because the list of loan beneficiaries also includes companies that benefited from the TA and grants provided by the project rather than from the assistance provided by LIVCD staff to submit loan requests. The midterm evaluation noted the reasons for the low number of loan requests by farmers. Farmers can easily get short-term loans to pay for harvest operations from traders (to whom they commit their production) and buy inputs from suppliers on credit. Long-term loans have to be sought from banks. FGDs revealed a general aversion of farmers to loans, with some exceptions for grape and avocado producers. During FGDs, farmers also reported that the conditions imposed by banks for loans are often considered unreasonable or impossible to fulfill. These include formal records of sales and mortgaging their own house. The Banque Libanaise pour le Commerce (BLC) was the bank with the highest agricultural loan portfolio at the time of the midterm evaluation. It was highly involved by LIVCD and its loan officers were trained on the target value chains. Also, BLC personnel gave a presentation on loan products to farmers at the end of technical trainings. However, the midterm evaluation reports that fewer than 40 loan applications were received by BLC.31 The current economic crisis was reported by a wide range of KIs as the main reason for credit reduction in the agriculture sector. LIVCD staff and an ex-BLC officer reported that the bank was no longer interested in working in agriculture after its top management changed. LIVCD staff also reported that banks started to require more loans to be in US$ rather than in LBP. As a consequence, farmers are even more reluctant to raise loans. Moreover, budget constraints led to the GoL’s decision to reduce funds available for Kafalat (i.e., a government-funded financial institution providing loan guarantees). One of the consequences was that interest rates of subsidized loans for agricultural investments have risen from 1 percent to 2–2.5 percent. Internal and external project factors An issue raised by three interviewed grantees and USAID was the long time needed for grant preparation. Concerns about the excessive preparation time of grants were also reported in the midterm evaluation. Two grantees reported to have lost one agricultural season because the equipment funded by the grant arrived too late. Table 6 reports the distribution of the number of grants by length of processing time of each grant preparation phase. Table 7 shows the number of grants evaluated per year and exceeding 60 days from the evaluation of the expression of interest (EoI) until submission to USAID. 32 31 This information could not be collected for the final evaluation because BLC did not grant the ET an interview. 32 The two tables exclude grants for the purchase of mechanical harvesters and grants that were withdrawn. 30 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 Table 6: Distribution of Grants by Processing Time Grant duration N of grants by days since receipt of EoI to its evaluation N of grants by days since EoI eval. to submission to USAID N of grants by days since submission to approval N of grants by days since approval to signature N of grants since receipt of the EoI to signature ≤ 30 days 130 17 95 117 5 31-60 days 6 26 38 30 10 ≥ 60 days 3 96 26 12 124 Obs 139 139 159 159 139 Source: LIVCD grant tracker Table 7: Number of Grants per Year Year N of grants evaluated by year N of grants with more than 60 days from EoI eval to submission to USAID Perc 2013 1 0 0% 2014 42 32 76% 2015 49 43 88% 2016 23 14 61% 2017 14 5 36% 2018 10 2 20% Source: LIVCD grant tracker Table 7 shows that the phase that had the highest number of grants exceeding 60 days was the development of the grant by LIVCD staff after the evaluation of the EoI. LIVCD staff reported that the development of all required annexes is a very time-consuming phase, especially when social vetting and environmental review are required. An analysis of the results shown in Table 7 reveals that the grant processing time has decreased since 2016. Important external factors have affected the development of the targeted value chains. The previous sections have already mentioned the contraction of the government-funded subsidy system, the economic crisis, unregulated imports of Syrian olive oil, the devaluation of the Egyptian pound for apple sales, and the presence of old apple orchards that produce varieties that are no longer in demand. A constant remark raised by trainers was the old age of farmers and their low educational level, which limit the capacity of farmers to learn and use new techniques. The LIVCD monitoring system did not include the age of beneficiaries in a consistent way. However, out of the 216 farmers that participated in FGDs only 68 were younger than 50 years. Also, the Syrian crisis was reported by all exporters to have dramatically affected the competitiveness of Lebanese products since the Syrian-Jordanian border crossing was closed, which made road-freight transportation to the Gulf impossible. Consequently, exports could occur only by sea or by air freight. Other reported elements that affected costs and competitiveness were the intermittent availability of electricity and the high costs of fuel, leading to the inclusion of solar power systems in grants for a cold storage facility and pickle producers. The MoA’s limited capacity was also reported to be a constraining factor. The grape, cherry, and apple exporters who were interviewed reported that a constraining factor to their activity is the limited capacity of the MoA’s laboratory (in Kfarshima) to process their requests in a timely manner to issue the required USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 31 NOVEMBER 2018 certificates for export, including the presence of pesticide residues. CCIAT is a one-stop shop that provides all required certificates to export. However, none of the visited fruit exporters reported that they were using CCIAT’s services to export (this is probably because the laboratory to check for pesticide residues was set up only recently). Interviewed honey experts also reported that the MoA has a very limited capacity to provide TA to beekeepers. Conclusions LIVCD’s achievements were found to vary by value chain and geographic location, with some groups receiving more support and follow-up than others. This unevenness is unavoidable to some extent, given the demand-driven nature of LIVCD’s interventions. All categories of economic operators reported significant factors that prevent farming from being profitable. Various interventions were ingeniously devised to enhance the farmers’ individual productivity in value chains that are plagued with a plethora of contextual challenges, each of which alone could send the sector into deep crisis. These challenges include the closure of land borders, the doubtful quality of pesticides sold by input providers, the unregulated import of olive oil, the old age of farmers, and the MoA’s weak capacity to provide TA and the laboratory analyses needed for exports. With these conditions, it is very difficult to “beat the system” with training, TA, and the introduction of modern equipment. However, there is some evidence of improved competitiveness and higher incomes in the processed foods sector and the olive oil value chain due to the introduction of more efficient technologies and the development of market linkages for some processed food companies. There was also an increase of incomes in the avocado and grape value chains. The LIVCD survey results for the GM and FGDs suggest that apple value chain competitiveness did not improve and that project benefits were lower for cherry value chain beneficiaries. In the rural tourism sector interventions increased the number of tourists visiting the target areas. However, except for one village, attributing the rise in the number of tourists to the LIVCD project is questionable since many other non-LIVCD interventions were implemented in the rural tourism sector. The total value of loans raised by LIVCD beneficiaries was considerable ($18.8 million). However, loans were awarded to a very small number of beneficiaries (102). In addition, the fact that only one beneficiary company used the LIVCD feasibility study to raise loans suggests that the loans awarded to companies (40 percent of the total value of loans) were more the result of investments leveraged by grants and TA than the result of activities specifically aimed at promoting access to finance. The combined use of TA and grants allowed the project to tailor its interventions to the beneficiaries’ investment capacities and risk aversion profile. The time needed for grant preparation was a reason for concern. However, the reduction in the number of grants exceeding 60 days since 2016 suggests that a learning process speeded up grant preparation. LIVCD partnered with cooperatives, private companies, and the MoT. Targeting cooperatives allowed the project to reach a high number of farmers at a low cost. It was also strategic to establish service centers that deliver pruning services and manage mechanical harvesters, but not to improve access to output markets. The targeted private companies that provide agricultural production and post-harvest services were instrumental in developing the chosen value chains. The involvement of the MoT was strategic to develop the National Strategy on Rural Tourism. However, not using the political consensus so created to develop the needed regulation was a missed opportunity to develop the sector. The beneficiary organizations in the processed foods sector were also strategic to create employment opportunities for women (and some disabled people) and provide a market outlet for minor quantities of low-quality grapes and apples. The timing and location of the training sessions were reported to be 32 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 convenient for women. The project reached the target for female participation, but except for the rural tourism and processed foods sectors, female participation was low in all agricultural value chains because agriculture is mainly male dominated and production is perceived as too tough for women. The CCIAZ was awarded five grants (three for grape, one for honey, and one for processed foods) and subcontracts. However, for the grape value chain the added value of involving CCIAZ remains unclear because farmers complained about the lack of proper advertisement of LIVCD’s opportunities and because CCIAZ has no plan to further employ agronomists to follow up on the project’s activities (see EQ4). EVALUATION QUESTION 4 To what extent can LIVCD’s interventions be considered sustainable? Findings Project design had several aspects related to ensuring sustainability, like the requirement that grantees contribute to the grant cost. The LIVCD staff also reported that working with the private sector is conducive to sustainability. In the apple value chain, the private sector became involved in the project’s last phase when a new, large exporting factory was established in Bcharre. The company has a capacity to trade 14,000 tons of apple per agricultural season and plans to increase its exports to Gulf countries and shift away from Egypt, where prices are considered too low. However, both the company and LIVCD staff reported that the presence of old orchards and the limited availability of high-grade apples represent a strong constraint to operating the factory at full capacity. The company expects to use 30 percent of its capacity during the 2018–2019 season. Other activities were implemented in the same area, including a grant to improve the production capacity of a family-managed apple juice factory to provide farmers with a market outlet for their low-quality apples. Also, in the District of Bcharre, LIVCD supported a farmers’ cooperative to act as a service center and a private company that sells agricultural inputs and services to provide TA. Access to TA services was a crucial element in the project’s strategy. One agronomist is currently employed by the municipality of Baskinta and six agronomists by the council of the municipalities of Bcharre. Specialized teams offering pruning services were established following the training provided by LIVCD and the delivery of equipment. To improve the lobby capacities of apple operators and coordinate with actors in the apple value chain, LIVCD facilitated round tables with all relevant sector stakeholders. These round-table meetings were reported by KIs (cooperative leaders and company managers) to be a useful experiment to promote more organization in the sector and lobby the government. An important element for sustainability is market access. FGDs revealed that avocados can be easily sold by farmers, and the LIVCD staff reported a growing demand for avocado. The project supported the production of 500 hectares of avocado and the creation of new avocado orchards for a total extension of 200 hectares. The new orchards had just started producing when this evaluation took place and are expected to increase their yields during the next three years. TA was provided by contracted experts and grantees. During FGDs, the farmers reported that the specialized experts contracted by LIVCD were still available to provide TA. A WhatsApp group was also created among farmers and experts to provide answers to technical questions. However, one concern raised during the FGD organized in Akkar is that the expert employed by the grantee in the area is no longer living in the country and no one is following up with the farmers in the region. However, the grantee has appointed another expert. In Akkar an EU￾funded project has provided TA to 50 LIVCD beneficiary farmers and is using two experts previously employed by LIVCD. USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 33 NOVEMBER 2018 The avocado value chain was described by the LIVCD staff as the only value chain where all the most important actors sit together for discussions regularly. According to LIVCD, these discussions contribute to more coordination and improve the lobbying capacity of the avocado value chain actors. For grape farmers, the main outlet was the Ferzol wholesale market and middlemen. But the LIVCD staff and the FGDs with grape farmers revealed that the Ferzol market was not transparent and that middlemen were suspected of cheating farmers when selling there. As a result, the project linked 48 farmers to three well-established grape exporters by funding the creation of modern vineyards planted with the varieties demanded by the exporters. One of the three exporters was no longer in the grape business at the time of the final evaluation, while the other two reported that they still planned to buy grapes from 30 farmers when the new vineyards increase their production (which is expected to occur in 2019). The two exporters have agronomists who also provide TA to the grape farmers they were linked with by LIVCD. The two exporters that are still in business reported their commitment to work with the LIVCD farmers. However, they also said that they would not engage in a similar project again. They both explained that following up on smallholders was too complicated and required significant supervision costs for their agronomists. LIVCD provided TA to grape farmers either directly or through the personnel employed by the grants awarded to CCIAZ. However, the grape expert employed by CCIAZ through the LIVCD funds moved to another job three months before the end of the grant (given the short duration of the remaining grant period he was not replaced). The CCIAZ has another project on table grapes funded by the EU, which employs three agricultural engineers until October 2019 (when the EU project is expected to end). To some extent the grape beneficiaries of the EU project and of LIVCD are the same. So, part of the LIVCD grape beneficiaries can ask for TA to the CCIAZ experts funded through the EU project. Complementary TA services were developed also in the honey value chain. They include wax recycling, wax uncapping, honey extraction, candy production, and queen bee rearing. FGDs revealed that a crucial element was the distance of beekeeping activities from TA providers. Indeed, the FGs located in Jbeil, Baabda, Jezzine, and Minnieh-Dinnieh reported to have used TA services, while the beekeepers of the FG in Baalbek-Hermel reported to have never used any TA service. One of the beneficiary beekeepers’ cooperatives was reported to be no longer using the equipment received by the project, which were a sterilizer for wax recycling and labelling and sealing equipment. The sterilizer is not used because its operation is too expensive and because beekeepers prefer to use the other wax recycling center supported by the project (it reaches a higher temperature, which is needed for proper sterilization). The sealing and labelling equipment was reported to be of low quality. LIVCD also funded the establishment of Lebanon’s first artificial insemination lab for queen bees through a grant awarded to a private company. It is also the first artificial insemination center in the Middle East. In Lebanon, the native bee variety is Apis Mellifera Syriaca, but other varieties were imported over the last few years, such as A. Mellifera Ligustica. The lab was explicitly funded to reproduce the latter. Serious concerns were raised by two academics and one honey expert about the appropriateness of establishing such a center in a private company, which may not have the proper scientific skills to handle it. Moreover, KIs reported that in other countries, insemination centers for queen bees are managed by public research institutions. Also, the development of such an initiative was not preceded by a study analyzing whether the Ligustica variety was the proper variety for Lebanon. In rural tourism, local project initiatives are expected to be maintained by grantees. The increase in the number of tourists visiting the target rural areas (reported in the Impact Assessment Study for Rural Tourism) suggests that the income rise generated by tourists motivates a continuation of the services delivered. The project also spearheaded the development of the National Rural Tourism Strategy. However, with the change of the minister of tourism, the strategy was reported by several KIs to be no 34 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 longer on the policy agenda. Moreover, the steering committee that was established to implement the agenda did not meet after the change of the minister. An unplanned project effect was the establishment of four start-up companies to provide TA services to LIVCD beneficiaries: a single-person company that still provides technical advice to avocado farmers, a wax recycling center that also offers other needed services to beekeepers (like honey extraction), and two engineering companiesthat developed an optical sorter for pickles and roasting equipment for freekeh production. Conclusions The project design included aspects related to ensuring sustainability, such as implementing demand-driven interventions based on grant and TA requests, working with the private sector, and requesting contributions to grantees. Implementing complementary interventions in a limited geographic area was also conducive to sustainability. Such is the case of Bcharre, which currently presents all the conditions needed to revitalize the apple subsector, including the presence of an active farmers’ cooperative, agronomists permanently employed by municipalities to advise farmers, an agro-input company that also provides well-reputed TA services to farmers, a juice factory buying low-grade apples, and a large apple factory that provides post-harvest services and has plans to increase its exports. However, without clear improvements in quality, the sustainability of interventions in the apple value chain is at risk. The presence of old orchards and the current production of varieties that are no longer in demand represent serious constraints for the development of the apple subsector. Continued access to TA is crucial for the project’s sustainability. The future provision of TA represents a constraining factor for part of the grape beneficiaries. This is because one of the three grape exporters is no longer in the grape business in Lebanon. Also, the CCIAZ grape expert left three months before the end of the grant and the EU grape project implemented through CCIAZ targets only part of the LIVCD grape beneficiaries. Moreover, while the two grape exporters reported commitment to keep working with the farmers they were linked to, the scheme developed by LIVCD to link farmers to exporters does not seem to be replicable. TA is expected to be provided to avocado farmers in the South through the services offered by experts recruited during the project life. However, in the North the ET found conflicting evidence of available TA opportunities for avocado farmers in Akkar. Currently avocado is a profitable crop. The new orchards will reach their full maturity in two years and the expected increase in their production may significantly reduce prices if new market outlets are not opened. Interventions in the honey value chain are partially sustainable. Services offered by private companies or cooperatives supported by the project are used by beekeepers to produce honey. These services include uncapping wax, wax recycling, and honey extraction. Nevertheless, the demand-driven nature of the supported services implies that some areas were properly catered for, whereas others (Baalbek-Hermel) had a clear lack of production and TA services. Also, part of the equipment procured by the project is no longer used. Finally, the sustainability of the first artificial insemination laboratory for queen bees in the Middle East is strongly contested as it is managed by a private company with limited scientific knowledge and its funding did not follow a comprehensive study on the appropriateness of the chosen variety for the country. The increase in the number of tourists in the target areas looks promising. Interventions at the local level have good prospects of sustainability, which lies in the capacities of beneficiary NGOs to keep providing USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 35 NOVEMBER 2018 services. Nevertheless, at the national level, the efforts to develop a rural tourism strategy did not prove to be sustainable for political reasons. Finally, the three start-up companies that were established to provide services funded by LIVCD are expected to keep providing TA to avocado farmers, beekeepers, and the companies and cooperatives that installed the optical sorter and the freekeh roaster promoted by the project. EVALUATION QUESTION 5 Were the positive results spread across all beneficiaries or mainly to few ones only? And to what extent did the interventions demonstrate additionality? Findings The LIVCD farmer survey results show that olive, avocado, and grape farmers experienced the highest increase in GM per hectare from 2013–2014 to 2017–2018, at 95 percent, 85 percent, and 19 percent, respectively. The increase in the GM per hectare for cherry was 6 percent in the same period and was due mainly to an increase in the selling price that cannot be attributed to the project (see EQ3 findings). For honey, the increase in the GM per hive was 5 percent. However, beekeepers also increased the number of hives by 35 percent and total production by 15 percent. Apple GM per hectare declined by 30 percent. These figures show that the two categories of farmer beneficiaries that experienced no or limited benefits were apple farmers and cherry farmers. The sampling document used to design the LIVCD farmer survey reports a total number of 10,163 farmer beneficiaries, of which 867 were apple farmers (8.5 percent) and 855 cherry farmers (8.4 percent). The survey did not calculate the gross margins of new orchards33 and covered only farmers, excluding the beneficiaries of the rural tourism and processed foods sectors. The ET interviewed six companies. One had just installed the equipment funded by LIVCD and had not yet started to use it. Five reported an increase in the number of employees, in the quantity of raw material bought from farmers, and in sales. Four also reported an increase in the number of farmers they sourced raw material from. For the rural tourism value chain, the project delivered 18 different interventions (as categorized by the impact assessment study). Fourteen of them were implemented as planned while four did not deliver results, meaning that no benefits accrued to final beneficiaries. The impact assessment study provides clear figures on incomes for the 14 interventions that were completed and on the number of tourists for 9 interventions, thus showing an increase in incomes generated for all of them. As mentioned in the EQ3 findings, the project achieved the overall target for female participation. A contribution to this achievement was the higher number of female beneficiaries in the processed foods and rural tourism sectors (the only two value chains where the number of female beneficiaries was higher than that of males), with a rate of 56 percent and 51 percent, respectively. The percentage of female beneficiaries is much lower in the other value chains: 7 percent in cherry, 10 percent in pome fruits, 11 percent in honey, 12 percent in grape and avocado, 20 percent in olive oil, and 30 percent in rural basket. The midterm evaluation already noted that LIVCD had conducted a gender analysis as a general assessment. However, it also stated that a gender analysis was not systematically included in the initial value chain assessment reports, which were supposed to provide a comprehensive analysis of the value 33 This is because there was not a baseline to compare the GM of the endline and also because new orchards had not yet reached maturity when the survey was conducted. 36 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 chains and propose interventions. Additionally, PMSPL II developed a resource guide for gender integration in value chain development for the USAID Mission in Lebanon, but the interviewed LIVCD staff could not remember what the resource guide was, showing that this guide was of little or no use. Regarding access to finance, LIVCD’s records show 90 out of 10,163 beneficiary farmers and 12 out of 1,301 beneficiary companies that raised a loan.34 USAID requires that grants and other interventions, especially those destined for private sector entities, demonstrate additionality. The project introduced avocado farming in the mountainous areas of Mt. Lebanon governorate and in Akkar. During FGDs, avocado farmers also reported to have learned new agricultural techniques that they did not know before. The interviewed cooperative heads also reported cases of farmers that replicated the orchards promoted by the project with their own funds. In addition, an EU-funded program expanded avocado farming in the North by replicating the LIVCD model and using the same LIVCD staff. Grantees and TA beneficiaries who adopted new technologies attributed this move to LIVCD and stated that similar investments would have not been made without LIVCD’s intervention. The new equipment adopted included mechanical harvesters for olives, roasters for freekeh, and tank fermenters and optical sorting machines for pickle production. For olive production, all interviewed cooperative leaders reported that the use of mechanical harvesters was very limited or nonexistent in Lebanon before LIVCD. They also reported cases of farmers who bought mechanical harvesters with their own money after having tried those distributed by the project. However, the mechanical harvesters they bought without LIVCD’s support were much cheaper and lower in quality (made in China) than the mechanical harvesters that were provided by the project. The freekeh roasters introduced by the project were replicated by UNDP. Caritas, one of LIVCD’s grantees, reported that two additional roasters (of the same type as the ones introduced by LIVCD) and one wheat harvester were offered to them by UNDP to boost their production capacity and that of their affiliated farmers and cooperatives. UNDP also delivered an optical sorter for cucumbers (used for pickles) developed with LIVCD’s support. The project also promoted completely new production techniques for cherry and apple by funding demo plots for high-density production systems based on trellis. These systems present clear advantages over traditional production systems because they start producing earlier than usual and enable farmers to harvest a greater quantity of produce per unit of land. The project funded 12 demo plots of intensive trellis system usage for apple and 12 demo plots for cherries. The ET found evidence of replication among two farmers. One reason reported by KIs for the low replicability of demo plots was the high cost of the investments for poor farmers. Another reason was that farmers still needed to see the quantity produced by demo plots to be convinced to replicate the proposed system. During FGDs and interviews, grape farmers reported that the support received through grants was essential to establish new vineyards and install drip irrigation systems on existing ones, since the investment costs are too high. As explained by a grape farmer during an FGD, “I had already thought about starting a vineyard. USAID told me that they could help out and support my vineyards. I wouldn’t have developed a vineyard if it weren’t for USAID’s support.” The technical knowledge acquired through training was also reported by grape farmers to be essential. As a farmer put it, “Everything we know about vineyards we learned through them!” 34 The LIVCD document describing sampling procedures reports that the LIVCD beneficiary database also includes nonfarmers. USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 37 NOVEMBER 2018 For the honey value chain, the wax recycling center was reported by the company managing it as a service that would not have seen the light without LIVCD’s funding. During FGDs, beekeepers had different perceptions of the additionality of hives distributed by the project. More specifically, beekeepers in two FGs reported that hives were affordable and that they have made similar investments even without USAID’s assistance. They also added that the knowledge and production techniques they acquired were more valuable to them than the hives they received. Conversely, beekeepers in another FG reported that they would not have bought beehives without LIVCD’s support. Two companies offering post-harvest services reported that LIVCD’s grants and TA were essential to mobilize investments. This was the case of a cherry-buying company that was funded by LIVCD to install cooling and pre-cooling equipment and an olive mill that received a grant and TA to upgrade and redesign its facility. In other cases, beneficiary companies reported that the main advantage of the LIVCD funding was the acceleration of an already planned investment rather than funding an investment that would not have occurred. This was the case of a company offering post-harvest services for apple and a social enterprise providing post-harvest and marketing services for olive oil and honey. Both companies reported that they would have made a similar investment even without USAID’s support, but the investment would have been made over a longer period of time. Conclusions The ET found that the benefits were spread across all beneficiaries, albeit with some exceptions. The four categories of farmers that experienced an increase in GM or sales (avocado, grape, honey, and olive farmers) make up 83 percent of the total number of LIVCD farmer beneficiaries. Apple and cherry farmers, who make up 17 percent of the beneficiary database, did not experience benefits attributable to the project. For the six agricultural value chains, more benefits accrued to men than to women because the percentage of male beneficiary farmers is much higher than that of female farmers. This is mainly explained by the fact that agriculture in Lebanon is a male-dominated environment. A greater attention from LIVCD to gender integration aspects would have probably facilitated the development of interventions aimed at improving women’s participation. The targeted companies’ interventions in the processed foods sector had significant backward linkages that caused an increase in the number of supplying farmers and the quantity of raw material bought from them. The targeted companies also increased the number of employees, thus creating benefits for an additional number of people. The combined use of TA and grants allowed the LIVCD team to tailor interventions to the risk-bearing capacity of different categories of beneficiaries, with grants playing a more important role for more risk￾adverse beneficiaries. The additionality of grants and TA was evident for the interventions that introduced technologies hitherto unknown to beneficiaries. Grants for avocado and grape farmers can be considered additional. On the contrary, the interventions’ additionality aspects were less important for the grants that were awarded to well-established companies and those that did not introduce innovations. Also, hive distribution was not completely additional, because hives are considered affordable and 79 percent of beekeepers were already producing honey before the project started (see EQ3 findings). 38 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 VI. RECOMMENDATIONS 1. Continue using grants to introduce innovations to risk-adverse beneficiaries and provide TA to well-established companies. LIVCD used grants and TA in a combined way to tailor interventions to beneficiary needs. In future projects, USAID should keep using grants with risk-adverse beneficiaries and beneficiaries with limited means and providing TA through process or product innovations to well-established companies that have more capacity to bear risks. 2. Prioritize interventions aimed at developing the apple value chain in districts where considerable investments have been made, to make them an example to be showcased and replicated. The apple sector still presents key challenges that LIVCD could not address, such as the low prices paid to farmers by traders exporting to the traditional export market (Egypt), the lack of a price incentive system to change agricultural practices, the farmers’ reluctance to change the varieties they cultivate and the production techniques they use, and the abundance of old orchards that produce varieties no longer in demand. LIVCD successfully implemented complementary activities in one district in the North. Municipalities also invested in the sector there by paying the salaries of six permanent agronomists. That district now possesses the proper conditions to revitalize its apple value chain. Further interventions represent an opportunity to test the effectiveness of complementary interventions in a single geographic area where the general context is characterized by adverse boundary conditions. In addition, given the substantial investments made in that district and the fact that the availability of quality apples is still very limited in the area, USAID should put in place actions through already funded projects (i.e., LINQ) to ensure that this successful model does not fail. 3. Consider providing support to relevant national authorities to develop regulations with the purpose of promoting an enabling environment for private sector development projects. LIVCD promoted the development of the National Rural Tourism Strategy, whose implementation is no longer on the policy agenda since the minister of tourism changed. (The strategy had been endorsed by the previous minister.) However, LIVCD did not take advantage of the political consensus so created to support developing necessary policies since the development of regulations and policies was not explicitly included in the project design. Should USAID decide to continue providing support to the development of national strategies it should also take into account the institutionalization of the supported strategies￾In future projects dealing with private sector development, USAID should also support in developing sector regulations to promote an enabling environment for the private sector, so that once a project is over the regulation will still govern the private sector even in the absence of an overarching strategy. 4. Continue targeting private companies to improve access to international markets and cooperatives to establish service centers for input and TA provision. With a few exceptions, cooperatives did not prove to be effective in providing marketing services, unlike the distribution of equipment and inputs that they were able to manage properly. Future projects should keep working with private companies to increase their market access. However, cooperatives should not USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 39 NOVEMBER 2018 be excluded from intervention strategies because they can still be instrumental to distribute equipment and inputs and provide TA. 5. In future private sector development (PSD) activities in agriculture, consider capturing the income generated, the employment created, and changes in the enabling environment. LIVCD aimed to increase the competitiveness of companies by introducing innovations. Economic growth and greater competitiveness are not ends in themselves but rather a means to create jobs and improve people’s well-being. However, LIVCD did not explicitly measure job creation or job loss. It measured only “the number of jobs impacted,” which was not an informative indicator until the activity’s final phase when it had to be aligned with the private sector development project. Indicators on job creation should capture agricultural and nonagricultural activities. Because a great part of agricultural work is conducted by family labor, which is not easily captured in surveys and is subject to seasonality, an indicator for measuring the additional income is necessary. LIVCD also included activities aimed at developing policy advocacy and improvement in the regulatory environment of honey trade. These activities were important for the overall intervention strategy but were not captured by the project’s indicators, thus limiting the appreciation of the project’s contribution to the related area of the CDCS. 6. Seek assistance from external and independent experts on the relevance and sustainability of risky and contested grant proposals when dealing with interventions with high scientific content. The laboratory for artificial insemination of queen bees was highly contested by the interviewed academics, who argued that such a center should be managed by public research institutions and not by a private company with limited scientific knowledge. They also argued that the establishment of the artificial insemination center did not follow a comprehensive study assessing whether the chosen bee variety (A. Mellifera Ligustica) is the right one for Lebanon. In the future, USAID should seek advice from independent experts to assess the relevance and potential risks of contested interventions with high scientific content. 7. Consider further interventions in the cherry value chain. The cherry value chain is still problematic. One main reason is that security conditions did not allow the project to properly cover the most important area for cherry production in Lebanon and which was reached by the project only during its final phase. The LIVCD farmer survey shows that cherry production is profitable; therefore, further interventions should be considered, but only if security conditions allow the project team and subcontractors to properly follow up on farmers there and if road conditions are good enough to transport the produce without it being damaged. A situation assessment is needed prior to any intervention in that area. 8. Consider developing interventions to promote avocado exports. Given that avocado is a relatively new crop in Lebanon and that technical knowledge was limited in this field, LIVCD focused its interventions on increasing production capacities rather than improving access to markets. An increase in avocado production is expected during the next few years when new orchards reach maturity. If domestic demand and export do not increase, avocado prices are likely to fall. Interventions aimed at improving the capacities of avocado exporters should be coordinated with other 40 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 international development organizations that are currently planning activities in the same area (e.g., the Center for the Promotion for Imports from Developing Countries, also known as CBI). 9. Ensure that agriculture project contractors apply a gender integration strategy in all project aspects. The project reached the target for female participation only by including the rural tourism and processed foods sectors. Female participation was low in the six agricultural value chains because the agriculture sector in Lebanon is a male-dominated environment. Therefore, greater attention to increasing female participation is still needed. This goal can be achieved by developing strategies to reach out to and engage women more effectively, improving the use of available resources for gender integration, and systematically including a gender analysis in all project design aspects in addition to the initial assessments (like the value chain assessment studies). 10. Consider ways to support the Ministry of Agriculture in key policy areas. LIVCD’s performance and sustainability were affected by a plethora of external factors that were clearly beyond its scope and capacity. These factors included illegal import and sales of low-quality pesticides, imports of cheap olive oil from Syria, a reported weak capacity of the MoA staff to provide extension services to farmers, a weak capacity of the MoA laboratory to respond to exporters’ demand in a timely way to issue the required certificates, and a general market structure that depresses farm gate prices. In these conditions it is very difficult to develop value chains and raise farmer incomes. USAID should consider ways to support the MoA in key policy areas that are more likely to have effects on the performance and sustainability of agriculture interventions supported by USAID. Without addressing the systemic and macroeconomic factors that prevent farming from being profitable, interventions aimed at raising farmers’ incomes will have little impact, especially for more mature value chains. For the exotic and relatively new crops for the country (e.g., avocado) that do not suffer from overproduction, there are more opportunities for learning and for technology transfer. Such is not the case in more mature value chains where there is less room for yield improvements. USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 41 NOVEMBER 2018 VII. ANNEXES ANNEX A: EVALUATION STATEMENT OF WORK LIVCD final evaluation SOW.docx 42 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 ANNEX B: DATA COLLECTION PROTOCOLS Key Informant Interviews KII protocol.docx USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 43 NOVEMBER 2018 Focus Group Discussions FGD guides.docx DISCLAIMER This document was produced for review by the United States Agency for International Development. It was prepared by Social Impact, Inc. contracted under AID-268-C-15-00001. LEBANON INDUSTRY VALUE CHAIN DEVELOPMENT (LIVCD) PROJECT FINAL PERFORMANCE EVALUATION Scope of Work SOCIAL IMPACT FOR USAID I | LIVCD FINAL PERFORMANCE EVALUATION SOW USAID.GOV USAID.GOV LIVCD FINAL PERFORMANCE EVALUATION SOW II LEBANON INDUSTRY VALUE CHAIN DEVELOPMENT (LIVCD) PROJECT FINAL PERFORMANCE EVALUATION Scope of Work April 2018 This document is available in printed or online versions. Documents are also made available through the Development Experience Clearing House (dec.usaid.gov). Additional information can be obtained from: Social Impact, Inc. Corporate Office 2300 Clarendon Boulevard Suite 10300 Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com Social Impact, Inc. Arz Street Librex Bldg. Bloc B – 3rd Floor Zalka, Metn, Lebanon Tel: +961-1-879260 III | LIVCD FINAL PERFORMANCE EVALUATION SOW USAID.GOV CONTENTS INTRODUCTION 1 AUDIENCE AND INTENDED USES 1 PROJECT DESIGN 2 SCOPE OF THE PROJECT 2 INSTITUTIONAL CONTEXT 3 LIVCD PARTNERS 4 DEVELOPMENT HYPOTHESIS 5 PROJECT INTENDED RESULTS 5 CRITICAL ASSUMPTIONS 7 EXISTING INFORMATION 7 EVALUATION OBJECTIVES AND QUESTIONS 8 OBJECTIVES 8 EVALUATION QUESTIONS 8 LESSONS LEARNED AND ADAPTIVE MANAGEMENT 9 RECOMMENDATIONS 10 EVALUATION DESIGN AND METHOD 10 ILLUSTRATIVE DATA COLLECTION METHODS AND TOOLS 10 DATA ANALYSIS METHOD 11 FINAL PERFORMANCE EVALUATION DELIVERABLES 15 REPORTING GUIDELINES 16 TEAM COMPOSITION 16 PROVISIONAL EVALUATION MANAGEMENT 18 ANNEX 1: LIVCD LOGICAL FRAMEWORK 21 ANNEX 2: LIVCD PERFORMANCE INDICATORS TARGETS 24 USAID.GOV LIVCD FINAL PERFORMANCE EVALUATION SOW IV ACRONYMS ADS Automated Directives System AOB Any Other Business CDCS Country Development Cooperation Strategy COP Chief of Party COR Contract Officer Representative CLA Collaboration, Learning and Adapting DAI Development Alternative, Inc. DEC Development Experience Clearing house DO Development Objective FGD Focus Group Discussion FTF Feed the Future FY Fiscal Year (USAID Oct 1 to Sept 30) IM Implementing Mechanism IP Implementing Partner IR Intermediate Result KDS Key Development Services KII Key Informant Interview LIVCD Lebanon Industrial Value Chain Development LOE Level of Effort M&E Monitoring and Evaluation MFI Micro-Finance Institutions MOU Memorandum of Understanding MSME Micro, Small and Medium Enterprises NGO Non-Governmental Organization PIR Project Intermediate Result PIRS Performance Indicator Reference Sheet PMP Performance Management Plan PMSPL Performance Management and Support Program for Lebanon RF Results Framework SI Social Impact SOW Scope of Work STTA Short-term technical assistance USAID United States Agency for International Development USG United States Government 1 | LIVCD FINAL PERFORMANCE EVALUATION SOW USAID.GOV INTRODUCTION The Lebanon Industry Value Chain Development (LIVCD) is a $42.18 million program that began in 2012 and will run through January 2019. The objective of the project is to strengthen the agriculture and agro-processing sectors through technical support to targeted value chains. It does so by enhancing access to finance for rural small businesses and by stimulating innovation and private sector investment. Supported value chains include olives, honey, grapes, avocados, cherries and apples. In addition, LIVCD supports food processing, a wide range of artisanal products and rural basket products—including thyme and pine nuts—as well as rural tourism activities to enhance economic benefits for rural communities. LIVCD uses a market-driven approach and coordinates with private sector and farmer cooperative partners to tailor business development services, allowing them to meet the demands of the market while creating sustainable linkages among input suppliers, producers, processors, traders, and consumers. Public-private partnerships and co-investments initiated by LIVCD help to increase incomes of rural beneficiaries. A mid-term evaluation was conducted for LIVCD in April 2016, it evaluated the diverse interventions under the selected value chains and provided recommendations for project improvement during the remaining timeframe and proposed different scenarios for an exit strategy. The objective of this final LIVCD evaluation is to document critical successes as well as shortcomings of the project in order to provide the Mission, the USG, and the wider donor-community with usable lessons learned for adaptation under USAID’s Collaborating, Learning, and Adapting (CLA) framework. It should provide the audience with input on the approach and activities implemented under LIVCD in terms of achieving outputs as well as outcomes and the sustainability of its interventions. It will also provide the Mission with a background, conclusions and recommendations package to be able to better plan for future activities and follow-on activities. AUDIENCE AND INTENDED USES This final performance evaluation is intended for USAID/Lebanon, specifically the Economic Growth Office, the Program Office, Development Alternatives Inc (DAI) and others at the discretion of the Mission. In the spirit of the USAID Evaluation Policy, the evaluation will provide USAID/Lebanon with concise, actionable recommendations based on evidence to be used by USAID for future program design or program improvement. The results from the evaluation will be used by USAID/Lebanon during its annual Portfolio Review. The report from this evaluation will become publicly available on the Development Experience Clearinghouse (DEC). PROJECT IDENTIFICATION The United States Agency for International Development (USAID) signed the Lebanon Industry Value Chain Development (LIVCD) contract with DAI in September 2012 (No. AID-268-C-12-00001) for an initial value of $41,682,272 until September 30, 2017. A contract modification was executed on June 23rd, 2017, extending LIVCD by 16 months through January 2019 with a budget increase of $4.5 million to $46,182,272. USAID.GOV LIVCD FINAL PERFORMANCE EVALUATION SOW 2 PROJECT DESIGN LIVCD objective is: “Increased competitiveness of value chains that increases incomes of the rural population including MSMEs.” LIVCD was designed to improve the lives of Lebanese citizens by creating stronger and more sustainable enterprises and organizations and by working with Micro and Small and Medium Enterprises (MSMEs) and individuals who can effectively implement activities and continue to provide support after project resources are no longer available. LIVCD aims to achieve its objective through targeted interventions in three components: 1) Initial Assessment- Analyze Prospects and Select Value Chains; 2) Value Chain Activities to Increase the Competitiveness of Lebanese Value Chains; and 3) Accessing Value Chain Finance. LIVCD was designed, awarded and implemented starting FY2013 prior to the approval of USAID/Lebanon CDCS in 2015. The draft CDCS development objectives (DO) and intermediate results (IR) that LIVCD was designed to align with were: - DO2: Enhance economic opportunity for the poorest segments of Lebanese society, particularly in areas outside of metro Beirut o IR 2.2: Enhanced competitiveness that expands micro-, small- and medium-sized enterprises  Sub-IR 2.2.1: Increased enterprise/producer productivity in targeted value chains o IR 2.3: Increased access to finance for micro-, small- and medium-sized enterprises As the CDCS was finalized, this resulted in updated results statements. Based on the approved CDCS, the DO and IRs LIVCD should contribute to are as follows: - DO2: Inclusive Economic Growth Enhanced o IR 2.1 Increased private sector competitiveness  Sub-IR 2.1.2 Increased business and trade linkages  Sub-IR 2.1.3 Increased workforce development, linked to job opportunities o IR 2.2: Increased access to finance, especially for business start-ups and women entrepreneurs  Sub-IR 2.2.1: Increased early stage investment financing for new and existing firms A link to the LIVCD Logical Framework can be found in Annex 1. SCOPE OF THE PROJECT Building on recent support to the rural sector through agricultural and other related economic development projects, LIVCD partners, together with private sector are to develop functioning and competitive value chains in order to increase the income of the rural population and local businesses such as farmers, agro-processors, input suppliers, transporters, exporters, small businessmen, service providers, retailers, and other entities involved in rural wealth creation. Entirely market-based, the project works throughout Lebanon to raise the competitiveness of selected rural value chains. LIVCD assists Micro, Small and Medium Enterprises (MSMEs) to access business development services including access to new technologies, training and equipment, and other assets to improve productivity and improve product quality to reach export markets. These gains are enhanced by facilitating linkages between value chain actors with high quality products and those with access to markets. Finally, LIVCD supports access to finance, through grants, as well as by introducing financial services from banks and 3 | LIVCD FINAL PERFORMANCE EVALUATION SOW USAID.GOV Micro-Finance Institutions (MFIs) and assisting farmers and MSMEs in preparing loan applications. To achieve its objective, LIVCD collaborates with actors within the selected value chains by providing technical assistance, equipment, grants, and access to investment for value chain financing in order to overcome constraints to competitiveness. The LIVCD activities are designed to achieve the following LIVCD IRs: • Increased access to markets in target value chains • Increased business linkages in target value chains • Increased productivity in target value chains • Constraints to lending and investment reduced in target value chains. In May 2013, eight value chains were approved for inclusion in the project. Those value chains include six that are focused on agriculture, one on processed food, and one on rural tourism and handicrafts: 1. Pome Fruits 2. Stone Fruits (Cherries and Avocados) 3. Grapes 4. Olive Oil 5. Floriculture 6. Rural Basket of Products (including honey, eggs, pine nuts and herbs) 7. Processed Foods 8. Rural Tourism and Handicrafts In May 2014, LIVCD updated the floriculture value chain assessment to decide whether to include it in the LIVCD portfolio. Based on the results of the assessment, LIVCD with the approval of USAID eliminated floriculture from LIVCD’s portfolio. INSTITUTIONAL CONTEXT LIVCD is implemented by DAI and its two approved subcontractors1, Key Development Services and Making Cents: • Key Development Services. KDS supports LIVCD in project Monitoring and Evaluation (M&E), project communications (i.e. regular reporting, success stories, value chain assessment facilitation, and value chain marketing support.) • Making Cents. Making Cents supports LIVCD in applying its capacity-building, training-of￾trainers approach and its agricultural enterprise curriculum to the needs of producer organizations and other enterprises servicing specific food chains in Lebanon. 1 The partnership with the two approved sub-contractors has changed. Reference to both contractors was left because for more than two years, they were involved in the project. USAID.GOV LIVCD FINAL PERFORMANCE EVALUATION SOW 4 To achieve the project goals and objectives, DAI works closely with local entities, non-governmental organizations and foundations. Also, using a “partner-led, LIVCD-assist” approach, the private sector leads sustainable, demand-driven strategies. LIVCD PARTNERS The project’s implementation approach entails partnering with a range of local partners. LIVCD entered into Public-Private Partnerships to leverage private sector investment in the targeted value chains. Most of these PPPs are with firms, cooperatives and NGOs who are actors in the targeted value chain linked to farmers. • Firms. LIVCD partnered with a range of firms in the targeted value chains to upgrade competitiveness, including agricultural producers, intermediaries/aggregators of agricultural products, other agribusinesses and business service providers, handicrafts and rural tourism enterprises, and other relevant firms. LIVCD partnered with financial institutions also to hold workshops at which the financial institutions present farmers with information about their loan products appropriate to invest in LIVCD-promoted technologies and upgrades, such as hives and new equipment in the beekeeping sector; new varieties, equipment and other orchard upgrades for avocado and grapes. • Cooperatives. LIVCD partnered with entrepreneurial cooperatives that have a range of functions in production and marketing of the targeted value chains. Cooperatives could be working in only one function, while others could be providing integrated services. • Non-governmental organizations. LIVCD partnered with NGOs to implement project activities through grant mechanisms, and work to build their capability to effectively manage USAID funded development programs, projects and activities in the future. Building capacities include bringing up their level to the U.S. and international standards in terms of structure, accountability, human resources and transparency of financial and project management systems. • Foundations. A number of Foundations are active in supporting farmers in some rural areas. LIVCD is to coordinate with these Foundations where appropriate, and in some cases formal partnerships are to be established. • Universities: LIVCD collaborates with universities to encourage the adoption of improved production practices, support purchase of needed equipment, and develop and implement improved production methodologies. LIVCD also develops internship and collaborative programs with agricultural departments at universities in Lebanon to increase students’ awareness and interest. Coordination with government includes: o Local Government (Municipalities). LIVCD coordinates with municipalities to implement interventions in the targeted value chains. LIVCD has planned to have formal partnerships with municipalities in the Rural Tourism and Rural Basket value chains, and in other value chains as opportunities arise. o Ministries. Per USAID policy, LIVCD does not have a formal relationship (such as a Memorandum of Understanding) with the Ministry of Agriculture or Ministry of Tourism. However, the project is to track government policy as it relates to the target value chains, and potential opportunities and constraints for value chain actors. 5 | LIVCD FINAL PERFORMANCE EVALUATION SOW USAID.GOV DEVELOPMENT HYPOTHESIS There was no development hypothesis written for LIVCD during the design phase or in the program description. A development hypothesis was later added during the development of the LIVCD M&E Plan. The hypothesis states that “IF actors including MSMEs in the target value chains increase investment into their businesses to increase productivity, make new business linkages, and identify and enter new markets, THEN the value chain will be more competitive and MSME incomes will rise.” PROJECT INTENDED RESULTS Through partnership with the private sector, LIVCD project is to develop and select fully functioning and competitive value chains in order to increase the income of the rural population as well as rural, peri-urban, and/or urban businesses. Entirely market-based, the project will work throughout Lebanon to raise the competitiveness of selected rural value chains. It is expected that by the end of the project in January 2019, as a result of project interventions, the following results will be achieved: • A minimum of seven fully functioning, competitive value chains. • An increase of at least 5,000 businesses or micro-enterprises benefiting from horizontal and vertical linkages. • At least 12,500 small and medium commercial growers benefiting from the dissemination of improved production and post-harvest technologies, and MSMEs who have applied improved technologies and management practices. • At least 150 new export markets, niche markets, or distribution channels for selected value chain products. LIVCD collaborated with value chain actors to develop the Value Chain Upgrading Strategies to overcome constraints to competitiveness and achieve the LIVCD objectives, using the following causal model methodology: USAID.GOV LIVCD FINAL PERFORMANCE EVALUATION SOW 6 Achievements of LIVCD are measured by 14 indicators; 7 of which are FTF indicators; and 3 are custom FTF based indicators. The FTF indicators against which the activity collects data include: - Value of exports of targeted agricultural commodities as a results of USG assistance; - Gross margin per hectare, animal or cage of selected product; - Number of farmers and others who have applied improved technologies or management practices as a result of USG assistance; - Number of private enterprises, producers organizations, water users associations, women’s groups, trade and business associations and community-based organizations (CBOs) that applied improved technologies or management practices as a result of USG assistance; - Number of individuals who have received USG- supported short-term agricultural sector productivity or food security training; - Number of public-private partnerships formed as a result of FTF assistance; - Value of new private sector investment in the agriculture sector or food chain leveraged by FTF implementation. Other indicators against which the activity collects data include: - Number of MSMEs, including farmers, and other organizations receiving business development services from USG assisted sources - (custom) based on FTF 4.5.2-37 (S) - Value of sales collected at farm-level for small holders and firm-level for MSMEs attributed to LIVCD implementation - (custom) Constraints to Value Chain Competitiveness Identified in Value Chain Assessments LIVCD Objective INTERVENTIONS Assumptions OUTPUTS OUTCOMES Interventions (Upgrading Strategy) designed to overcome specific constraints & meet LIVCD Objectives Interventions Produce Outputs (Services, Products) Outputs Lead to Outcomes: Change (benefit) - Constraints to competitiveness overcome Overcoming Constraints to Competitiveness Leads to achievement of LIVCD Objective Grants, Technical Assistance Actors Trained, Assisted Increased sales, exports, jobs; markets accessed, productivity increased, technology applied, investments leveraged 7 | LIVCD FINAL PERFORMANCE EVALUATION SOW USAID.GOV - Proportion of female participants in USG-assisted programs (Custom based on GNDR-2 ) - Number of new or improved jobs as a result of LIVCD assistance - (custom) based of FTF 4.5.2.(S) - Number of export markets, or distribution channels for selected products accessed as a result of USG assistance - Number of MSMEs, including farmers, benefiting from new horizontal & vertical linkages - Number of MSMEs, including farmers, assisted by USG to apply for value chain finance - Number of enterprise/farmers that successfully accessed loans, private equity, or both as a result of USAID assistance - Total value of loans issued as a result of USG assistance The Performance Data Table can be found in Annex 2. CRITICAL ASSUMPTIONS Critical assumptions listed in the M&E Plan are: • A stable national and regional political scene will not impede ability to export; • Policies of the Government of Lebanon relevant to the target value chains will not change in a way that constrains the identified opportunities for value chain actors; • Policies of governments of export will not change in a way that constrains the ability of Lebanese exporters to export to these markets; • Product quality (including food safety standards) demanded, and prices, of LIVCD targeted products will not change significantly in target markets; • Environmental factors such as weather, new diseases/pests and other factors will not affect production quantity or qualities. EXISTING INFORMATION - Country Development Cooperation Strategy (CDCS) USAID/Lebanon - LIVCD Contract and contract modifications - LIVCD RIG Audit Report - 2014 - LIVCD Work Plans - LIVCD M&E Plan - LIVCD Periodic reports (Quarterly Reports, Annual Reports, IPPT, etc.) - List of Grants - LIVCD Grants Manual - Data Quality Assessment (DQA), Project Site Visit Reports (PSV), and Data Verification (DV) reports conducted by Social Impact - LIVCD Mid-term Performance Evaluation Report - 2016. - LIVCD Internal Assessment of Economic Impact (by Value Chain) – 2018 - Resource Guide for Gender Integration in Value Chain Development in Lebanon, 2016 USAID.GOV LIVCD FINAL PERFORMANCE EVALUATION SOW 8 EVALUATION OBJECTIVES AND QUESTIONS OBJECTIVES The objective of this final LIVCD evaluation is to document critical successes as well as shortcomings of the project in order to provide the Mission, the USG, and the wider donor-community with usable lessons learned for adaptation under USAID’s Collaborating, Learning, and Adapting (CLA) framework. It should provide the audience with input on the approach and activities implemented under LIVCD in terms of achieving outputs as well as outcomes and the sustainability of its interventions. It will also provide the Mission with a background, conclusions and recommendations package to be able to better plan for future activities and follow-on activities. The report will be conducted independently and will challenge the assumptions underlying and analyze the impact and sustainability of the following main project components: - Value Chain approach in development. To what extent is this model effective and efficient toward achieving cost-effective results sought by the original program objectives? How does it compare with other approaches used by other donors to develop the rural economy through private sector support in a similar context? - Technical assistance approach. How effective and efficient was the type of technical assistance delivered to local partners? To what extent did the project encourage and create a sustainable market for locally led and implemented technical assistance and business development services? - Grants / Partnerships. To what extent did the grants provided to private sector and other partners offer “additionality?” How effectively were project partners chosen and what value did they ultimately add to the outcomes of the project? Were the grants and partnerships chosen conducive to scalability? Meetings with the Mission and the stakeholders will be held prior to the Evaluation kick-off in order to determine priorities, and to add clarity to the evaluation questions based on the Mission stakeholders intended use of the evaluation findings, conclusions and recommendations. EVALUATION QUESTIONS Relevance 1. To what extent were the value chain approach and LIVCD’s objectives and performance indicators appropriate for meeting the CDCS Development Objective? Efficiency and cost-effectiveness 2. To what extent were selected interventions cost-effective? a. Which were the relatively more/less cost-effective interventions? b. Was the project able to achieve more value for money than alternative approaches (achieve the same results at a lower cost)? If yes, what would have been the alternatives? 9 | LIVCD FINAL PERFORMANCE EVALUATION SOW USAID.GOV Effectiveness 3. To what extent have LIVCD’s interventions resulted in meeting the objectives and Key Performance Indicators (KPIs) of the initial and extended contract, including: a. Fully functioning and competitive value chains as defined by the contract? b. Value chain upgrades and new leveraged private sector investments, leading to improved competitiveness, increased incomes, and impacted/created jobs? c. Improved access to local and international markets for the supported value chain products? 4. Were the selected interventions (Technical assistance/Grants) and entry points in each value chain strategic and optimal in leading to the intended results? Could LIVCD have achieved outcomes and results without grants/partnerships, solely using TA? 5. What were the internal and external factors that contributed to the achievement (or non￾achievement) of the above results? Sustainability, scalability, additionality 6. To what extent are LIVCD’s interventions sustainable? What major long-term results did these interventions generate? To what extent was the sustainability of results affected by the selection of value chain partners and stakeholders? Were the positive results spread across all beneficiaries, or mainly to a few beneficiaries? 7. To what extent are the project’s interventions scalable? Which particular aspects of the project have resulted in growing replication at declining marginal cost? Was there any particular aspect of the project that ended up being more “one-off”, i.e. not scalable? 8. To what extent did interventions demonstrate “additionality”2 ? LESSONS LEARNED AND ADAPTIVE MANAGEMENT The evaluation is to provide insights on the capacity of the LIVCD management to adapt and to adjust to implementation challenges as they arose. More specifically the evaluation is to identify the major 2 USAID requires grants and other interventions, especially to private sector entities, to demonstrate "additionality", or evidence that similar outcomes targeted by the grant or intervention would not have been realized in the absence of the USAID intervention. Simply citing the likely benefits and outcomes attributed to the grant/intervention is not sufficient justification in and of it, as it does not meet the additionality requirement. Evidence of additionality is normally provided by offering evidence that the purposes or proceeds of the grant/intervention: 1) Could not, or likely would not have, been financed or purchased by the proposed beneficiary without USAID support; 2) Serve a risk-reducing purpose, catalyzing the beneficiary to make an investment in a mostly unfamiliar technology with locally unproven benefits; 3) Produce a demonstration effect, meaning that the success of the grant will induce significant numbers of others to copy the intervention, thus magnifying its impact; 4) Allows the beneficiary to access finance that they otherwise could not have obtained; 5) Crowds in outside resources/investment by offering a missing component to a transaction that induces other parties to participate and dedicate resources to the activity; 6) Adds a non-commercial but important development component to a larger activity. USAID.GOV LIVCD FINAL PERFORMANCE EVALUATION SOW 10 challenges faced and what lessons can be generated from the way LIVCD team addressed those challenges. The evaluation is to address the extent and effect to which the CLA aspect was incorporated in LIVCD; and whether the activity level assessments and success stories were useful and well-done. Also, the evaluation is to inform the Mission on how better results for similar exercises can be produced and communicated as a more useful means for CLA. RECOMMENDATIONS Finally, the evaluation is to provide recommendations on the following: - Future interventions that will effectively and efficiently address any remaining gaps within the targeted value chains. - How future activities in the size (amount value) of LIVCD would be able to successfully achieve their objectives while contributing to the Mission objectives. - Measures that can be taken to address the challenges faced during the implementation and particularly when dealing with partners at the value chains levels. EVALUATION DESIGN AND METHOD This evaluation will rely on review of project documentation (e.g. RFA, CA, work plan, quarterly reports, annual report, and technical reports) and data collected from interviews with the Mission, project staff, project beneficiaries and other stakeholders. Site visits, direct observations, and other mixed methods outlined in the evaluation matrix below will be employed to inform the evaluation. These approaches should allow the evaluation team adequate information to assess the project’s implementation and results against its objectives, scope of work, deliverables, work plan, etc. and to determine whether project activities were accomplished, how well activities were implemented and to what extent they meet with targets set in project documentation. Data collected during this evaluation should also provide information on whether the target audience was reached, and how external and internal factors affected project implementation and results. In accordance with ADS 205.3.8.2, this evaluation is to adequately capture the situations and experiences of both males and females. The evaluation should be examining the extent to which gender equality was achieved as a result of the activities that were implemented. ILLUSTRATIVE DATA COLLECTION METHODS AND TOOLS The evaluation is expected to use a mixed methods approach. The methodology for this evaluation will draw upon qualitative methods, including a desk review, key informant interviews (KII) and focus group discussions (FGD). Methods that will be used for data collection are the following: - Desk review: The evaluation team will do a systematic literature search to identify evaluations, tools and other documents on agricultural value chain projects; the evaluation team will also review existing documents including CDCS, RFA, contract agreement, contract modification, , RIG Audit report, , initial assessments, periodic reports (annual, quarterly, other) and all other reports produced by USAID, Social Impact, and LIVCD as well as previous evaluation reports of similar projects if available to determine service delivery mechanisms, quality of project results, alignment with CDCS, validity of development 11 | LIVCD FINAL PERFORMANCE EVALUATION SOW USAID.GOV hypothesis, quality of program management and the progress against project targets. - Interviews of LIVCD and USAID Key Staff: KII to obtain more detailed, in-depth understanding about specific issues will be conducted with key personnel, such as USAID Economic Growth Office, LIVCD COR, DAI/LIVCD COP, DAI/LIVCD M&E team, gender focal point, grants team representatives and other LIVCD technical team. Discussions will focus on how the value chains and beneficiaries were selected, how the gender gaps were identified and addressed, how the activities were designed and implemented as well as suggestions for strengthening the value chain approach in the future. Interview with other key stakeholders will also be conducted using a structured questionnaire as a guide during interviews. - Focus Group Discussions with LIVCD beneficiaries and stakeholders: The Evaluation Team will conduct focus group meetings (as appropriate) with selected value chain actors and the project beneficiaries (cooperatives, NGOs, private sector entities, individual farmers, etc.) Discussions will be derived from the initial findings of KIIs. -Direct site’s observations: The evaluation team will undertake field visits to project sites and direct value chain beneficiaries utilizing a direct observation guide to obtain a first-hand experience of project outcomes. DATA ANALYSIS METHOD The data gathered from the KII and the FGD will be used as the basis for data analysis. Techniques in data analysis will include review of existing documentation obtained, conceptualizing, coding and categorizing of information obtained in the KII and FGD, examining relationships between the different activity components, activities and achievements as revealed by stakeholders, and authenticating conclusions by triangulation of data obtained from different sources. The evaluation team will prepare a data source matrix (also called an evaluation matrix) that clarifies how the data is collected. It will include the evaluation questions, the evaluation tool(s), data source(s) and analysis plan for each question. An illustrative version of the matrix is provided below. The technical expert will consider the illustrative evaluation matrix and propose a final matrix for consensual agreement by the evaluation team. To the extent possible given time and budget the methods used may be revised in response to preliminary findings or obstacles that prevent timely implementation of the proposed method. This matrix will ensure that a multitude of data sources are considered and the team will be able to use triangulation of data to answer each question with enriched validity. All the data collected from different sources will be reviewed for reliability and validity and findings will be compared using multiple methods, data forms and sources of data. Data will be organized to answer evaluation questions in the final report. Differentiation between the findings, interpretations, judgments, recommendations, and the specific sources of evidence supporting these, should be made clear in the final evaluation report. USAID.GOV LIVCD FINAL PERFORMANCE EVALUATION SOW 12 Table 1. Evaluation Matrix (illustrative Template) Evaluation criteria Evaluation questions related to the criteria Sub question/ Illustrative queries Data sources Data collection methods, sample and tools Data analysis plan Relevance To what extent were the value chain approach and LIVCD’s objectives and performance indicators appropriate for meeting the CDCS Development Objective? -RFA -LIVCD Project Proposal -LIVCD Project modification -LIVCD Project Documents (Workplans, Annual Reported, Quarterly Reports, M&E plans, M&E reports) - LIVCD Initial Assessments -LIVCD Mid-Term Evaluation -KII with USAID staff, stakeholders and technical experts Secondary Sources - Desk review of project documents Primary Sources - Key Informant Interviews (KIIs) with LIVCD staff, technical experts and USAID stakeholders - Qualitative analysis Efficiency and cost￾effectiveness To what extent were selected interventions cost-effective? Which were the relatively more/less cost-effective interventions? - LIVCD Project modification -LIVCD Project Documents (Workplans, Annual Reported, Quarterly Reports, M&E plans, M&E reports) - LIVCD Initial Assessments -LIVCD Mid-Term Evaluation Secondary Sources - Desk review of project documents -Mid-term evaluation - LIVCD Internal Assessment of cost-effectiveness (by Value Chain) Primary Sources - Key Informant Interviews (KIIs) with LIVCD and USAID staff - Qualitative analysis Was the project able to achieve more value for money than alternative approaches (achieve the same results at a lower cost)? If yes, what would have been the alternatives? Qualitative analysis Effectiveness To what extent have LIVCD’s interventions resulted in meeting the objectives and Key Performance Indicators (KPIs) of the initial and extended contract Objectives and KPIs of the LIVCD that include: a. Fully functioning and competitive value chains as defined by the contract? b. Value chain upgrades and new leveraged private sector investments, leading to improved competitiveness, increased incomes, and impacted/created jobs? -LIVCD Mid-Term Evaluation - KII with USAID and LIVCD staff -KIIs and FGD with LIVCD beneficiaries -LIVCD Project Documents (Workplans, Annual Reported, Quarterly Reports, M&E plans, M&E reports) Primary Sources - Key Informant Interviews (KIIs) with USAID staff Secondary Sources - Desk review of project documents - Mid-term evaluation Qualitative analysis 13 | LIVCD FINAL PERFORMANCE EVALUATION SOW USAID.GOV c. Improved access to local and international markets for the supported value chain products? Were the selected interventions (Technical assistance/Grants) and entry points in each value chain strategic and optimal in leading to the intended results? Could LIVCD have achieved outcomes and results without grants/partnerships, solely using TA? -LIVCD Mid-Term Evaluation - KII with USAID and LIVCD and stakeholders -Project documents: grants and TA databases -KII and FGD with LIVCD grantees and beneficiaries Primary Sources - Key Informant Interviews (KIIs) with USAID staff Secondary Sources - Desk review of project documents -Mid-term evaluation Qualitative analysis What were the internal and external factors that contributed to the achievement (or non￾achievement) of the above results? Could LIVCD have achieved outcomes and results without grants/partnerships, solely using TA? -LIVCD Mid-Term Evaluation - KII with USAID and LIVCD and stakeholders -Project documents: grants and TA databases -KII and FGD with LIVCD grantees and beneficiaries Primary Sources - Key Informant Interviews (KIIs) with USAID staff and grantees Secondary Sources - Desk review of project documents -Mid-term evaluation Qualitative analysis Sustainability, scalability, additionality To what extent are LIVCD’s interventions sustainable? What major long-term results would these interventions generate? To what extent was the sustainability of results affected by the - LIVCD Mid-Term Evaluation - KII with USAID and LIVCD and stakeholders Primary Sources Qualitative analysis USAID.GOV LIVCD FINAL PERFORMANCE EVALUATION SOW 14 selection of value chain partners and stakeholders? To what extent are the project’s interventions scalable? Which particular aspects of the project have resulted in growing replication at declining marginal cost? Was there any particular aspect of the project that ended up being more “one-off”, i.e. not scalable? -LIVCD Mid-Term Evaluation - KII with USAID and LIVCD staff -KIIs and FGD with LIVCD grantees and beneficiaries -LIVCD Project Documents (Workplans, Annual Reported, Quarterly Reports, M&E plans, M&E reports) Primary Sources - Key Informant Interviews (KIIs) with USAID staff Secondary Sources - Desk review of project documents - Mid-term evaluation Qualitative analysis To what extent did interventions demonstrate “additionality”? -LIVCD Mid-Term Evaluation - KII with USAID and LIVCD staff -KIIs and FGD with LIVCD grantees and beneficiaries -LIVCD Project Documents (Workplans, Annual Reported, Quarterly Reports, M&E plans, M&E reports) Primary Sources - Key Informant Interviews (KIIs) with USAID staff Secondary Sources - Desk review of project documents - Mid-term evaluation Qualitative analysis 15 | LIVCD FINAL PERFORMANCE EVALUATION SOW USAID.GOV FINAL PERFORMANCE EVALUATION DELIVERABLES - Briefing: The evaluation team will provide a briefing to USAID prior to the start of the evaluation and at the end of the evaluation. The initial briefing will focus on presenting the evaluation team plan for conducting the evaluation. Discussions with the Mission on the proposed methodologies for the evaluation will result in an Inception Report (a deliverable) that states the agreed methodologies. - Inception Report: PMSPL II team will prepare an Inception Report that is the result of kickoff meetings with the Mission and stakeholders. The Inception Report makes final the methodologies, tools, and evaluation work plan and time schedule. The Inception Report will be submitted to the COR at USAID/Lebanon for approval. Approval of the Inception Report signals agreement by the Mission and the evaluation team to use the proposed methodologies and tools with adherence to the work plan and time schedule. The Inception Report is Annex A of the evaluation report. - Interim Findings Presentation: Toward the end of the field work, prior to the development of the Draft Report, the team will carry out a preliminary findings presentation at USAID to discuss findings, conclusions and recommendations. - Draft report: A draft report of the findings and recommendations will be submitted to USAID COR clearly describing findings, conclusions, and recommendations, within two weeks of completing the data collection. USAID will provide comments on the draft report within two weeks of submission. ADS 201 directives state that draft reports “must undergo a peer review organized by the office managing the evaluation”, and that “reports must be shared with implementing partners of the projects or activities addressed in the evaluation.” The evaluation team will consider USAID comments and the comments of other stakeholders and revise the draft report accordingly and as appropriate for an independent evaluation. Where differences of opinion exist these will be presented in a Statement of Differences Annex. - Final Report: The final evaluation report will be provided to USAID in MS Word and Adobe PDF within 15 calendar days following receipt of final comments from USAID. The report will include all of the components outlined in the “Structure of the Evaluation Report”, but not necessarily in the order specified below. The report will not exceed 25 pages (excluding appendices). Appendices should at a minimum include the Inception Report for the evaluation; a list of individuals interviewed; a complete description of the methodology used for the evaluation; and any data collection used. The report will be submitted in English, electronically in MS Word format and compliant with USAID Graphic Standards. Utilization and Dissemination Plan: The approved final evaluation report will be shared with USAID/Lebanon staff members. The report will follow USAID branding requirements. Evaluations at USAID should be transparent: Findings from evaluations will be shared as widely as possible with a commitment to full and active disclosure. ADS 201.3.5.15 operationalizes this principle by requiring Operating Units to plan for evaluation dissemination: “Missions and Washington OUs must plan for dissemination and use of the planned evaluation. Missions or Washington OUs should consider how key partners and other development actors would best receive evaluation information and plan their communications products accordingly.” A dissemination plan will be prepared and filled prior the kick off the evaluation and upon the approval of the SOW. USAID.GOV LIVCD FINAL PERFORMANCE EVALUATION SOW 16 As a part of the post Evaluation action plan, the evaluation team will follow up to facilitate and enhance use through one or more of the following steps: o Distribute the final approved evaluation report to DAI, LIVCD and other stakeholders at the discretion of USAID/Lebanon. o Uploaded the final approved evaluation report to the DEC for public access o Develop a utilization plan with primary intended users o Adapt and reformat findings for different audience. o Watch for emergent opportunities to reinforce findings o Continue to build evaluation capacity during the follow up process. o Conduct a client satisfaction survey, o Conduct a tracer survey. REPORTING GUIDELINES USAID’s Evaluation Policy requires that all evaluation SOWs include USAID’s Criteria to Ensure the Quality of the Evaluation Report. The LIVCD final performance evaluation team is advised to incorporate these guidelines in their report where relevant and applicable to the evaluated contract agreement. Guidelines on the Structure of the Evaluation Report The findings from the evaluation will be presented in a draft report at a full briefing with USAID/Lebanon and key stakeholders. The format for the evaluation report is as follows: 1. Abstract of not more than 250 words briefly describing what was evaluated, evaluation questions, methods, and key findings or conclusions. 2. Glossary & Acronyms 3. Table of Contents (1 pp.); 4. Executive Summary: that summarizes key points (purpose and background, evaluation questions, methods, findings, and conclusions). (2 to 5 pp.); 5. Introduction: purpose, audience, and synopsis of task (1 pp.); 6. Background: Brief overview of LIVCD project context, USAID/Lebanon program strategy and activities implemented in response to the development problem, a brief description of LIVCD, and purpose of the evaluation (3 pp.); 7. Methodology: Describes evaluation methods, including constraints and gaps (1 pp.); 8. Key Findings and Conclusions for each evaluation question; including a reference to the data quality of the evidence provided (10 pp.); 9. Issues: provide a list of key technical and/or administrative issues, if any (1 pp.); 10.Recommendations: Identify list of recommendations based on the evaluation findings. 11.Annexes: (A) Scope of Work, (B) Persons Interviewed, (C) Bibliography, D) Data Collection Tools, (E) Data Analysis, (F) Focus Group results, (G) AOB, (H) Statement of Differences TEAM COMPOSITION USAID’s ADS requires that at least one member of every evaluation team be an evaluation specialist. The PMSPL II team will provide evaluation specialization support to the evaluation team backstopped by Social Impact’s home office. 17 | LIVCD FINAL PERFORMANCE EVALUATION SOW USAID.GOV The team will include two experts, one of whom will act as the Team Leader. 1) Team Leader (Expatriate)/ Evaluation Value Chain Expert with a professional background in international development and strong working knowledge of value chain approach, evaluation methodologies, agri-business development, etc. The expert should preferably have work experience in development projects design and delivery in developing countries. Experience working on issues related to gender in the context of economic development is preferable. This person shall be responsible for coordinating and directing the overall evaluation, including preparation and submission of the draft and final evaluation report. He/she should have a minimum of 10 years’ experience in the design, implementation, and/or evaluation of foreign assistance programs including USAID-related value chain programs. He/she should have an advanced degree in agricultural economics, international development, agribusiness or related field. The team leader should have excellent written and oral communication skills in English. Previous overseas experience in the region is desirable. (48 days LOE) 2) Lebanon Agricultural Expert (local) with at least 5 years of experience in value chain development of agriculture and manufacturing sector, private sector development, enterprise development, economic and social analysis of development programs/interventions of donor funded projects in Lebanon. He/She should have an advanced degree in agriculture, agricultural economics, development sciences, business administration or a related field. He/She should have a good understanding of Lebanon’s development sector, agricultural sector, local agricultural value chains as well as public and private agribusiness sector institutions. Knowledge of USAID and other international donors that provide support to the agricultural sector is a bonus. He/ She is expected to develop data collection tools, conduct KII and FGDs, and participate in the analysis and drafting of the evaluation report. He/She is also expected to work with the Team Leader on other aspects of the evaluation (32 days LOE). The evaluation team will consist of a Team Leader/Value Chain Expert, Local Agricultural expert with technical support from PMSPL II evaluation team. PMSPL II staff works alongside the technical expert/evaluator and forms a part of the evaluation team. SI HO has budgetary provision for additional technical evaluation support that might be needed and providing quality assurance for the evaluation through SI’s EQUI (Evaluation, Quality, Use and Impact) initiative. PMSPL II resident staff and HO staff participates in drafting, reviewing and editing the draft final report before circulating to the stakeholders for comment and review. Preparation of the evaluation report is the responsibility of the team leader/technical expert. Final editorial responsibility and quality control for the report is with Social Impact. Comments received are to be address by the team leader and the technical experts supported by the PMSPL II staff. The draft evaluation report is to be reviewed by USAID/Lebanon and, at the request of USAID/Lebanon, a broad range of stakeholders. Comments received after the budgeted end date for this evaluation, may be addressed in a Statement of Differences Annex at the discretion of USAID/Lebanon. PMSPL II proposes to recruit technical experts to collaborate in this evaluation as described above. PMSPL II will certify that there is no conflict of interest or potential conflict of interest with respect to the performance of this evaluation on the part of the contractor and the contractor’s team members. Social Impact will guarantee that substitutions will not be made for individuals proposed as team members without the approval of USAID. USAID.GOV LIVCD FINAL PERFORMANCE EVALUATION SOW 18 PROVISIONAL EVALUATION MANAGEMENT Pre Field-Work (Week 1): PMSPL II staff Identify and recruit the technical experts and assign a team leader. Obtain key documents, establish key contact, liaison with LIVCD to set up interviews with project staff and prepare interview schedule with LIVCD stakeholders and beneficiaries. Evaluation Kick-off (Week 2): The Team Leader arrives in country. PMSPL II COP meets with the Team Leader, Local Expert and PMSPL II evaluation team to assign roles and responsibilities, agree on the evaluation work plan, list of people to be interviewed, number and type of focus group discussions to be carried out, outline of the final evaluation report. During this week the evaluation team will develop the interview guide, the focus group discussion guide and all the necessary forms for collection of informant narratives, design of field work, and strategy for qualitative analysis takes place. Technical advice and counsel are sought from SI HQ at this time. The pre-field work should be accomplished prior to the data collection process. The team will meet with USAID Program Office and COR for an in-country briefing to agree on the evaluation details, methodology and reporting guidelines. Additional documentation may be requested at that time. Logistics for the field visits will also be made final. Field Work (weeks 3 to 5): Interviews with key stakeholders including, USAID key staff, LIVCD staff, partners and sub-contractors will be carried out. The team will also interview project stakeholders and beneficiaries (cooperatives, private sector partners, farmers and other actors within the value chain), who’s work have been impacted by project activities. The evaluation team will also carry out focus group discussion with project beneficiaries and stakeholders. Post Field-Work (weeks 6 to 10): Submission and presentation of the completed draft final report is the responsibility of the Team Leader; an out-brief will be carried out by the team leader and the evaluation team to the Mission. The expat technical expert travels from Lebanon, but remains available to respond to USAID and stakeholder comments as needed. Review Draft Report (week 11): USAID and stakeholders are requested to review the draft report and prepare comments within two weeks of the submission of the final evaluation report. Respond to Draft Report Comment (week 12): PMSPL II responds to comments from USAID and stakeholders revising the report as necessary. Final Report Preparation and Quality Assurance: SI HO applies EQUI quality assurance steps to the draft report. The final report will be submitted no later than two weeks following receipt of final comments from USAID/Lebanon. This includes time needed by SI HQ to edit and review the report prior to submission as a final evaluation report to USAID/Lebanon. The deadline for final report submission is end October 2018. 19 | LIVCD FINAL PERFORMANCE EVALUATION SOW USAID.GOV Table 2. LIVCD Evaluation Illustrative Work Plan LIVCD eval workplan.xlsx USAID.GOV LIVCD FINAL PERFORMANCE EVALUATION SOW 20 Table 3. Illustrative Evaluation LOE/STTA Start date Early August 2018 Final DEC-ready report due Proposed Staff Labor Category Pre-Field –PMSPLII Pre-Field – Evaluation Team Evaluation Kick-Off Fieldwork and Data Collection Post Field Work Final Report Submission Recruitm ent, SOW, Workpla n, etc. Preparation - Reading, Work plan Travel to Leban on In-Brief and final planning Inception Report Interviews, Focus Group Discussions Preliminary analysis Present Main Findings & Debriefin g Travel from Lebanon Data Analysis, Report Writing, Submit Draft to USAID Mission Respond to USAID comments -Submit Final Report to USAID mission Total Team Leader (Expat) Consultant 0 5 2 4 15 5 1 1 10 5 48 Technical Specialist (Local) Consultant 0 3 0 3 15 4 1 0 4 2 32 Social Impact Home Office Support PE Specialist 3 0 0 2 0 1 0 0 2 2 10 PMSPL 2 Evaluation Support M&E Officer / Full time equivalent 5 5 0 3 15 4 1 0 6 5 44 TOTAL LOE 8 13 2 12 45 14 3 1 22 14 134 21 | LIVCD FINAL PERFORMANCE EVALUATION SOW USAID.GOV ANNEX 1: LIVCD LOGICAL FRAMEWORK Table 1. LIVCD Activity Logical Framework Narrative Summary Indicators Data Sources Assumptions CDCS DO2: Inclusive Economic Growth Enhanced LIVCD Goal: Develop functional, competitive value chains to increase incomes of the rural population including MSMEs DO2.a: Value of new private sector investment leveraged by USG assistance Primary/secondary; LIVCD records - Assumes that beneficiaries are able and willing to co-invest with LIVCD. Value of new private sector investment made by farmers leveraged by USG assistance Primary/secondary; LIVCD records - Assumes that beneficiaries are able and willing to co-invest with LIVCD. DO2.b: Number of new and better jobs attributed to USG assistance Primary/secondary; Review of LIVCD Records - A created/ better (impacted) job is assumed to lead to higher and/or more stable income for the job holder; opportunity cost is assumed to be zero. - Assumes that valid employment data can be collected from Partners. Number of Jobs Impacted by LIVCD Implementation Primary/secondary; Review of LIVCD Records - Assumes that valid employment data can be collected from Partners. DO2.c: Proportion of female participants in USG-assisted programs designed to increase access to productive economic resources Primary; Review of LIVCD Records - Assumes that the participation of women will be visible to LIVCD staff and partners, and can be recorded. CDCS IR2.1: Increased private sector competitiveness LIVCD Objective 1: Increased Competitiveness of Lebanese value chains IR.1.a: Number of beneficiaries who have applied new technologies or management practices as a result of USG assistance Primary; Survey of direct beneficiaries; activity or association records, farm records (if available) - Assumes that application of new technologies or management practices will be sustained beyond the life of the activity. IR.1.b: Number of MSMEs receiving business development services from USAID-assisted sources Primary; Data will be collected from LIVCD Records - Assumes that receipt of business development services supported by LIVCD is a good indication of the reach of LIVCD activities in target value chains and geographic areas. IR2.1.c: Value of sales Primary / secondary; Census - The regional situation will remain static in USAID.GOV LIVCD FINAL PERFORMANCE EVALUATION SOW 22 collected at farm-level for small holders and firm –level for MSMEs as a result of USG assistance or sample survey of beneficiaries (depending on value chain), with verification of beneficiary records of sales (quantity and value) for supporting documentation when available terms of export limitation, and impacts on agricultural trade especially with Syria and Egypt. - Assumes that a valid baseline can be established - Assumes that valid data can be collected from Direct Beneficiaries - Assumes that valid sales data can be collected from Direct Beneficiaries. Number of individuals who have received USG￾supported short-term training Primary; LIVCD records - Assumes that the training will be designed appropriately to impart new skills, knowledge or attitude to beneficiaries as expected. CDCS Sub-IR2.1.2: Increased Business and trade Linkages LIVCD IR1: Increased access to markets Number of export markets or distribution channels for selected products accessed as a results of USG assistance Primary; LIVCD Direct Beneficiary records (invoices other documents demonstrating sales to new markets or distribution channels) - Assumes that partners are able and willing to achieve upgrades necessary to enter new markets. LIVCD IR2: Increased business linkages Sub-IR2.1.1.a: Number of MSMEs, including farmers, and other organizations benefiting from new horizontal and vertical linkages Primary; LIVCD Direct Beneficiary records (invoices and receipts demonstrating commercial transactions; cooperative/producer organization membership documents or statements of informal cooperation) for supporting documentation LIVCD direct beneficiary interviews - Assumes that these linkages are ‘win-win’ and not predatory LIVCD IR3:Increased productivities Gross margin per hectare, animal or cage of selected product Primary; Data will be collected from a sample surveys (baseline and follow-up) of LIVCD Direct Beneficiaries, with collection of Direct Beneficiary records indicating - Assumes that partners are willing and able to achieve upgrades necessary to improve productivity. - Assumes that valid data can be collected from Direct Beneficiaries. 23 | LIVCD FINAL PERFORMANCE EVALUATION SOW USAID.GOV production volumes. Total production, Value of sales, Quantity (volume) of sales, Input costs, and Units of production , value and quantity of sales, production costs, units of production for supporting documentation, where Direct Beneficiaries have written documentation. CDCS IR2.2: Increased access to finance, especially for new business start-ups and women LIVCD Objective 2: Increased access to value chain finance IR2.2.a: Number of enterprises that have successfully accessed loans, private equity, or both as a result of USAID assistance Secondary CDCS Sub-IR2.2.2: Increased early stage investment financing for new and existing reduced LIVCD IR4: Constraints to lending and investment reduced Sub-IR2.2.2.a Total value of loans issued as a result of USG assistance Secondary Number of public-private partnerships formed as a result of FTF assistance LIVCD Primary; LIVCD records - Assumes that beneficiaries are able and willing to enter into partnerships with LIVCD to achieve a common development objective that contributes to LIVCD’s objective. Number of MSMEs, including farmers, reached by USG to apply for value chain finance Primary; LIVCD records - Assumes that beneficiaries are able and willing to apply for financing. USAID.GOV LIVCD FINAL PERFORMANCE EVALUATION SOW 24 ANNEX 2: LIVCD PERFORMANCE INDICATORS TARGETS # Indicator Title LOP Target – End of January 2019 1 Number of MSMEs, including farmers, and other organizations receiving business development services from USG assisted sources 17,835 2 Value of new private sector investment leveraged by USG assistance $12,828,000 3 Value of investment made by farmers and other value chain actors leveraged by LIVCD implementation. $20,830,000 4 Number of jobs impacted by LIVCD implementation 9,195 5 Proportion of female participants in USG-assisted programs designed to increase access to productive economic resources (assets, credit, income or employment) 4,129 6 Number of beneficiaries who have applied improved technologies or management practices as a result of USG assistance 12,960 7 Value of sales (collected at farm-level for small holders and firm-level for MSMEs) as a result of USG assistance $160,000,000 8 Number of individuals who have received USG- supported short-term training. 16,420 9 Number of export markets, or distribution channels for selected products accessed as a result of USG assistance 199 10 Number of MSMEs, including farmers, and other organizations benefiting from new horizontal & vertical linkages 5,745 11 Gross margin per hectare, animal or cage of selected product 10% 12 Number of enterprise/farmers that successfully accessed loans, private equity, or both as a result of USAID assistance 102 13 Total value of loans issued as a result of USG assistance $19,870,000 14 Number of public-private partnerships formed as a result of FtF assistance 205 15 Number of MSMEs, including farmers, reached by USG to apply for value chain finance 1,337 Coversheet and introductory statement *Names and position titles will be documented in a separate document to protect anonymity Interview Code: Date of Interview: Time Start: Team (Sub-Team A, B): Time End: Interviewer(s): Location: Note taker(s): Category of Interviewee: Age (if group interview, list all): No. of Interviewees _______ M _____ F______ Good morning/afternoon and thank you for meeting with us today. As mentioned during our interview request, we are working with USAID to conduct the final evaluation of the LIVCD Project. We work for a company called Social Impact. The evaluation is intended to provide an informed assessment of LIVCD progress to date learn lessons for new projects if needed. It is very important for you to know that this study will not determine if you will receive any services/support/funding now or in the future. The way you answer our questions will not determine if you will receive any kind of government or NGO assistance now or in the future. There is no direct benefit to you or your community for your participation in this study, besides toward the goal of improving this project or new project. Our team has had the opportunity to review some background documents to get a better sense of the design and implementation of the project. However, these documents can only tell us so much. We would like to speak with you today to hear about your experience, in your own words, in order to help us better understand how these projects look and function “on the ground.” It is important to understand that while we would like your help in this study, you do not have to take part if you do not want to, and you do not have to answer any questions if you do not feel comfortable doing so. If you decide to take part, your responses will be kept strictly confidential. This means that your name will not be mentioned anywhere in the report, and will not be provided to anyone, including USAID/DAI/LIVCD or anyone in your community. Specifically: • We will collect individuals’ names and the farmer group they belong to. A list of focus group participants will be made available as an annex to the final evaluation report, but those names and positions will not be associated to any particular findings or statements in the report. • We may include quotes from respondents in the evaluation report, but will not link individual names, organizations, or personally identifiable information to those quotes, unless express written consent is granted by the respondent. Should the team desire to use a particular quote, photograph, or identifiable information in the report, the evaluators will contact the respondent(s) for permission to do so. • All data gathered will be used for the sole purposes of this evaluation, and will not be shared with other audiences or used for any other purpose. Your participation in this focus group discussion is voluntary and if you do not feel comfortable answering a particular question, please let us know and we will simply go on to the next question. Do you have any questions for us before we get started? If you have questions or concerns about the study after we leave today, you can contact [name of national or international evaluation team member]; the contact information is on the card I gave you. KII QUESTIONS BY RESPONDENT CATEGORY Notes to Facilitator/Interviewer The interview should be led by the interviewee, and only guided by the interviewer. The interview should be a relaxed conversation. The topics reported below present a list of topics to be covered during interviews. They also include concrete questions to guide the conversation and to cover each topic. The reason for focusing on topics rather than on questions is that the way in which a question would need to be formulated varies considerably depending on the interviewee’s specific knowledge, attitude and circumstances. The listed topics will allow the interviewer to achieve clear predetermined interview goals, but the actual questions will be adapted –as one does in a normal conversation- to the conversation tone, interviewee’s attitude, and knowledge. It is to be expected that the same key issue may naturally be transformed into several questions with one interviewee and only into one question with another. Processed food companies Work with LIVCD 1. Please tell the main produce or services of your companies 2. Please discuss your experiences with LIVCD: 3. Which kind of assistance did you receive from LIVCD (prompt: trainings, technical assistance, grant)? 4. Can you please describe the process and time-line of the grant you received? Processing 5. Did you change any processing process with the assistance of LIVCD? Please, describe? 6. Which are the main benefit of the new processing equipment? 7. How much have you invested to upgrade your processing equipment? 8. Did you attend a training organized by LIVCD in GMP? If so, to what extent did you find it useful? 9. Did LIVCD provided you any assistance to get certifications (ISO, etc.)? Was it useful? Employment 10. How many employees does your company have now? (how many males and how many females) 11. How many employees did the company have before LIVCD assistance? 12. If there is a change in the number of employees, what is the main reason? 13. To what extent the change in the number of employees is due to the assistance received by LIVCD? Market 14. Which is your main market (e.g. domestic, international, retail, wholesale, etc.)? 15. Did LIVCD provided you any assistance to increase the number of customers? 16. Did LIVCD provided you any assistance to enter in a new market? 17. Did LIVCD provided you any assistance to use a new distribution channel? 18. To what extent was the help provided by LIVCD on market aspects effective? 19. May I ask you how much was your turnover in 2017? 20. And before LIVCD assistance? 21. If there has been a change in turnover, to what extent do you think this change is due to the assistance of LIVCD? 22. Was the GMP useful to get new customers and to access new market? Sourcing 23. How many tons of xxx (main raw material, like cucumber for pickles companies, apples for juice companies, wheat for freekeh cooperatives) did you process in 2017? [The interviewer should clarify where the responded is using unit of finished product or raw material) 24. How many tons of xxx (main product) did you process before LIVCD assistance? 25. If there has been a change in the quantity of produced processed, to what extent do you think this change is due to the assistance of LIVCD? 26. May I ask you how do you source row material (e.g. honey, cucumbers, apple, grape, wheat, etc,)? from individual farmers? From cooperatives? From your own production? 27. In case you rent out your equipment, how many farmers are using your services? 28. How much do you charge for the use of your equipment? How much were you charging before LIVCD? 29. Do you have long-term contract with your suppliers of xxxx (main raw material)? 30. When is the price of the raw material set? At the moment of the delivery? Before planting? 31. If buying from individual farmers, from how many farmers did you buy the main raw material (e.g. cucumber) in 2017? And in before LIVCD assistance? 32. How much are you paying xxx (main raw material) now? 33. How much were you paying xxx (main raw material) before LIVCD? 34. If there has been a change in price, which was the main cause? 35. Do you provide seeds or any other agricultural inputs to the farmers that supply? 36. If so, do you require to be paid at the delivery of the seeds or after the harvest? 37. To how many farmers do you provide seeds or any other agricultural inputs and you ask to be paid only after the harvest? Finance 38. Which is your main source of finance? 39. Did you apply for any loan during the last few years? With which bank? 40. Did LIVCD provided you any assistance to apply for a loan? (prompt: development of feasibility studies, linkages with banks, trainings and workshops on loan products with bank officers)? 41. To what extent did you find the activities conducted by LIVCD to facilitate access to credit useful? Gender: 42. How do you see men’s and women’s roles differently in this sector? 43. How do you see men’s and women’s roles differently In this specific project? a. In managmenet/ decision making processes b. In production/ processing c. In marketing Additionality 44. How much was the value of the grant hat you received? Which was your contribution? 45. Would you have made the same investments without USAID’ assistance? Why? 46. As far as you know did some of your competitors replicated what you did with LIVCD approach? Conclusions and recommendations 47. Did you have experiences with other donors or international organizations? If yes, can you please tell me the main difference? Can you please tell me whether you think the approach used by the other donors you mentioned worked better or worse for you and why? 48. Do you have any recommendations for projects like this? 49. In the hypothetical case that a project like this were to come again in the future, what do you think should be done differently? USAID COR Introduction 1. Please, tell us since when you have been the COR and your main responsibility Communication with the project 2. Who do you communicate most frequently with at LIVCD? 3. To what extent do you receive all the information you need from LIVCD to do your job? What additional information would you like to receive that you do not? 4. To what extent do you have enough monitoring data to make good decisions about how to improve the achievement of project outputs? Why or why not? Project Coordination 5. Until the MTE the project had almost no relationship with the Ministry of Agriculture and the government in general. What is the current state of relationship with the government? If relationships have been established, can you tell us why? 6. Is there any coordination mechanism with the government of Lebanon and other donors’ project? Project achievements 7. Which are the project achievements do you consider are the most important? 8. Is there any specific external factor that hindered project achievements? 9. Where do you feel the project is falling short in terms of achievements? Why? 10. Is there any external factor that hindered project achievements? 11. Why were the impact assessment studies conducted? Efficiency of the project 12. Please describe how the project uses consultants (international and local experts), grantees, sub￾contractors, etc. 13. Which is your perception regarding cost-effectiveness of the project? Which interventions do you feel are more cost-effective and which less? Why? 14. In which value chain do you feel that the entry points (TA and grantees) were effectively chosen? 15. In which value chain do you feel that the choice of the entry points (TA and grantees) was more questionable? Sustainability 16. In which value chain do you feel that interventions are more sustainable? Why? 17. In which value chain do you feel that interventions are less sustainable? Why? 18. What sustainability plans has the project developed thus far, if any? Gender: 19. How do you see men’s and women’s roles differently in this sector? 20. How do you see men’s and women’s roles differently In this specific project? a. In management/ decision making processes b. In the production/ processing c. In the marketing Challenges 21. Which are the main challenges that affected the project? 22. How has the Lebanese context changed, affecting markets or project participants, if at all? Conclusions and recommendations 23. In the hypothetical case that a project like this were to come again the future, , what would you like to see done differently? 24. Any other aspects you think it is important for us to know? LIVCD COP 1. To which extent do you think that LIVCD Value chain approach and achievements contributed to enhancing inclusive economic growth (CDCS objective)? i.e. increasing private sector competitiveness and increased access to finance. 2. To what extent was LIVCD able to achieve its stated objectives: 3. Fully functioning and competitive value chains as defined by the contract? 4. Value chain upgrades and new leveraged private sector investments, leading to improved competitiveness, increased incomes, and impacted/created jobs? 5. Improved access to local and international markets for the supported value chain products? 6. What were the internal and external factors that contributed to the achievement (or non￾achievement) of the above results? 7. How was the TA vs Grants used as mechanisms in the implementation? Which of the 2 was easier to be managed? Which one was more conducive to the achievement of the project objective? Any differences noted from one VC to another? 8. How were the entry points identified and selected in each value chain? Were there any challenges in selecting the entry points? 9. Considering the overall budget of your award, and their line items, would you say that the budget was adequate for the achievement of the objectives? How? (If not adequate, what should have been different?) 10. How do you evaluate the economic impact assessment that were recently conducted? Were the time and the resources sufficient? 11. Which were the relatively more/less cost-effective interventions taking into consideration all direct costs on the project and the analysis presented in the impact assessment? (reference economic impact assessments). 12. To what extent you feel that the project would have been able to achieve more value for money than alternative approaches? 13. On the basis of your experience also with other donor (if applicable), which alternative implementation approaches could have been considered by USAID for projects with similar objectives? Why? 14. To what extent you felt that the project M&E system was useful to take strategic decisions and to guide the implementation of the project? 15. Was the project M&E system mainly used for accountability purposes (towards USAID) or as a project management tool? Why? 16. Do you think that some key indicators were not included at the design stage? Which one? 17. In the hypothetical case that a project like this were to come again in the future, what would like to see that is done differently? 18. How do you see men’s and women’s roles differently in this sector? 19. How do you see men’s and women’s roles differently In this specific project? a. In management/ decision making processes b. In production/ processing c. In marketing 20. You arrived more or less half-way during the implementation of the project. In the hypothetical case you had implemented this project since the beginning what do you think should have done differently during the first two years of the project and during the design phase?12. Can you please give us your personal email address in case the team leader of the evaluation wants to contact you later? LIVCD DCOP Involvement in the project 1. Can you please tell me your main functions and responsibility? 2. Do you have any responsibility in procurement? Which one? 3. Do you have any function or responsibility for grant processing? 4. To what extent do you feel that the grant processing system was fast and smooth? If not why? 5. What did Frank meant with this sentence: “Dividing the grants into two (two grant contracts with the same beneficiary signed on the same day) was purposely done to make the contract more efficient to manage. But the Mission has had to approve them as one Contract. The LIVCD would later divide the approved contract into two? Project achievements 6. In which value chain do you feel that achievements were most significant? 7. By value chain, to extent you feel that the project has achieved its objective in terms of: 8. Fully functional value chain? 9. Improved access to local and international markets? 10. Income and employment generation? 11. Is there any external factor that hindered the achievements of the project? 12. Which other factor facilitated the achievement of the project? 13. Regarding apple Frank said: “Investing in apples during the extension phase was based on the request made by the Mission”? Can you please explain it? 14. Who the project certified with GMP? Benefits? To what extent does the lack of this certification cause problems Project implementation 15. How was the TA vs Grants used as mechanisms in the implementation? Which of the 2 was easier to be managed? Which one was more conducive to the achievement of the project objective? Any differences noted from one VC to another? 16. In which value chain do you feel that the entry points (TA and grantees) were effectively chosen? 17. In which value chain do you feel that the choice of the entry points (TA and grantees) was more questionable? 18. How do you evaluate the economic impact assessment that were recently conducted? Were the time and the resources sufficient? Efficiency of the project 19. Please describe how the project uses consultants (international and local experts), grantees, TA, sub￾contractors, etc. 20. Which interventions do you feel are more cost-effective and which less? Why? 21. Which were the relatively more/less cost-effective interventions taking into consideration all direct costs on the project and the analysis presented in the impact assessment? (reference economic impact assessments). 22. Considering the overall budget of your award, and their line items, would you say that the budget was adequate for the achievement of the objectives? How? (If not adequate, what should have been different?) 23. To what extent you feel that the project would have been able to achieve more value for money than alternative approaches? M&E 24. Was the project M&E system mainly used for accountability purposes (towards USAID) or as a project management tool? Why? 25. Do you think that some key indicators were not included at the design stage? Which one? 26. Why were the impact assessment studies conducted? Coordination with government 27. How do you evaluate the coordination the project had with the government? Gender 28. To what extent you think the guidance you received on gender was sufficient for you to take into account gender aspects? 29. Do you know a document named “Resources Guide for Gender Integration in Value Chain Development in Lebanon? Did you use it? 30. Are you aware of USAID’s GE/FE policy and Mission Order and ADS 205 on gender? 31. How do you see men’s and women’s roles differently in this sector? 32. How do you see men’s and women’s roles differently In this specific project? a. In the managmenet/ decision making processes b. In the production/ processing c. In the marketing Sustainability 33. Do you know cases that the interventions promoted by the projects were replicated? 34. In which value chain do you feel that interventions are more sustainable? Why? 35. In which value chain do you feel that interventions are less sustainable? Why? 36. What sustainability plans has the project developed thus far, if any? Integration of People with Disabilities: 37. To which extent does this project involve people with disabilities? Conclusions and recommendations 38. In the hypothetical case that a project like this were to come again the future, , what would you like to see done differently? 39. Di you have any recommendations for the project that you would like us to take into account? 40. Any other aspect that you think it is important for us to know? Data needed 41. Costs by value chain 42. Can I have the contact of Jad Abur Arrange? LIVCD VALUE CHAIN MANAGERS Involvement in the project 1. Can you please tell me your main functions and responsibility? Project achievements 2. Which are the main achievements in your value chain? 3. For your value chain to extent you feel that the project has achieved its objective in terms of: 4. Development of the value chain? 5. Improved access to local and international markets? 6. Income and employment generation? 7. Is there any external factor that hindered the achievements of the project? 8. Which other factor facilitated the achievement of the project? 9. Regarding apple Frank said “Investing in apples during the extension phase was based on the request made by the Mission”. Can you please explain it? 10. Who the project certified with GMP? Benefits? To what extent does the lack of this certification cause problems (question only for processed foods) 11. How many service center have been set-up b the project? How many of them are operational (mainly for apple)? Project implementation 12. How was the TA vs Grants used as mechanisms in the implementation? Which of the 2 was easier to be managed? Which one was more conducive to the achievement of the project objective? Any differences noted from one VC to another? 13. To what extent do you feel that the entry points (TA and grantees) were effectively chosen? 14. To what extent do feel that the choice of the entry points (TA and grantees) was more questionable? 15. How do you evaluate the economic impact assessment that were recently conducted? Were the time and the resources sufficient? 16. To what extent you feel you had sufficient information from LIVCD top management to conduct your work? 17. How would you consider the coordination with field staff? Efficiency of the project 18. Which activities in your value chain you feel are more cost-effective and which less? Why? 19. Which were the relatively more/less cost-effective interventions taking into consideration all direct costs on the project and the analysis presented in the impact assessment? (reference economic impact assessments). 20. Did you have a specific budget for your value chain? If so, considering the budget for your value chain to what extent you feel that the budget was adequate for the achievement of the objectives? How? (If not adequate, what should have been different?) 21. To what extent you feel that the project would have achieved more value for money than alternative approaches? 22. Do you have any responsibility in procurement? Which one? 23. Do you function or responsibility for grant processing? 24. To what extent do you feel that the grant processing system was fast and smooth? If not why? 25. why was the grant to Ziad Tayeh Est withdrawn (apple)? Coordination with government 26. How do you evaluate the coordination the project had with the government? Gender 27. To what extent you think the guidance you received on gender was sufficient for you to take into account gender aspects? 28. Do you know a document named “Resources Guide for Gender Integration in Value Chain Development in Lebanon? To what extent did you use it? 29. Are you aware of USAID’s GE/FE policy and Mission Order and ADS 205 on gender? 30. How do you see men’s and women’s roles differently in this sector? 31. How do you see men’s and women’s roles differently In this specific project? a. In management/ decision making processes b. In the production/ processing c. In the marketing Sustainability 32. Do you know cases that the interventions promoted by the projects were replicated buy others? 33. Which interventions in your value chains are more sustainable? Why? 34. Which interventions in your value chain are less sustainable? Why? 35. What sustainability plans has the project developed thus far, if any? Conclusions and recommendations 36. In the hypothetical case that a project like this were to come again the future, , what would you like to see done differently? 37. Di you have any recommendations for the project that you would like us to take into account? 38. Any other aspect that you think it is important for us to know? COOPERATIVES LEADERS Introduction 1. Please, tell me about cooperative that you represent 2. Number of members 3. When it was established 4. Current activities and services offered to the cooperative members 5. Number of salaried staff Work with LIVCD 6. Please discuss your experiences with LIVCD 7. Which kind of assistance did you receive from LIVCD (promt: trainings, technical assistance, grant) 8. Can you please describe the process and time-line of the grant you received? Production 9. Can you please mention a few example of things that have changed after working with LIVCD? 10. Did you attend a training organized by LIVCD? If so, to what extent did you find it useful? 11. Can you provide a few example of introduced after the trainings? Market 1. Why do cooperative members produce xxxx? Do you produce mainly to sell it or consume it and give it to your relatives and friends? [This question is mainly relevant for olive oil producers and maybe cherry] 2. Who is the main buyers of product xxx for the cooperative members? The interviewer should note whether the respondent is referring to local traders, larger farms, exporters, supermarkets, cooperatives, etc. 3. To extent LIVCD helped the cooperative members to find a buyer or a market? 4. Does the cooperative organize group sale of produce xxxx? If so, when was the last time this happened? To what extent LIVCD helped you to organize group sales and to find a buyer? 5. To what extent LIVCD helped you to fetch a better price? [Prob. Did LIVCD suggested to charge the harvest period to fetch a better price? If you did it did you actually fetch a better price?] 6. Did you use cold storage or sorting equipment (mainly relevant for pome fruits and cherry)? 7. Did you rent it from some company/organization or did you use your own equipment? If you rented , my I ask you from whom you rented cold storage services and sorting equipment? 8. How much did you pay for renting cold storage and sorting equipment? As far as you know, was the rented cold storage facility and sorting equipment supported by USAID? Can you please, comment on you experience with the rented cold storage and sorting facility? Access to inputs 9. Does the cooperative organize group purchase of agricultural inputs? If so, when was the last time this happened? 10. To what extent did LIVCD helped you do it? 11. Where does the cooperative buy seedlings? Technical assistance and complementary services 12. If you need technical assistance on production aspect, who do approach? How far is he/she from the cooperative office? Finance 13. Did some cooperative members apply for any loan during the last few years? With which bank? 14. Did LIVCD provided you any assistance to apply for a loan? (prompt: development of feasibility studies, linkages with banks, trainings and workshops on loan products with bank officers)? 15. To what extent did you find the activities conducted by LIVCD to facilitate access to credit useful? Gender 16. How do you see men’s and women’s roles differently in this sector? 17. How do you see men’s and women’s roles differently In this specific project? a. In the management/ decision making processes b. In the production/ processing c. In the marketing Sustainability 18. Which equipment has the cooperative received from LIVCD? 19. Is the cooperative still using all of them? 20. Does the cooperative charge any fees to use the equipment received? How much? 21. Did you buy new equipment after receiving the equipment delivered by LIVCD? If so, which one? How much did you spend? 22. As far as you know, did some other cooperatives or farmers (not belonging to your cooperative) replicated the approach you had through LIVCD? 23. Does the cooperative have any specific plan for the future? Additionality 24. How much was the value of the grant hat you received? Which was your contribution? 25. Would you have made the same investments without USAID’ assistance? Why? Conclusions and recommendations 26. Did you have experiences with other donors or international organizations? If yes, can you please tell me main difference? Can you please tell me whether you think the approach used by the other donors you mentioned worked better or worse for you and why? 27. Which are the main challenges faced by the cooperative? 28. Do you have any recommendations for projects like this? 29. In the hypothetical case that a project like this were to come again in the future, what do you think should be done differently? Ministry of Tourism Work with LIVCD 1. Can you please tell me how the LIVCD project has worked with the Ministry of Tourism 2. Can I ask you your general opinion of LIVCD project? 3. Can you please comment on the development of the Rural Tourism Strategy? 4. To what extent did LIVCD support the development of the strategy? 5. To what extent would the Ministry have developed the Strategy without LIVCD? Policy and contextual changes 6. Did the government have any policy change that could affect the development of the tourist sector in general and more specifically of rural tourism in addition to the development of the Strategy? 7. Could you please tell if in Lebanon there has been any contextual change that may have affected the development of the tourist sector in general and more specifically of rural tourism? Synergies with other initiatives 8. Is there any other donors working in the area of rural tourism? 9. Do you envisage any synergies between what has been done by LIVCD and these projects? 10. Do you know cases where some of LIVCD approaches were replicated by others? Conclusions and recommendations 11. Which are the main challenges for the development of the rural tourism sector? 12. What should still be done to further develop the rural tourism sector? 13. How do you evaluate the capacity of guesthouse to lodge touriss? 14. Do you have any recommendations for future projects with USAID in the field of rural tourism? 44 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 ANNEX C: DOCUMENTS REVIEWED Documents referenced: IDAL Olive Oil Industry in Lebanon. 2017 Factsheet. [Online] Available from: http://investinlebanon.gov.lb/Content/uploads/SideBlock/171011013554317~Olive%20Oil%20Factsheet% 202017.pdf ILO estimate. [Online] Available from: http://databank.worldbank.org/data/home. Ministère de l’Agriculture e FAO. (2012). Résultats Globaux du Module de Base du Recensement de l’Agriculture 2010. Projet « Observatoire Libanais de Développement Agricole » de la Coopération Italienne. [On line]. Available from: http://www.agriculture.gov.lb/html/RESULTATS_RECENCEMENT_AGRICULTURE_2010/RAPPORT_R ESULTATS_GLOBAUX_DU_RECENCEMENT_2010.pdf Nuktrety N., & Al Jamal, S. (2016). Poverty Inequality and Social Protection in Lebanon. American University of Beirut and Oxfam. [Online] Available from: http://website.aub.edu.lb/ifi/publications/Documents/research_reports/20160426_poverty_inequality.pdf UNDP. (2016). Rapid Poverty Assessment for Lebanon in 2016. Press release. [Online] Available from: http://www.lb.undp.org/content/lebanon/en/home/Response_to_the_Syrian_Crisis/successstories/Rapid￾Poverty-Assessment-in-Lebanon-for-2016.html UNDP. (2008). Poverty, Growth, and Income Distribution in Lebanon. UNDP. [Online] Available from: http://www.undp.org/content/dam/lebanon/docs/Poverty/Publications/Poverty,%20Growth%20and%20Inc ome%20Distribution%20in%20Lebanon.pdf USAID. (2016). LIVCD Midterm Evaluation. [Online] Available from: https://dec.usaid.gov/dec/home/Default.aspx USAID. (2016). How-To Note: Evaluation Statement of Work. Program Cycle Guidance. [Online] Available from: https://usaidlearninglab.org/library/how-note-evaluation-statement-work LIVCD Project Documents: USAID. (2013). LIVCD Annual Progress Report: October 1, 2012 – September 30, 2013. USAID. (2014). LIVCD Annual Progress Report: October 1, 2013 – September 30, 2014. USAID. (2015). LIVCD Annual Progress Report: October 1, 2014 – September 30, 2015. USAID. (2016). LIVCD Annual Progress Report: October 1, 2015 – September 30, 2016. USAID. (2017). LIVCD Annual Progress Report: October 1, 2016 – September 30, 2017. USAID. (2017). LIVCD Quarterly Progress Report, Year 6 Quarter 1: October 1 – December 2017. USAID. (2018). LIVCD Quarterly Progress Report, Year 6 Quarter 2: January 1– March 31. USAID. (2018). LIVCD Quarterly Progress Report, Year 6 Quarter 3: April 1– June 30. USAID. Lebanon, Country Development Cooperation Strategy, 2014 – 2018. USAID. (2010). Request for Proposals No 268-11-004. USAID. (2013). Contract no AID-268-C-12-00001. Modification No 1. USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 45 NOVEMBER 2018 USAID. (2013). Contract no AID-268-C-12-00001. Modification No 2. USAID. (2013). Contract no AID-268-C-12-00001. Modification No 3. USAID. (2013). Contract no AID-268-C-12-00001. Modification No 4. USAID. (2014). Contract no AID-268-C-12-00001. Modification No 5. USAID. (2015). Contract no AID-268-C-12-00001. Modification No 6. USAID. (2016). Contract no AID-268-C-12-00001. Modification No 7. USAID. (2016). Contract no AID-268-C-12-00001. Modification No 8. USAID. (2016). Contract no AID-268-C-12-00001. Modification No 9. USAID. (2017). Contract no AID-268-C-12-00001. Modification No 10. USAID. (2017). Contract no AID-268-C-12-00001. Modification No 11. USAID. (2017). Contract no AID-268-C-12-00001. Modification No 12. USAID. (2018). Contract no AID-268-C-12-00001. Modification No 13. LIVCD. (2014). Olive Oil Value Chain Assessment Report. USAID/Lebanon. LIVCD. (2014). Pome Fruits Value Chain Assessment Report. USAID/Lebanon. LIVCD. (2014). Processed Foods Value Chain Assessment Report. USAID/Lebanon. LIVCD. (2014). Rural Tourism Value Chain Assessment Report. USAID/Lebanon. LIVCD. (2014). Table Grape Value Chain Assessment Report. USAID/Lebanon. LIVCD. (2014). Avocado Value Chain Assessment Report. USAID/Lebanon. LIVCD. (2014). Cherry Value Chain Assessment Report. USAID/Lebanon. LIVCD. (2014). Floriculture Value Chain Assessment Report. USAID/Lebanon. LIVCD. (2014). U.S. Market Recommendations Report. USAID/Lebanon. LIVCD. (2012). Six-Month Start-Up Plans. USAID/Lebanon. LIVCD. (2013). Years One and Two Work Plan. August 2013 – September 2014. USAID/Lebanon. LIVCD. (2014). Year Three Work Plan: October 1, 2014 – September 30, 2015. USAID/Lebanon. LIVCD. (2015). Year Four Work Plan: October 1, 2015 – September 30, 2016. USAID/Lebanon. LIVCD. (2015). Year Four Work Plan: October 1, 2015 – September 30, 2016. USAID/Lebanon. LIVCD. (2016). Year Five Work Plan: October 1, 2016 – September 30, 2017. USAID/Lebanon. 46 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 LIVCD. (2016). Year Six Work Plan: October 1, 2017– January 31, 2019. USAID/Lebanon. LIVCD. (2013). Grants Manual. USAID/Lebanon. LIVCD. (2015). Fresh Fruits Exports to UAE and KSA (presentation held in December 2015). LIVCD. (2017). Cherries Exports Opportunities to UAE, KSA & EU (presentation held on January 2017) LIVCD. (2016). Round Table Discussion - Working Paper on Improving the Competitiveness of Lebanese Apples’ Value Chain. LIVCD. (2014). Gender Assessment Report. USAID. (2016). PMSPLII Resource Guide for Gender Integration in Value Chain Development in Lebanon. USAID. (2015). Toolkit for Compliance with USAID Policy on Gender Equality: Guidance for USAID/Lebanon and Implementing Mechanisms. LIVCD. (2017). Monitoring and Evaluation Plan. USAID/Lebanon. LIVCD. (2018). Indicators Tracking Tables. LIVCD. (2018). List of Beneficiaries, August 2018. LIVCD. (2018). Grantees Tracking table, August 2018. LIVCD. (2018). List of Feasibility Studies for Loan Applications. LIVCD. (2018). Case Study, The Apple Service Centers. LIVCD. (2018). Case Study, Introducing New Grafting Technology. LIVCD. (2018). Case Study, A Holistic Upgrade of the Grapes Value Chain in Rashaya Al Foukhar. LIVCD. (2018). Case Study, Revitalizing the Honey Value Chain in Lebanon. LIVCD. (2018). Case Study, The Mechanical Harvesting Program Olive Value Chain in Lebanon. LIVCD. (2018). Economic Impact Assessment, Apples Value Chain. LIVCD. (2018). Economic Impact Assessment, Avocado Value Chain. LIVCD. (2018). Economic Impact Assessment, Cherries Value Chain. LIVCD. (2018). Economic Impact Assessment Grapes Value Chain. LIVCD. (2018). Economic Impact Assessment Honey Value Chain. LIVCD. (2018). Economic Impact Assessment, Rural Tourism Value Chain. LIVCD. (2013). Annual Progress Report: October 1, 2012 – September 30, 2013. LIVCD. (2014). Annual Progress Report: October 1, 2013 – September 30, 2014. LIVCD. (2015). Annual Progress Report: October 1, 2014 – September 30, 2015. LIVCD. (2016). Annual Progress Report: October 1, 2015 – September 30, 2016. LIVCD. (2017). Annual Progress Report: October 1, 2016 – September 30, 2017. USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 47 NOVEMBER 2018 LIVCD. (2017). Quarterly Progress Report, Year 6 Quarter 1: October 1 – December 2017. LIVCD. (2018). Quarterly Progress Report, Year 6 Quarter 2: January 1– March 31. LIVCD. (2018). Quarterly Progress Report, Year 6 Quarter 3: April 1– June 30. LIVCD. (2018). Final Evaluation Survey: Presentation of Results and Technical Interpretation. 48 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 ANNEX D: EVALUATION RESPONDENTS Key Informants (Redacted) FGD Respondents Note: FGD Respondent names have been withheld. Pome fruit farmers FG1- Metn, Mount Lebanon Participant Number Age Gender Value Chain Location 1 78 Male Apple Baskinta 2 50 Male Apple Baskinta 3 64 Male Apple Baskinta 4 60 Male Apple Baskinta 5 40 Male Apple Baskinta 6 20 Male Apple Baskinta 7 44 Male Apple Baskinta FG2- Bcharre, North Lebanon Participant Number Age Gender Value Chain Location 1 70 Male Apple Hadath El Jebbeh 2 57 Male Apple Hadath El Jebbeh 3 44 Male Apple Hadath El Jebbeh 4 57 Male Apple Hadath El Jebbeh 5 55 Male Apple Hadath El Jebbeh 6 21 Female Apple Hadath El Jebbeh 7 23 Female Apple Hadath El Jebbeh 8 45 Male Apple Hadath El Jebbeh FG3- Baalbek, Bekaa Participant Number Age Gender Value Chain Location 1 26 Male Apples Ainata 2 52 Male Apples Ainata 3 79 Male Apples Ainata 4 65 Male Apples Ainata 5 65 Female Apples Ainata 6 35 Female Apples Ainata 7 22 Female Apples Ainata 8 73 Male Apples Ainata 9 59 Male Apples Ainata 10 68 Male Apples Ainata FG4- Chouf, Mount Lebanon Participant Number Age Gender Value Chain Location 1 43 Male Apples Breeh 2 42 Male Apples Breeh 50 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 3 54 Male Apples Breeh 4 54 Male Apples Breeh 5 80 Male Apples Breeh 6 71 Male Apples Breeh 7 53 Male Apples Breeh 8 54 Male Apples Breeh FG5- Jbeil, Mount Lebanon Participant Number Age Gender Value Chain Location 1 57 Male Apple Aaqoura 2 55 Male Apple Aaqoura 3 48 Female Apple Aaqoura 4 49 Male Apple Aaqoura 5 53 Male Apple Aaqoura 6 74 Male Apple Aaqoura Avocado Value Chain FG1- Chouf, Mount Lebanon Participant Number Age Gender Value Chain Location 1 71 Male Avocado Bater 2 64 Female Avocado Bater 3 45 Male Avocado Bater 4 59 Male Avocado Bater 5 43 Male Avocado Bater 6 49 Male Avocado Bater 7 56 Male Avocado Bater 8 56 Male Avocado Bater 9 61 Male Avocado Bater FG2- Jezzine, South Lebanon Participant Number Age Gender Value Chain Location 1 55 Male Avocado Bisre 2 57 Male Avocado Bisre 3 58 Male Avocado Bisre 4 65 Male Avocado Bisre 5 49 Male Avocado Bisre FG3- Akkar, North Lebanon Participant Number Age Gender Value Chain Location 1 56 Male Avocado Bebnine 2 55 Male Avocado Bebnine 3 57 Male Avocado Bebnine 4 63 Male Avocado El Aabde 5 48 Male Avocado Qobet Chamra FG4- Saida, South Lebanon USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 51 NOVEMBER 2018 Participant Number Age Gender Value Chain Location 1 40 Male Avocado Saida 2 60 Male Avocado El Najjariye 3 50 Male Avocado Saida 4 57 Male Avocado Aaqbiye 5 62 Male Avocado El Ghazieh Cherry value Chain FG1- Baalbek, Bekaa Participant Number Age Gender Value Chain Location 1 34 Male Cherry Arsal 2 40 Male Cherry Arsal 3 57 Male Cherry Arsal 4 51 Male Cherry Arsal 5 44 Male Cherry Arsal 6 67 Male Cherry Arsal 7 40 Male Cherry Arsal 8 73 Male Cherry Arsal 9 53 Male Cherry Arsal FG2- Bcharre, North Lebanon Participant Number Age Gender Value Chain Location 1 53 Male Cherry Ed Dimane 2 61 Male Cherry Bqorqacha 3 43 Male Cherry Bqaa Kafra 4 38 Male Cherry Hasroun 5 48 Male Cherry Bcharre FG3- Zahle, Bekaa Participant Number Age Gender Value Chain Location 1 64 Male Cherry Qaa Er Rim 2 67 Male Cherry Qaa Er Rim 3 70 Male Cherry Qaa Er Rim 4 31 Male Cherry Qaa Er Rim 5 62 Female Cherry Qaa Er Rim 6 56 Male Cherry Qaa Er Rim 7 45 Male Cherry Qaa Er Rim 8 56 Male Cherry Qaa Er Rim 9 65 Male Cherry Qaa Er Rim 10 72 Male Cherry Qaa Er Rim FG4- Baabda, Mount Lebanon Participant Number Age Gender Value Chain Location 1 64 Female Cherry Hammana 2 37 Female Cherry Jouar El Haouz 52 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 3 56 Male Cherry Hammana 4 52 Male Cherry Jouar El Haouz 5 50 Male Cherry Jouar El Haouz 6 67 Female Cherry Jouar El Haouz 7 54 Male Cherry Qornayel 8 66 Male Cherry Jouar El Haouz 9 43 Male Cherry Qornayel Grapes Value Chain FG1- Zahle, Bekaa Participant Number Age Gender Value Chain Location 1 72 Male Grapes Haouch Hala 2 68 Male Grapes Ferzol 3 57 Male Grapes Oussaya 4 71 Male Grapes Tamnin 5 56 Male Grapes Ferzol 6 46 Male Grapes Ferzol 7 64 Male Grapes Ferzol 8 60 Male Grapes Ferzol 9 64 Female Grapes Ferzol 10 61 Male Grapes Ferzol FG2- Baalbek, Bekaa Participant Number Age Gender Value Chain Location 1 54 Male Grapes Yammoune 2 70 Male Grapes Deir Al Ahmar 3 59 Male Grapes Shmistar 4 53 Male Grapes Bechwat 5 50 Male Grapes Bechwat 6 70 Male Grapes Bechwat 7 60 Male Grapes Chaat FG3- Hassbaya, El Nabatiyeh Participant Number Age Gender Value Chain Location 1 61 Male Grapes Rachaiya Al Fokhar 2 26 Male Grapes Halta 3 35 Male Grapes Halta 4 27 Male Grapes Kfar Chouba 5 25 Male Grapes Halta 6 58 Male Grapes Halta 7 20 Male Grapes Rachaiya Al Fokhar 8 36 Male Grapes Halta 9 38 Male Grapes Halta FG4- Zahle, Bekaa Participant Number Age Gender Value Chain Location 1 73 Male Grapes El Fourzol USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 53 NOVEMBER 2018 2 67 Male Grapes Majdaloun 3 61 Male Grapes Terbol 4 55 Male Grapes El Fourzol 5 66 Male Grapes Zahle Aradi FG5- Zahle, Bekaa Participant Number Age Gender Value Chain Location 1 53 Male Grape El Fourzol 2 74 Male Grape El Fourzol 3 50 Male Grape El Fourzol 4 53 Male Grape El Fourzol 5 50 Male Grape El Fourzol 6 77 Male Grape El Fourzol 7 50 Male Grape El Fourzol Beekeepers FG1- Jbeil, Mount Lebanon Participant Number Age Gender Value Chain Location 1 72 Male Honey Jbeil 2 40 Male Honey Saqi Rechmaya 3 47 Male Honey El Ghine 4 54 Male Honey Mechmech 5 60 Male Honey Bejje 6 47 Male Honey Aamchit FG2- Baalbek, Bekaa Participant Number Age Gender Value Chain Location 1 53 Female Honey Zighrine 2 44 Female Honey Zighrine 3 64 Male Honey Zighrine 4 42 Male Honey Zighrine 5 34 Female Honey Zighrine 6 26 Female Honey Zighrine 7 25 Female Honey Zighrine 8 40 Female Honey Zighrine FG3- Jezzine, South Lebanon Participant Number Age Gender Value Chain Location 1 NA Female Honey Jezzine 2 50 Male Honey Jezzine 3 57 Male Honey Jezzine 4 56 Female Honey Jezzine 5 36 Male Honey Jezzine 6 54 Male Honey Bkassine 7 73 Male Honey Sfaray 8 62 Male Honey Bkassine 54 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 FG4- Minnieh-Dinnieh, North Lebanon Participant Number Age Gender Value Chain Location 1 54 Male Honey Bet El Faqs 2 66 Male Honey Bet El Faqs 3 56 Male Honey Bet El Faqs 4 29 Male Honey Bet El Faqs 5 54 Male Honey Bet El Faqs 6 29 Male Honey Bet El Faqs 7 28 Male Honey Bet El Faqs 8 31 Male Honey Bet El Faqs 9 32 Male Honey Bet El Faqs 10 30 Male Honey Bet El Faqs 11 54 Male Honey Bet El Faqs 12 54 Male Honey Bet El Faqs FG5- Baabda, Mount Lebanon Participant Number Age Gender Value Chain Location 1 53 Male Honey Qornayel 2 64 Male Honey Abadieh 3 36 Male Honey Abadieh 4 40 Male Honey Qornayel 5 46 Female Honey Qornayel 6 64 Male Honey Qornayel 7 60 Male Honey Qornayel 8 55 Male Honey Sawfar 9 58 Male Honey Kousseibeh 10 43 Male Honey Salima Olive farmers FG1- Tripoli, North Lebanon Participant Number Age Gender Value Chain Location 1 54 Male Olive Oil Tripoli 2 58 Male Olive Oil Tripoli 3 23 Male Olive Oil Tripoli 4 36 Male Olive Oil Tripoli 5 48 Female Olive Oil Tripoli 6 43 Male Olive Oil Tripoli 7 20 Male Olive Oil Tripoli 8 52 Male Olive Oil Tripoli 9 46 Male Olive Oil Tripoli 10 62 Male Olive Oil Tripoli FG2- Akkar, North Lebanon Participant Number Age Gender Value Chain Location 1 63 Male Olive Oil Chadra 2 83 Male Olive Oil Qobayyat USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 55 NOVEMBER 2018 3 NA Male Olive Oil Chadra 4 75 Male Olive Oil Qobayyat 5 NA Male Olive Oil Aydamoun 6 NA Male Olive Oil Qobayyat 7 60 Male Olive Oil Qobayyat 8 NA Male Olive Oil Qobayyat 9 66 Male Olive Oil Qobayyat FG3- Hassbaya, El Nabatiyeh Participant Number Age Gender Value Chain Location 1 55 Male Olive Oil Hassbaya 2 50 Male Olive Oil Hassbaya 3 39 Male Olive Oil Mimess 4 56 Male Olive Oil Hassbaya 5 50 Male Olive Oil Mimess 6 50 Male Olive Oil Hassbaya 7 53 Male Olive Oil Hassbaya FG4- Jezzine, South Lebanon Participant Number Age Gender Value Chain Location 1 68 Male Olive Oil Lebaa 2 77 Male Olive Oil Lebaa 3 67 Female Olive Oil Lebaa 4 64 Female Olive Oil Lebaa 5 60 Male Olive Oil Lebaa 6 50 Male Olive Oil Lebaa 7 50 Male Olive Oil Lebaa 8 52 Male Olive Oil Lebaa 9 85 Male Olive Oil Lebaa FG5- Baalbek-Hermel, Bekaa Participant Number Age Gender Value Chain Location 1 51 Male Olive Oil Hermel 2 48 Male Olive Oil Hermel 3 39 Male Olive Oil Hermel 4 59 Male Olive Oil Hermel 5 67 Male Olive Oil Hermel 6 67 Male Olive Oil Hermel 7 59 Male Olive Oil Hermel 8 68 Male Olive Oil Hermel ANNEX F: EVALUATION DESIGN MATRIX Evaluation criteria Evaluation questions related to the criteria Data sources Data collection methods, sample and tools Data analysis plan Relevance To what extent were the value chain approach and LIVCD’s objectives and performance indicators appropriate for meeting the CDCS development objective? -RFP -LIVCD contracted SoW -LIVCD Project modification -LIVCD Project Documents (Workplans, Annual Reported, Quarterly Reports, M&E plans, M&E reports) - LIVCD Initial Assessments -LIVCD Midterm Evaluation -KII with USAID staff, stakeholders and technical experts Secondary Sources - Desk review of project documents Primary Sources - Key Informant Interviews (KIIs) with LIVCD staff, technical experts, and USAID stakeholders - Analysis of CDCS Development Objectives and indicators with LIVCD’s expected and actual results - Content analysis of literature review Efficiency and cost￾effectiveness To what extent were the selected interventions cost-effective? - LIVCD impact assessments - LIVCD 2018 farmer survey - LIVCD performance tracking tables - List of grants (with value) - List of technical assistance and sub￾contracts (with value) - LIVCD LOE - LIVCD Mid-Term Evaluation - LIVCD Project modification - LIVCD Project Documents (Workplans, Annual Reports, Quarterly Reports, M&E plans, M&E reports) Secondary Sources - Desk review of project documents - Midterm evaluation - LIVCD Internal Assessment of cost￾effectiveness (by Value Chain) Primary Sources - Key Informant Interviews (KIIs) with LIVCD and USAID staff - Comparison of benefit-to-cost ratios (BCR) of interventions in different value chains - Analysis of the awarded amount of grants and cost-share - Qualitative analysis of KIIs Effectiveness To what extent have LIVCD’s interventions resulted in meeting the objectives and key performance indicators - LIVCD Midterm Evaluation - KII with USAID and LIVCD staff - KIIs and FGDs with LIVCD beneficiaries - LIVCD Project Documents (Workplans, Annual Reported, Quarterly Reports, M&E plans, M&E reports) Primary Sources - Key Informant Interviews (KIIs) with USAID staff - FGDs Secondary Sources - Desk review of project documents - Comparison of achieved results versus planned results USAID/LEBANON LIVCD FINAL EVALUATION REPORT | 57 NOVEMBER 2018 (KPIs) of the initial and extended contract? - Midterm evaluation - Content analysis of FGDs and KIIs with regards to credit market, input market, TA market, and output market. Were the selected interventions and entry points in each value chain strategic in leading to the intended results? - LIVCD Midterm Evaluation - KIIs with USAID and LIVCD and stakeholders -Project documents: grants and TA databases - LIVCD Assessment of Economic Impact Primary Sources - Key Informant Interviews (KIIs) with USAID staff, and sectoral experts Secondary Sources - Desk review of project documents - Midterm evaluation Content analysis of primary and secondary sources What internal and external factors contributed to the achievement (or nonachievement) of the above results? - LIVCD Midterm Evaluation - KIIs with USAID and LIVCD and stakeholders - Project documents: grants and TA databases - KIIs and FGDs with LIVCD grantees and beneficiaries Primary Sources - Key Informant Interviews (KIIs) with USAID staff, grantees, and sectoral experts - FGDs Secondary Sources - Desk review of project documents - Midterm evaluation Content analysis of primary and secondary sources Sustainability and additionality To what extent can LIVCD’s interventions be considered sustainable? - LIVCD Midterm Evaluation - KIIs with USAID and LIVCD and stakeholders Primary Sources Qualitative analysis 58 | LIVCD FINAL EVALUATION REPORT USAID/LEBANON NOVEMBER 2018 Were the positive results spread across all beneficiaries or only a few? And to what extent did the interventions demonstrate additionality? - LIVCD Midterm Evaluation - KIIs with USAID and LIVCD staff - KIIs and FGD with LIVCD grantees and beneficiaries - LIVCD Project Documents (Workplans, Annual Reported, Quarterly Reports, M&E plans, M&E reports) - 2018 LIVCD farmer survey Primary Sources - Key Informant Interviews (KIIs) with USAID staff and grantees Secondary Sources - Desk review of project documents - Mid-term evaluation - Analysis of the budget of individual grants - Qualitative analysis of FGDs and KIIs U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523