This publication was produced for review by the United States Agency for International Development. It was prepared by EnCompass LLC for the Monitoring and Evaluation Program for Lebanon (MEPL) under Contract No. 7200AA20D00011, Task Order 72026821F00005. Note: Limited redactions have been made to this version of the report in accordance with the principled exceptions to the presumption in favor of openness established in OMB Bulletin 12-01, “Guidance on Collection of U.S. Foreign Assistance Data.” FINAL PERFORMANCE EVALUATION LEBANON ENTERPRISE DEVELOPMENT (LED) CHEMONICS INTERNATIONAL FINAL REPORT SEPTEMBER 2022 FINAL PERFORMANCE EVALUATION LEBANON ENTERPRISE DEVELOPMENT (LED) CHEMONICS INTERNATIONAL FINAL REPORT SEPTEMBER 2022 Prepared by EnCompass LLC for the USAID Monitoring and Evaluation Program for Lebanon (MEPL) under Prime Contract No. 7200AA20D00011, Task Order 72026821F00005. Significant contributions were provided by subcontracting partner Management Systems International (MSI), a Tetra Tech Company. This document is not available in print. Documents of this nature are made available to the public through the Development Experience Clearinghouse (DEC) repository (dec.usaid.gov). For additional information, please contact: EnCompass LL Corporate Office 1451 Rockville Pike, Suite 600 Rockville, Maryland 20852 USA Or EnCompass LLC Lebanon Office Arz Street Librex Bldg. Bloc 2 – 3rd Floor Zalka, Metn, Lebanon DISCLAIMER: This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents of this report are the responsibility of EnCompass LLC and its implementing partners and do not necessarily reflect the views of USAID or the United States Government. COVER PHOTO: A Lebanese catering business received technical assistance from LED related to food safety and hygiene. Pictured is one of their cooks holding a homemade casserole. (Photo courtesy of beneficiary) ABSTRACT Lebanon Enterprise Development (LED) was a five-year, $24-million United States Agency for International Development (USAID) Activity aimed at increasing employment opportunities in Lebanon by adopting a Buyer-Led Approach (BLA) with local firms to enable them to increase their sales and create jobs. This evaluation assesses the extent to which LED was able to meet its intended objectives and ensure the sustainability of its interventions. The report’s findings, conclusions, and recommendations can help USAID’s Mission to Lebanon to plan follow-on activities. The evaluation team used a mixed-methods approach to answer a set of evaluation questions, conducting 33 key informant interviews with key project stakeholders and several focus group discussions and individual meetings with 36 firms that received assistance from LED. The team supplemented the qualitative data collection with a document/desk review and an extensive examination of available quantitative data. The evaluation found that LED had adopted a more flexible and inclusive form of BLA, the market￾led approach, to be more responsive to the assisted firms’ evolving needs and market demands (with respect to export and import substitution) during a highly challenging time for Lebanese businesses. There was an overall positive response by the firms to LED’s technical assistance (TA), especially after LED adapted its approach following Lebanon’s crises. However, many firms also expressed a strong need for complementary funding and additional support to maximize the impact of the TA. Whilst over half of the assisted firms created jobs, the total number of jobs created was much lower than LED’s targets. The biggest constraint to LED achieving its key performance indicator (KPI) targets was the country’s context and economic crisis. The LED approach overall was not as cost-effective as originally projected since LED did not achieve its outcome targets for Objectives 1 and 2. At the TA level, the most cost-effective type of TA was Human Resources & Restructuring, Production & Process Management, Quality Management, and Knowledge Management. The most cost-effective sectors were oil and gas and tourism/food services. The most cost-effective interventions were in the Beqaa and Baalbek-Hermel governorates. There was overwhelmingly positive feedback from all stakeholders regarding the role and effect of LED/USAID’s assistance to firms. The biggest enabler for the sustainability of LED’s impact on businesses is the change in business owners’ view of external assistance, combined with the increased capacity of local market actors and local business consultants as a result of LED’s interventions. This evaluation’s main recommendations for future USAID enterprise-development projects in Lebanon include: (1) Re-assessing the relevance and practicality of the job creation focus, indicators, and targets, given the major contextual changes and challenges; (2) Conducting “value for money” assessments at the project level for the timely identification and implementation of improvements and adaptations to project design, objectives, and implementation processes; (3) Exploring additional types of support that could have a higher impact and value for money compared to offering purely firm-level TA; and (4) Taking into account the lessons learned by LED and its stakeholders in order to increase the efficiency and efficacy of implementation processes (such as for firm and business consultant selection and TA delivery). ACKNOWLEDGMENTS The evaluation team consisted of Hoda Salman (Team Leader) and Maya Trad (Team Member) and received support from Joanna Khater and Nour El Zaouk of EnCompass LLC. The evaluation team would like to thank MEPL’s staff; USAID’s Contracting Officer’s Representative for LED, Georges Frenn*; LED’s senior management, Douglas Griffith, Ibrahim Basha, Borhan Kreitem, Fadi Naffah, and Farah Maalouf; LED’s staff, including those at Berytech and BIAT; LED’s business advisors and business consultants; the LED-assisted firms that agreed to participate in this study; and all those who contributed to this work. * Mr. Frenn passed away in October 2022. May his soul rest in peace. i | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 TABLE OF CONTENTS FIGURES AND TABLES ................................................................................................................ II ACRONYMS ................................................................................................................................ III EXECUTIVE SUMMARY............................................................................................................... IV EVALUATION PURPOSE ...................................................................................................................................... IV EVALUATION QUESTIONS ................................................................................................................................. IV ACTIVITY BACKGROUND ................................................................................................................................... V EVALUATION DESIGN, METHODS, AND LIMITATIONS ................................................................................... V FINDINGS AND CONCLUSIONS ......................................................................................................................... V EVALUATION PURPOSE AND EVALUATION QUESTIONS ........................................................... 1 EVALUATION PURPOSE ....................................................................................................................................... 1 EVALUATION QUESTIONS (EQS) ...................................................................................................................... 1 ABOUT LED ............................................................................................................................... 2 ACTIVITY BACKGROUND ................................................................................................................................... 2 ACTIVITY OBJECTIVES ......................................................................................................................................... 2 ACTIVITY APPROACH ......................................................................................................................................... 3 DEVELOPMENT HYPOTHESIS .............................................................................................................................. 3 CRITICAL ASSUMPTIONS .................................................................................................................................... 3 EFFECTS OF LEBANON’S ECONOMIC CONTEXT ............................................................................................ 4 OVERVIEW OF LED’S ACTIVITIES AND OUTPUTS ........................................................................................... 5 EVALUATION METHODS AND LIMITATIONS .............................................................................. 7 EVALUATION FINDINGS AND CONCLUSIONS ............................................................................ 8 EVALUATION QUESTION 1 (RELEVANCE) ........................................................................................................ 8 EVALUATION QUESTION 2 (RELEVANCE) ...................................................................................................... 12 EVALUATION QUESTION 3 (EFFECTIVENESS) ................................................................................................ 13 EVALUATION QUESTION 4 (EFFECTIVENESS) ................................................................................................ 31 EVALUATION QUESTION 5 (SUSTAINABILITY, SCALABILITY, ADDITIONALITY)........................................ 36 RECOMMENDATIONS ................................................................................................................ 44 JOB CREATION TARGET AND MEASUREMENT .............................................................................................. 44 FIRM-LEVEL APPROACH .................................................................................................................................... 44 PROJECT IMPLEMENTATION AND PROCESSES ................................................................................................ 46 GENERAL ............................................................................................................................................................ 47 ANNEXES.................................................................................................................................. 48 ANNEX I: EVALUATION SCOPE OF WORK .................................................................................................... 49 ANNEX II: LED RESULTS FRAMEWORK AND INDICATORS ......................................................................... 50 ANNEX III: EVALUATION METHODS AND LIMITATIONS .............................................................................. 51 ANNEX IV: DATA COLLECTION AND ANALYSIS TOOLS ............................................................................ 55 ii | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 ANNEX V: SOURCES OF INFORMATION ......................................................................................................... 56 ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST....................................................................... 57 ANNEX VII: CHARACTERISTICS OF LED DEALS / ASSISTED FIRMS.............................................................. 58 FIGURES AND TABLES Figure 1: Number of deals closed by LED over its five years of operation. ................................................. 9 Figure 2: Percentage of Closed Deals, by Industry (n=521) ........................................................................... 58 Figure 3: Percentage of Closed Deals, by Location (n=521) .......................................................................... 58 Figure 4: Percentage of Closed Deals, by Firm Size (n=521) ......................................................................... 59 Figure 5: Percentage of Closed Deals, by Type of Technical Assistance (n=521) .................................... 59 - Table 1: Status of LED's key performance indicators ...................................................................................... 13 Table 2: LED’s revised KPI targets (April 2022) ............................................................................................... 14 Table 3: Number of firms that hired new staff, increased their sales, or leveraged investment thanks to LED’s TA ............................................................................................................................................................... 15 Table 4: LED KPI results - actual vs. committed to by firms ......................................................................... 25 Table 5: Number of firms achieving their committed targets for the three main LED outcome indicators .................................................................................................................................................................... 26 Table 6: Incremental Sales and Leveraged Investment KPI actual results vs. targets (in terms of USD and implied multiple). ............................................................................................................................................... 28 Table 7: Average total technical assistance value .............................................................................................. 29 Table 8: Percent contribution of each type of TA, by KPI. ............................................................................ 32 Table 9: Cost-effectiveness ratios, by KPI and type of TA ............................................................................. 33 Table 10: Cost-effectiveness ratios, by size of assisted firms......................................................................... 34 Table 11: Cost-effectiveness ratios, by sector of assisted firms .................................................................... 35 Table 12: Cost-effectiveness ratios, by location (governorate) of assisted firms ..................................... 35 Table 13: Percentage change in average size/value of TA, by type of TA ................................................... 40 Table 14: Breakdown of interviewees by type of stakeholder ...................................................................... 53 iii | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 ACRONYMS AFID Agri-Food Innovation Days AI Appreciative Inquiry B2B Business to Business BEE Business Enabling Environment BLA Buyer-Led Approach CA Cooperative Agreement COP Chief of Party COR Contracting Officer’s Representative DCOP Deputy Chief of Party EBRD European Bank for Reconstruction and Development FGD Focus Group Discussion FTE Full-Time Equivalent FSSC Food Safety System Certification FY Fiscal Year (USAID Oct 1 to Sept 30) IP Implementing Partner ISO International Organization for Standardization IT Information Technology KII Key Informant Interview KPI Key Performance Indicator LED Lebanon Enterprise Development LBP Lebanese Pound (Lira) LOE Level of Effort LOP Life of Project M&E Monitoring and Evaluation MEL Monitoring, Evaluation, and Learning MLA Market-Led Approach MOU Memorandum of Understanding MSEs Micro and Small Enterprises MSMEs Micro, Small, and Medium-Sized Enterprises NGO Non-Governmental Organization PoP Period of Performance MEPL Monitoring and Evaluation Program for Lebanon PSD Private Sector Development RFQ Request for Quotation SOW Scope of Work SMEs Small and Medium-Sized Enterprise TA Technical Assistance USAID United States Agency for International Development USEK Holy Spirit University of Kaslik USG United States Government iv | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 EXECUTIVE SUMMARY EVALUATION PURPOSE With the Lebanon Enterprise Development (LED) Activity approaching its end, the United States Agency for International Development (USAID)’s Mission to Lebanon tasked the Monitoring and Evaluation Program for Lebanon (MEPL), implemented by EnCompass LLC and Management Systems International (MSI), with conducting a Final Performance Evaluation of the Activity to assess the extent to which it was able to meet its intended objectives. The evaluation provides USAID/Lebanon with information on the approaches followed and activities implemented under LED to achieve the Activity’s outputs and outcomes. It also evaluates the sustainability of LED’s interventions. The report’s findings, conclusions, and recommendations will help the Mission plan follow-on activities. The main audience of this Final Performance Evaluation is USAID/Lebanon, specifically the Program Office and Economic Growth Office, in addition to LED’s implementer Chemonics International and other parties and stakeholders at the discretion of the Mission. EVALUATION QUESTIONS ACTIVITY RELEVANCE 1. How relevant was the LED approach of firm-level, buyer-led, and generally market-led technical assistance through business consultants, in terms of meeting the evolving needs of the private sector in Lebanon, particularly MSMEs? 2. To what extent was the LED approach appropriate to help stabilize enterprises and jobs, support niches of growth through export and import substitution, and prepare the ground for future recovery? ACTIVITY EFFECTIVENESS 3. What are the primary successes of the LED Activity? How and to what extent has the Activity achieved its planned results? What went well, and what contributed to this success? 3.a. How and to what extent was LED able to adapt its approach to major context changes in the country (financial and economic crisis, Covid-19)? 3.b. How was the success of LED influenced by contextual factors? What were the supporting factors? What challenges did the Activity face? 3.c. To what extent has the firm-level, buyer-led/market-led approach been cost￾effective in meeting LED's objectives and KPI targets? 4. Which LED interventions and technical support have been the most and/or least beneficial, productive, and cost-effective? ACTIVITY SUSTAINABILITY, SCALABILITY, AND ADDITIONALITY 5. To what extent are the successes of LED likely to be sustainable and scalable and demonstrate additionality? 5.a. Which LED interventions and results show the most promise of being sustained, and why? 5.b. Which particular aspects of the LED Activity have resulted in the growing replication of technical assistance at declining marginal costs? Has there been any particular aspect of the Activity that ended up being more "one-off," i.e., not scalable? v | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 5.c. How likely is it that beneficiary firms would have been able to effectively research, procure, purchase, monitor, and achieve similar results had USAID's assistance not been available? ACTIVITY BACKGROUND USAID/Lebanon signed the LED contract with Chemonics in October 2017 (No. AID-268-C-17- 00001) for an initial value of $14 million until September 30, 2020. LED later received a two-year, $10.3 million contract extension through September 30, 2022. LED was designed as an Activity under USAID/Lebanon's Private Sector Development (PSD) Project. Its main purpose is to provide business development services and solutions to Lebanese firms to enable them to expand their activities, increase their sales, and create jobs. LED’s primary approach was to work with private-sector firms to identify and overcome specific constraints to accessing new markets. It thereby facilitated new relationships with buyers in existing and new markets (a “market-led” approach (MLA) that builds on the “buyer-led” approach (BLA) to enterprise development). LED prioritized support, in the form of direct technical assistance (TA) through external business consultants, for enterprises that produce goods and services for import substitution in the domestic market and for exports to international markets. This support, in turn, aimed to assist the firms to become more competitive and hiring additional staff – with new jobs serving as a primary measure of success across the LED Activity. LED also worked to identify problems affecting the business enabling environment (BEE) in Lebanon and collaborated with private-sector actors (through its grants facility) to analyze and propose solutions to them. EVALUATION DESIGN, METHODS, AND LIMITATIONS The evaluation team used a mixed-methods approach to answer the evaluation questions, conducting key informant interviews (KIIs) with key project stakeholders and focus group discussions (FGDs) with several businesses that received LED support. The evaluation team supplemented the qualitative data collection with a document/desk review and an extensive examination of the available quantitative data. The team also made sure to take the necessary measures to mitigate any limitations related to recall, response, and selections biases. FINDINGS AND CONCLUSIONS RELEVANCE (EQ 1 & 2)  LED adopted a market-led approach (MLA) to enterprise development. It is a more inclusive and flexible version of the traditional buyer-led approach (BLA) and is better suited for Lebanon’s context and the evolving needs of Lebanese firms as they navigate high uncertainty in the market and face many challenges that put their survival and growth at risk. LED complemented this approach by working with market actors to increase job creation and firm-level competitiveness. However, LED’s objectives did not specifically or intentionally target relationships between local service providers (such as business consultants) and firms (as local producers), which a more comprehensive BLA or MLA would have been able to do in order to meet the increasing market demand.  The tailored TA delivered through LED was driven by business and market demand and aimed to satisfy specific needs that were rapidly evolving amidst Lebanon’s crises. Most enterprises were satisfied with the TA provided and confirmed that the TA approach adopted by LED had been relevant to them for identifying successful solutions to some of their business problems, particularly with respect to expanding their access to foreign markets and/or improving their productivity. However, more TA per firm could have produced higher impact for the firms and therefore a higher attribution to LED.  The findings revealed some constraints related to the ambitious job-creation target expected from the TA in the context of Lebanon. LED’s job creation goal, therefore, does vi | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 not fully reflect the challenging context of the country and local industries, and therefore it cannot be achieved solely through firm-level TA interventions.  A missing essential component of the approach is complementary funding, particularly in the form of cash grants, which most firms viewed as very important, given the economic and financial challenges they faced. Cash support would have been very helpful, especially for small businesses to help them fully implement the TA they received and therefore maximize its impact.  LED’s interventions were very appropriate to help stabilize enterprises and maintain existing jobs therein, with most interviewed firms attributing their survival to LED’s support, which they say came at the right time. The majority even created new jobs thanks to LED’s assistance. LED’s grants to market actors helped enhance job creation by supporting firms and creating employment and entrepreneurship opportunities for individuals.  LED’s support for export and import substitution was more focused after the onset of the crises and was critically timed for businesses that needed to find sources of fresh United States dollars (USD) due to financial constraints and/or control or reduce their increasing production costs due to hyperinflation and more expensive imports of equipment and raw materials. EFFECTIVENESS (EQ 3 & 4)  LED was quick to adapt its approach following the onset of Covid-19 and the economic crisis, but it do so in a limited fashion, as it had to remain within the confines of its original mandate and objectives. These changes were important to incentivize more firms to apply and benefit from LED’s TA. The majority of these firms were able to create new jobs as a result, but the numbers were much lower than LED’s targets.  Undoubtedly, the major challenge to LED’s achievement of its key performance indicator (KPI) targets was the deterioration of Lebanon’s socio-economic situation that started in late 2019. The financial, economic, and COVID-19 crises created more acute problems for businesses, and many had to close shop temporarily or permanently. However, the crises also pushed business owners to look at their businesses through a different lens. As such, many started looking for new opportunities and considering getting external support to help them overcome the new (and existing) challenges.  A major factor that contributed to LED’s success was the commitment of its staff, the client firms’ staff, and most importantly, the front-line service providers, i.e., the business consultants. The latter were highly flexible when (smaller) firms were unable to pay their cost-share of the TA due to liquidity issues; with many consultants even forgoing their own payments to compensate for that.  Other challenges the assisted firms faced were related to implementing the TA on their own without follow-on guidance or assistance from the consultants later on due to the limited size and/or scope of the TA. Additionally, the sometimes poor quality of the TA provided prevented the firms from fully benefiting from the assistance.  Based on the achieved results, which are very far behind the KPI targets, LED’s approach does not appear to have been as cost-effective as originally projected. Using LED’s cost-per￾job metric definition, LED’s cost per actual full-time equivalent (FTE) job created through direct assistance to firms, over the project’s life, was $1,827 – about four times the average cost per projected/targeted job of $443. The most reasonable explanation for this is that the original cost per forecasted job was estimated too low in the first place, although this underestimation is partly offset by the fall in the real USD value of local goods and services following the devaluation of the local currency. Furthermore, even after adjusting the figures based on the market exchange rate, any conclusion on the cost-effectiveness of the approach will be inaccurate, as it does not factor in the economic costs resulting from the vii | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 major context changes in the country. More importantly, without benchmarking the metric with comparable projects (in similar contexts), no final judgement can be made on the cost￾effectiveness of the approach. Cost-Effectiveness of Interventions (EQ4)  Quantitative analysis shows that LED’s most cost-effective TA was related to Human Resources (HR) & Restructuring, followed by Production & Process Management, then Knowledge Management and Quality Management. The least cost-effective was Supply Chain Management.  The most cost-effective firm size in LED’s interventions was large. Micro and small enterprises (MSEs) trail far behind the larger firms in terms of cost-effectiveness, likely because they were affected more heavily by Lebanon’s crises.  The most cost-effective sectors in LED’s interventions were Oil and Gas, Power and Water, and those that created new jobs, such as Tourism/Food Services and Healthcare. Trade and Transport and Logistics were cost-effective when looking at the incremental sales the assisted firms therein achieved. The least cost-effective sector was Recycling and Environment, which is a possible indication of the under-development of this sector in Lebanon.  In terms of regions, the Beqaa was the most cost-effective followed by Baalbek and Nabatiyeh, although Mount Lebanon and the North had the highest sales multiple and showed a potential for improving cost per job. The least cost-effective region was South Lebanon, likely because of the high prevalence of informal enterprises in the region. SUSTAINABILITY, SCALABILITY, ATTRIBUTION (EQ 5) Sustainability  There is no indication that LED’s key results in terms of job creation and sales increase at the assisted firms will not be sustained beyond LED’s life.  Primary and secondary data also point to moderate benefits reaped by the assisted firms, such as improved resilience, increased ability to identify business issues and gaps, increased capacity and confidence to hire business consultants in the future, etc. These are also likely to be sustained beyond LED’s lifetime.  The biggest contributor to the sustainability of LED’s impact on businesses is the change the interventions were able to make in the mindset of business owners and managers, most of whom now recognize (through first-hand experience) the importance of looking for new and more productive ways to run their business.  There is uncertainty regarding the sustainability of the market linkages established through LED between business consultants and MSEs, especially without third-party support (financial and non-financial). The severe impact of Lebanon’s crises on the businesses’ finances means that many firms cannot afford or would not prioritize external support. Scalability  Increasing efficiencies in the firm selection process along with the use of “standardizable” types of TA are possible factors that led to lower marginal costs when replicating the TA in general and specific types of TA.  In the future, after LED concludes, it is feasible to replicate and scale LED’s main interventions by building on LED’s success in creating positive experience for assisted firms and beneficiaries of LED grantee projects. Additionally, the increased supply of TA due to a larger, more viable, and more competitive business consultancy market, along with the market dynamics established between business consultants and firms, are key enablers to ensure the sustainability of quality viii | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 business development services that address firms’ needs.  Similarly, for future replication of firm-level TA, delivery challenges could arise from not being able to replicate other aspects of the LED Activity, such as the business advisor’s role as a facilitator of the trust-based relationship between the consultant and the firm, overseeing the design of the TA and tailoring its scope of work (SOW), supporting the firms in the selection of professional and credible business consultants, and ensuring quality TA deliverables. Additionality  There is overwhelmingly positive feedback from the various stakeholders and beneficiary firms regarding the role of LED/USAID’s assistance in adding a needed element to help sustain and advance the assisted firms in a time of crisis when there were only a few resources and channels of support available to them.  The business owners’ engagement in selecting business consultants and working closely with them to achieve the intended results, along with their commitment to their cost-share contribution, increased their buy-in of the process. Business owners also know now that not only can they hire business consultants in the future, when needed, but that they can also hold them accountable should they fail to produce relevant quality outputs that lead to results. RECOMMENDATIONS  Recommendation 1: USAID/Lebanon should re-assess the relevance and practicality of its focus on job creation, including in terms of indicators and targets, especially in light of the major changes that occurred in LED’s context.  Recommendation 2: Partners implementing Activities similar to LED should regularly conduct “value for money” assessments at the project level for the timely identification and implementation of improvements and adaptations to the design, objectives, and implementation processes.  Recommendation 3: In future enterprise development Activities, USAID/Lebanon should consider whether or not other/additional types of support can result in a higher impact and value for money compared to offering firm-level TA only. Examples of other types of interventions include focusing on specific sectors or value chains, strengthening the capacity of key market actors (including business consultants and business support organizations), and providing other types of firm-level assistance (such as financing/access to finance and facilitating business linkages).  Recommendation 4: Partners implementing Activities similar to LED should take into consideration the lessons learned by LED and its partners that could contribute to increased efficiencies in implementation processes and an increased efficacy of TA outputs, i.e., in the TA design phase (for higher relevance and applicability of TA SOWs), during TA delivery (for quality outputs), and post-TA (for continuous program improvements).  Recommendation 5: For future interventions, USAID/Lebanon should consider the various suggestions made by LED’s beneficiary firms and partners, such as continuing to use cost-share mechanisms, focusing on supply chain management in the form of capacity building for factories, organizing networking opportunities among assisted firms, and increasing beneficiary firms’ and business consultants’ awareness of USAID projects . 1 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 EVALUATION PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE The United States Agency for International Development (USAID) signed the Lebanon Enterprise Development contract with Chemonics International in October 2017 (No. AID-268-C-17-00001) for an initial value of $14 million until September 30, 2020. LED later received a two-year, $10.3 million contract extension through September 2022. LED was designed as an Activity under USAID/Lebanon’s Private Sector Development (PSD) Project. Its main purpose is to provide business development services and solutions to Lebanese firms to enable them to expand their activities, increase their sales, and create jobs. With LED approaching its end, its Contracting Officer’s Representative (COR) at USAID/Lebanon tasked the Monitoring and Evaluation Program for Lebanon (MEPL)—implemented by EnCompass LLC and Management Systems International (MSI)—with conducting a Final Performance Evaluation to assess the extent to which LED met its intended objectives. The evaluation provides USAID/Lebanon with information on LED’s intervention approach, its outputs and outcomes, and the likelihood of the sustainability of LED’s interventions. The report’s findings, conclusions, and recommendations will also help the Mission to plan follow-on Activities. This Final Performance Evaluation is mainly intended for USAID/Lebanon, specifically the Program Office and Economic Growth Office, in addition to Chemonics International and other parties and stakeholders at the discretion of the Mission. In the spirit of the USAID Evaluation Policy, this evaluation aims to provide USAID/Lebanon with concise, actionable recommendations based on evidence for the Mission to use in the design of future Activities and the improvement of program work plans. Additionally, in line with USAID’s Collaborating, Learning, and Adapting (CLA) framework, this evaluation aims to provide USAID/Lebanon, the United States Government, and the broader donor community with practicable lessons learned. USAID/Lebanon may use the findings and results of this evaluation in its annual Portfolio Review. MEPL will make the final evaluation report available to the public through the Development Experience Clearinghouse (DEC) repository and will carry out supplementary dissemination efforts in collaboration with USAID. The evaluation team followed a mixed-methods approach that draws upon quantitative and qualitative data collection and analyses. The data-collection methods included a desk review, key informant interviews (KIIs), and focus group discussions (FGDs) with USAID and LED staff and micro, small, medium, and large-sized enterprises (firms) assisted by LED. MEPL kicked off the process with a design meeting with USAID/Lebanon and LED’s staff on May 12, 2022, submitted the Inception Report on June 17, 2022, and conducted data collection between the end of June and the end of July. The evaluation team consists of Hoda Salman as Team Leader and Maya Trad as Team Member, with support from the following MEPL Beirut staff: Joanna Khater, Senior Monitoring and Evaluation (M&E) Specialist and Nour El Zaouk, Data Analyst. EVALUATION QUESTIONS (EQS) This evaluation explored the following lines of inquiry, in accordance with the Scope of Work (Annex I). ACTIVITY RELEVANCE 1. How relevant was the LED approach of firm-level, buyer-led, and generally market-led technical assistance through business consultants, in terms of meeting the evolving needs of the private sector in Lebanon, particularly MSMEs? 2. To what extent was the LED approach appropriate to help stabilize enterprises and jobs, support niches of growth through export and import substitution, and prepare the ground for future recovery? 2 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 ACTIVITY EFFECTIVENESS 3. What are the primary successes of the LED Activity? How and to what extent has the Activity achieved its planned results? What went well, and what contributed to this success? 3.a. How and to what extent was LED able to adapt its approach to major context changes in the country (financial and economic crisis, Covid-19)? 3.b. How was the success of LED influenced by contextual factors? What were the supporting factors? What challenges did the Activity face? 3.c. To what extent has the firm-level, buyer-led/market-led approach been cost￾effective in meeting LED’s objectives and KPI targets? 4. Which LED interventions and technical support have been the most and/or least beneficial, productive, and cost-effective? ACTIVITY SUSTAINABILITY, SCALABILITY, AND ADDITIONALITY 5. To what extent are the successes of LED likely to be sustainable and scalable and demonstrate additionality? 5.a. Which LED interventions and results show the most promise of being sustained, and why? 5.b. Which particular aspects of the LED Activity have resulted in the growing replication of technical assistance at declining marginal costs? Has there been any particular aspect of the Activity that ended up being more “one-off,” i.e., not scalable? 5.c. How likely is it that beneficiary firms would have been able to effectively research, procure, purchase, monitor, and achieve similar results had USAID’s assistance not been available? ABOUT LED ACTIVITY BACKGROUND Originally, LED was had a commissioned implementation period of three years with a $14-million budget. It later received a two-year extension through the end of September 2022 with an additional budget of $10.2 million. LED was implemented under a contract with Chemonics International (No. AID-268-C-17-00001) and aimed to increase employment opportunities for Lebanese citizens. The Activity facilitated the provision of business development services and solutions to Lebanese firms to enable them to expand their activities, increase their sales, and create jobs. ACTIVITY OBJECTIVES As per its contract and Results Framework (Annex II), LED had three objectives: 1. Create jobs by increasing the competitiveness and enhancing the growth of Lebanese client firms. 2. Improve Lebanon’s business enabling environment. 3. Communicate results through outreach and share knowledge with the public and private sectors. To achieve its primary objective of increasing job opportunities for Lebanese citizens, LED worked to increase sales and foster the growth of Lebanese enterprises to enable them to hire more people. LED’s primary approach was to work with private-sector firms to identify and overcome specific constraints to accessing known markets, thereby facilitating new relationships with buyers and boosting sales—i.e., through a “buyer-led” approach (BLA) to enterprise development. This support, 3 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 in turn, aimed to assist the firms in becoming more competitive and leading to new hiring—with new jobs serving as a primary measure of success across the LED Activity. LED’s second and third objectives were subsidiary to the first, supporting its goals and the overall Activity purpose of job creation. The three objectives were implemented in parallel in a tightly integrated fashion to achieve the purpose of job creation. LED collaborated with two local partners to conduct its work: Berytech and the Business Incubator Association of Tripoli (BIAT). ACTIVITY APPROACH In its work, LED applied the sector-neutral BLA to address the key constraints that prevent Lebanese firms (especially MSMEs) from concluding sales transactions, growing, and hiring more people. The main part of LED’s work was transactional, i.e., helping enterprises solve business problems constraining their growth. As a practical matter, such work involved expanding trade linkages and customer bases through acquiring necessary trade certifications, new modes of salesforce training, and other targeted actions. In particular instances, LED facilitated job matching, internships/apprenticeships, and enterprise-led workforce development. LED collaborated with other development partners, donors, business associations, and other USAID/Lebanon Activities to build synergies and leverage new opportunities to achieve optimum results. When binding constraints (for example, those that hinder exporting) resulted from the business environment, LED tried to engage business associations to play an advocacy role. LED partnered with enterprises and assisted them in trade facilitation, export promotion, and other regulatory or non-regulatory aspects related to job creation. Above all, the initiatives under LED aimed to increase sales that create new jobs, thereby contributing to private-sector growth and promoting a more inclusive economy. DEVELOPMENT HYPOTHESIS LED’s development hypothesis was that increased private-sector employment would contribute to economic stability in Lebanon through job creation. The theory of change was that private sector jobs had not kept pace with the growing numbers of Lebanese looking for work because of constraints to growth for private enterprises. Lebanese enterprises would expand their workforce if they could grow the demand for their products and services or identify new products, services, or markets. To achieve this, LED applied a problem-solving BLA to help Lebanese enterprises increase their sales and hire more Lebanese citizens. LED aimed to assist enterprises (that it selected based on specific criteria) to (i) identify specific buyer/s and understand their requirements; (ii) diagnose the key enterprise-level constraints that prevented them from concluding sales contracts with the buyer/s; and (iii) find solutions to these problems using tailored firm-specific solutions that would be delivered principally by Lebanese business service providers and consultants, or as needed, by international consultants. LED prioritized support for businesses that produce goods and services for a known buyer or market. Its work focused on making deals happen because sales growth leads to jobs. LED also worked to identify problems affecting the business enabling environment (BEE) in Lebanon and collaborated with the private sector to analyze them and propose solutions to them. CRITICAL ASSUMPTIONS According to Activity documents, key implementation assumptions for LED included the following:  Lebanon’s political, economic, financial, monetary, social, and security stability will improve. Stability and the prospect of a better future will create an environment for business expansion. However, the current political, economic, financial, monetary, social, and security situations in the country (including the dramatic effects of the Beirut port 4 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 explosion in August 2020, the Covid-19 lockdowns, the October 2019 revolution, and the difficulties the government is facing to propose and implement urgently needed reforms) have caused deep and serious challenges that affect the economic activities in Lebanon. Many sectors considered as target sectors for LED (such as hospitality and services, industry, etc.) have been hit hard, with many companies closing down, reducing their operations, and or letting staff go. Without stability, international buyers and investors will fear doing business with Lebanese firms, local entrepreneurs will delay expansion plans, and access to finance will not resume (it has effectively stopped) or become tighter. If stability returns, local businesses may look to expand their operations. But if instability remains or worsens, it will likely damage investor confidence and make expansion unlikely.  LED will be able to deploy its resources in a timely manner. The LED contract requires that project staff, consultants, subcontractors, and grantees receive various USAID approvals and pass robust vetting rules. LED will ensure rapid processing of client enterprises' support needs and assumes timely approvals and vetting results from USAID.  USAID is supportive of CLA as LED progresses. A necessary criterion for LED’s success is being responsive to evolving private sector needs and Lebanese economic realities. This may require adaptive course corrections during implementation per CLA principles. Adaptations that shorten the path to LED goals will be presented to USAID for discussion and approval. EFFECTS OF LEBANON’S ECONOMIC CONTEXT Lebanon defaulted on its sovereign debt in March 2020, and its currency has lost almost 90% of its value since October 2019. Efforts by the government to reduce public debt by printing currency have led to hyperinflation, which has cost most Lebanese their bank savings while reducing their salaries and pensions to some 10 percent of their 2019 value. Lebanon no longer has sufficient hard currency to pay for imports, constituting around 80 percent of its essential goods, including foodstuffs. In addition, over the past two decades, an overvalued currency undermined most productive sectors in favor of a rentier economy dependent upon constant infusions of foreign capital. But from 2011 onward, even this rentier economy stagnated. The Lebanese are left with little on which to fall back by way of domestic economic foundations and have few export opportunities to exploit, even with what is now a much more competitive currency1. According to the Lebanon Economic Monitor report published by the World Bank in January 2022, Lebanon’s real growth domestic product (GDP) has declined by an estimated 10.5 percent in 2021 following a 21.4 percent contraction in 2020. Lebanon continues to grapple with multiple crises: financial, economic, social, public heath, and political. All the underlying challenges that have been affecting the country for the past two years remain. The currency has lost 90% of its value. People’s savings, especially U.S. dollars, are locked in a defunct banking system. There is no access to finance or credit facilities. Inflation is soaring, including for food, fuel, and medicine. Supplies of many goods and services are lacking. Public electricity is limited to a few hours per day. Poverty is on the rise. Record numbers of people are leaving the country. In May 2022, Lebanon successfully completed its latest parliamentary election. Some new faces were elected, but in general, the status quo remains, and most people do not expect much to change. As of the writing of this report, Lebanon remains with a caretaker cabinet, with holdover ministers from the previous one. No reforms have been enacted that would unlock needed assistance from the International Monetary Fund (IMF)or other donors. 1 Source: USAID/Lebanon Political Economy Assessment 2021. 5 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 These unfortunate factors created an extremely challenging business environment that made it difficult for enterprises to plan ahead, grow, and hire new employees, which largely prevented LED from achieving its KPI targets. OVERVIEW OF LED’S ACTIVITIES AND OUTPUTS In the last two years, LED focused on export-oriented firms, firms that offer products or services that substitute imports, and firms operating in underserved or growing local market niches. LED’s assistance focused on a variety of technical assistance (TA) interventions, such as quality management systems, sales improvement, salesforce development, franchising development, production and process improvements, and human resources (HR) training and restructuring. By the end of its life, LED had directly or indirectly assisted a total of 766 enterprises and organizations. As a result of this TA, these beneficiary firms were expected to create a total of 8,001 new jobs. However, they only managed to create 4,263. Lebanon’s ongoing economic and financial crises, exacerbated by the Covid-19 lockdowns, negatively affected these firms’ hiring processes. OBJECTIVE 1 The activities and outputs under Objective 1: Create jobs by increasing the competitiveness and enhancing the growth of Lebanese client firms are described below. Types of Interventions By the end of its lifetime, LED had assisted or started to assist 766 private-sector enterprises and organizations through three types of interventions: (i) Core intervention: Direct TA through external business consultants to 521 firms (micro, small, medium firms mainly, but also some large firms) across various sectors (sector agnostic) and across all geographic regions. (ii) Grants to 14 organizations (e.g., ecosystem actors, such as enterprise support organizations, business associations and networks, consultancy firms, business schools, foundations, and employment platforms) which in turn supported 196 firms. (iii) Sub-contracts to four market actors to provide advisory services and coaching to groups of companies in specific sectors/industries (e.g., information technology IT, creative industries) or general sectors/industries (e.g., innovation and business continuity for small and medium enterprises-SMEs) which in turn assisted 48 firms (including 35 unique firms). Direct TA Over its five years of operation, LED profiled a total of 1,882 firms and closed 539 TA projects or “deals” (including second/repeat TAs to 18 firms) through memoranda of understanding (MOUs) signed with client firms. Most deals were:  In the food processing (36%) and manufacturing (18%) industries.  Located in the Mount Lebanon (46%) and Beqaa (17%) governorates.  Focused on quality management (31%), followed by Access to Market (13%) and Sales Training (17%).  Conducted with small (50%) and micro (30%) enterprises. Grants LED’s grant-supported solutions fall under one of two categories: 1. Firm or industry competitiveness: Grants address constraints that result in directly 6 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 improving business operations for firms. Examples include: Workforce Training, Skills Building and Improved Employability, Strengthening Franchising Capabilities in Lebanon, and Standardizing the Management and Development Consultants Industry. 2. Improved BEE: Grants that address policy, regulatory, or administrative constraints to doing business that will reduce the cost of doing business, increase investment, or have other beneficial impacts, thereby accelerating private sector growth and job creation. 2 Examples include: the Economic Framework and Roadmap for the Creation of Sustainable Private Sector Employment in Lebanon. Direct Support to Groups of Enterprises In its final year (Year 5), LED pursued other technical activities aimed at creating opportunities for new employment within the private sector.3 In addition to grantees delivering such opportunities, LED sub-contracted four local service providers to deliver four (pilot) initiatives of capacity building and TA to groups of SMEs: two initiatives with firms from miscellaneous sectors, one with IT firms, and one with firms working in the creative industries. These initiatives ran from early 2022 to June/July 2022. OBJECTIVE 2 Per LED’s contract, Objective 2: Improve Lebanon’s business enabling environment is not a main component of the Activity4 and therefore accounts for no more than 10% of the total level of effort (LOE) by its implementing partners. As such, activities under Objective 2 were by design limited to a two-pronged approach. First, LED planned activities that aimed to strengthen the tourism sector (following an assessment it carried out in Year 1). Second, LED planned to identify opportunities to engage in “traditional” BEE interventions (policy, legal, regulatory, administrative, budgetary, or institutional reforms that could have demonstrable links to creating jobs for Lebanese; reducing the cost of doing business; increasing private investment)5 utilizing a “bottom-up” approach that engages a large number of assisted firms to understand what the priority issues are. Through two grants issued to two private sector organizations, LED produced a total of 16 BEE outputs covering two areas: (i) expert roundtables leading to a report that examined the challenge of employment creation by the private sector during Lebanon’s crises (in Year 4), and (ii) an assessment of the tourism industry in Lebanon and its potential to support rapid economic growth in the country (in Year 1) and webinars on hot topics of the tourism and hospitality industry (in Year 4). As a result, LED reported results that exceeded the targets under three of its key performance indicators (KPIs) (PIR 10, 11 and 12) but was unable to achieve the BEE outcome target PIR 96. Implementing activities under this objective was challenging due to two reasons: finding interest among public- or private-sector players (thus delaying the start of these activities) and the impact of Covid-19 on planned activities under Objective 2 (mainly events with stakeholders that had to be delayed or canceled). Although it was able to implement some of its planned activities, LED could 2 LED’s Year 2 Annual Report – page 33 3 Funding for these initiatives came from a surplus in the budget, resulting from a calculation error according to LED. 4 The primary component is Objective 1 (direct job creation through a firm-level buyer-led approach). 5 LED Annual Reports 6 PIR 9 indicator is the “Number of policies/regulations/administrative procedures in developmental stages of analysis, drafting and consultation, legislative review, approval or implementation as a result of USG assistance.” 7 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 not progress far or achieve significant impact for the following reasons. Since the first year, LED faced issues with identifying private- or public-sector collaborators to work on BEE issues, likely due to people’s frustrations with the lack of reforms made by the Government to-date. Political volatility and uncertainty, particularly from October 2019 to May 2022, with short-lived or limited￾functionality caretaker governments also likely played a role. LED, instead, focused on sector-specific approaches to implementation (as a secondary objective to its firm-level approach). Thus, in Year 2 LED funded (i) a series of activities to strengthen the tourism sector, and later in Year 4 (ii) (through a grant) a study to identify the main challenges that Lebanon-based media organizations were facing and to develop an action plan for improving the business environment and developing strategies that are aligned with international industry best practices. However, the activities did not extend beyond these outputs, and therefore no outcomes (policies, regulations, etc.) were achieved. OBJECTIVE 3 Objective 3: Communicate results through outreach and share knowledge with the public and private sectors is complementary to Objective 1 and contributed to its success through the use of communications to report externally and internally on LED’s overall successes and achievements throughout the past five years. LED used stories to showcase the impact of its support to client firms and advertised them mainly through conventional and social media channels. Therefore, the evaluation team will assess Objective 3’s activities and results as part of our assessment of Objective 1’s. EVALUATION METHODS AND LIMITATIONS MEPL’s evaluation team utilized the Appreciative Inquiry (AI) model to conduct LED’s Final Performance Evaluation.7 AI is a theory and a practice of inquiry and change that shifts the perspective of evaluation by suggesting that the very act of asking generative questions has a profound impact on systems. It is both a worldview and a process for facilitating positive change. Its assumptions are simple: Every human system has something that works. We believe that what we focus on grows. We move in the direction of the questions we ask. AI is a strengths-based approach that studies best experiences and practices, builds on strengths, and facilitates improvement and change. The evaluation team used AI throughout the evaluation when conducting the following:  Planning for the evaluation, engaging in appreciative conversations with stakeholders, and appreciatively framing the evaluation questions.  Designing the data-collection tools in a way as to include appreciative elements and focus on identifying successes  Analyzing data using an appreciative lens to highlight successes and explore key enabling and constraining factors.  Conducting a participatory interpretation of data with LED’s senior management, where needed, to ensure context relevance. The data-collection methods, processes, and limitations are presented in Annex III. 7 https://www.centerforappreciativeinquiry.net/more-on-ai/what-is-appreciative-inquiry-ai/. 8 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 EVALUATION FINDINGS AND CONCLUSIONS The findings and conclusions below describe whether or not, how, and to what extent LED has been able to meet its three objectives by a) supporting firms to sustain, expand, and increase their competitiveness outside Lebanon amidst the crises, and b) improving the BEE while sharing knowledge with the public and private sectors. EVALUATION QUESTION 1 (RELEVANCE) EQ 1: How relevant was the LED approach of firm-level, buyer-led, and generally market-led technical assistance through business consultants, in terms of meeting the evolving needs of the private sector in Lebanon, particularly MSMEs? FINDING 1.1 LED shifted from the buyer-led approach (BLA) it had followed initially to a more flexible and inclusive market-led approach (MLA) that focused on having a clear market demand in specific market(s) instead of a specific buyer. Therefore, LED’s firm-level, market-led TA was more relevant to addressing the specific needs of Lebanese firms than a traditional buyer-led approach, as it tackled critical and changing business constraints faced by the firms, especially MSMEs, thereby helping them to sustain and create jobs. For most firms, it came at a time when they were in a dire need for sustaining their business. However, collected data also reveal some constraints related to the ambitious job creation target expected from the TA. Firm-Level Approach LED implemented its firm-level approach based on the premise that the firms – with support from LED’s business advisors – are able to identify a pertinent issue that is hindering the development, competitiveness, and growth of their business. Data collected for this evaluation show that LED used a problem-solving approach to address constraints to improving productivity (through improved businesses practices and technology) and improving access to markets. From a Buyer-Led to a Market-Led Approach At the start of the LED Activity, the program team worked by linking TA directly to a specific (named) buyer, i.e., supporting a business in meeting the requirements to conclude the sales of products or services for a specifically identified buyer – an “orthodox” or strict version of the BLA that focused on sector-agnostic firm-level assistance with full additionality by and attribution to LED. These requirements could be related to standards, price, quantity, or other aspects of the product or service. However, this approach proved too restrictive, because, unlike other enterprise￾development interventions, the “traditional” BLA works with buyers and then links them back to potential suppliers, rather than selecting firms to become the target of assistance. The BLA was contributing to delayed enterprise engagement and resulted in a slow traction for LED in the first year. Following discussions with USAID/Lebanon, LED ended up revising this approach and adopting a “more inclusive and flexible” definition of BLA, which ultimately helped jump-start the enterprise￾engagement process. The revised approach also created space for LED to take advantage of targets of opportunity to help firms boost demand for their goods and services, leading to new jobs.8 Thus, after the first year of operation, LED relaxed the requirement that a firm should have a specific (i.e., named) buyer at the time of application and instead focused on having clear market demand in 8 LED Year 1 Annual Report – page 15 9 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 certain markets that the business was aiming to serve (with implied buyer interest). The new approach was dubbed “market-led approach” (MLA). According to LED, an MLA that is rooted in the BLA aims to identify clear business-growth opportunities to link local producers to new markets or to strengthen and expand existing market connections between producers and consumers, thereby increasing sales (exports or domestic) and creating jobs. LED then worked with these actors to define problems or constraints that may limit or inhibit this potential.9 To accelerate job creation, LED also sought out other opportunities to create employment by, for example, supporting sector initiatives through private-sector associations or syndicates (or non￾governmental organizations-NGOs) to offer solutions to common problems, in its initial years, and, in its final year, by grouping firms by sectors (e.g., IT and creative industries) to address common constraint/s to sales and job growth. LED gave grants to local NGOs and a few private enterprises to build their capacity and support new initiatives aimed at increasing employment and enterprise￾development opportunities. In 2022, LED co-designed and funded pilot interventions targeting business sectors or groups of companies (e.g., in the IT sector). The FGDs and KIIs with beneficiary firms show that most firms were satisfied or highly satisfied with the TA they received from LED, with the majority saying it was the right TA that came at the right time and motivated them to find new avenues for growth despite the country’s difficult circumstances. The same was reported in a survey of beneficiary firms conducted by LED (the Self￾Evaluation Survey for Client Enterprises), as well as in a study conducted by the Holy Spirit University of Kaslik (USEK)10. This was also evidenced by the surge in the number of deals closed by LED (i.e., MOUs signed with client firms) following the crises. (see Figure 1) The BLA model traditionally aims to facilitate contacts between local producers and domestic and foreign buyers and to help these producers meet market demand by organizing themselves to become serious suppliers. One of the lessons learned from applying the BLA elsewhere successfully 9 LED Year 2 Annual Report – pages 6-7 10 The USEK Study “LED Impact on Assisted Enterprises Since Inception,” funded by a LED grant, was conducted over the period January to June 2022 and included surveys of LED-assisted firms and business consultants. Figure 1: Number of deals closed by LED over its five years of operation. Data source: Data of LED-assisted firms, as received from LED 25 105 109 183 117 25 130 239 422 539 0 100 200 300 400 500 600 FY17/18 FY18/19 FY19/20 FY20/21 FY21/22 LED Deals No. of Deals Cumulative No. of Deals 10 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 is that it is important to promote local service providers and their abilities to provide services to clients.11 By design, LED does not have an objective of building the capacity of local service providers, although indirectly it did contribute towards this BLA element through one of the grants and also through its engagement of many business consultants (about 160) to deliver TA to firms. The Goal of Job Creation Both USAID/Lebanon and LED’s leaderships confirmed the MLA’s relevance to supporting enterprise growth and job creation in Lebanon, particularly since LED was able through its flexibility to address the firms’ evolving needs during the crises (as described further down in EQ3A). However, LED’s leadership also raised a concern about the Activity’s primary goal of job creation and the very ambitious target set for that – about 10,000 full-time-equivalent (FTE)12 new jobs – through firm-level interventions only. The main reason is linked to the Lebanese market’s structure, which consists of only a small number of large corporations and has an overall limited production of goods (as most goods are imported). This market structure therefore offers limited opportunities for value-chain backward linkages that can absorb large employment numbers, thereby limiting job creation at scale. In addition, a LED senior manager questioned the strength of the developmental impact on firms when the TA is restricted to one consultancy project and its size is limited by the minimum number of jobs that the firm can commit to creating as a result of LED’s TA. (Only 18 firms [<4%] received a second TA13, and it is not clear on what basis they were selected since not all were able to achieve all their first TA targets.) Most firms have many problems to solve, especially as they attempt to recover, sustain, and grow in a time of crisis. Referring to USAID-funded projects in other countries where any given firm might benefit from five or more different types of interventions (TA, training, workshops, business-to￾business B2B events, trade shows, etc.), the LED senior manager wondered whether more attributable impact would have resulted had the LED-assisted firms been able to benefit from more than one TA or type of intervention. It is likely that the main constraints that prevented LED from facilitating additional support were (i) its aim to reach as many firms as possible with TA, (ii) the eligibility of firms for a second TA, and (iii) the available budget. Most second TA rounds were implemented in the last 18 months of the Activity. 11 Pattie Preston, (2014, May 22). “10 ‘Micro’ Tips for the Buyer-Led Approach to Fighting Poverty”, 10 'micro' tips for the buyer-led approach to fighting poverty | Devex 12 One FTE job equals 260 working days (8 hours/day) or the cumulative of 12 months of full-time paid work, e.g., 3 jobs of 4 months each can be counted as 1 FTE job. FTE jobs allow for counting full-time and part-time (seasonal) employment. 13 Dr. James Riordan, Expert on Buyer Led Approach, https://www.youtube.com/watch?v=bzo2AIWUKkc “LED was unable to work with us on the legal part of the needed assistance to expand outside Lebanon, because it is not within the scope of LED. We also wanted an ISO certification, but LED refused, as we are only allowed to receive one type of assistance. The priority was marketing and rebranding, but we also needed to certify the platform and the security of the platform.” — FGD Respondent 11 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 FINDING 1.2 The majority of assisted firms expressed a crucial need for external funding (cash grants), particularly in light of the negative effects that the financial crisis had on their cashflow. Cash grants, they said, would have helped them apply the TA they received from LED and maximize its impact of on their business. However, by design, LED did not include a cash-assistance component. The firm-level approach was narrow in focus and did not complement the TA with co-investment funds (cash or in-kind grants) that would have allowed firms struggling with tight cashflows to implement the TA’s recommendations and roadmap by directly investing in their business. About 20% of the firms and 50% of the consultants interviewed by the evaluation team noted the inability of firms to implement/continue implementing the TA (e.g., through re-hiring the consultant, investing in business infrastructure changes, and/or acquiring quality management certifications) after LED’s intervention, or to even pay their cost-share to the business consultant, primarily due to their tight cashflow. Additionally, in LED’s survey of client firms, only 20% of respondents said that TA had been of greater value to them than funding. Indeed, most firms indicated that funding (grants) coupled with TA would have been more effective for them. Tight cashflows were mostly an issue for firms that received TA on quality management/certification, as they were unable to pursue and invest in the required business changes and infrastructure renovations to qualify for certification and/or be able to pay the certification fees. It was apparent during the FGDs that for many of them this was a major shortcoming of the TA, as they were unable to implement the required steps on their own without external financial assistance. This was also considered as a major obstacle for many firms by several of the interviewed business consultants, who said that LED had not offered TA related to access-to-finance facilitation. It is worth mentioning though that in Year 1, LED did attempt to create partnerships with local financial providers (banks, donor funds) to leverage funds for its assisted firms. However, these discussions did not go far, and the process was aborted soon after. It should be noted that the traditional BLA model assumes that finance is not a problem and that the banks are willing to lend to businesses if the latter’s market / buyers are secure, and good technologies and business practices are ensured.14 However, the situation in Lebanon is extraordinary, as the ongoing financial and economic crises, which exacerbated as of the second half of 2020, have left companies “high and dry” with no access to credit or finance in addition to huge restrictions on payments and access to their bank accounts. EQ 1 CONCLUSIONS  LED adopted a more inclusive form of BLA—the MLA—which proved to be more flexible and productive in Lebanon’s context in terms of responding to the evolving needs of Lebanese firms as they tried to navigate high market uncertainty amid risks to their survival and growth. LED complemented this approach by working with market actors to increase employment creation and firm-level competitiveness. However, LED’s objectives did not specifically or intentionally target relationships between local service providers (such as business consultants) and firms (as local producers), which a more comprehensive BLA or MLA would have done in order to meet the increasing market demand.  LED’s tailored TA was driven by business and market demand and aimed to satisfy specific needs that were rapidly evolving amidst Lebanon’s multi-dimensional crisis. Most enterprises 14 Dr. James Riordan, Expert on Buyer Led Approach, https://www.youtube.com/watch?v=bzo2AIWUKkc 12 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 were satisfied with the TA they had received and confirmed that the TA approach had been relevant for them to identify successful solutions to some of their business problems, particularly in relation to expanding their access to markets outside Lebanon and/or improving their productivity. However, more TA per firm could have generated better results for the firms and had more developmental impact by and attribution to LED.  The findings reveal some constraints related to the ambitious job creation target expected from the TA in the context of Lebanon. The job-creation goal of the LED Activity does not reflect fully the challenging context of the country and local markets, and therefore it cannot be tackled solely by firm-level TA intervention.  An essential component that was missing from LED’s approach was complementary funding (mainly in the form of cash grants) given the local context. This was viewed by most firms as important, if not crucial, during hard times. It would have been quite vital, especially for small businesses, to help them fully implement the TA they received and maximize its impact on their finances and operations. EVALUATION QUESTION 2 (RELEVANCE) EQ 2: To what extent was the LED approach appropriate to help stabilize enterprises and jobs, support niches of growth through export and import substitution, and prepare the ground for future recovery? FINDING 2.1 The LED approach was appropriate to support firms to sustain their operations and employees and, in many cases, to grow and create more jobs. LED tailored its support to the needs of the targeted firms but also focused its approach (more so post-crises) on interventions that increased export and import substitution. All stakeholders agreed that LED had been successful in helping firms to sustain their business despite the worsening business climate in the country. More importantly, LED gave hope to firms in a crisis situation, “holding their hand” while they were adapting their businesses to the new circumstances and prioritizing the retention of their existing employees. While a few assisted firms did close their operations in Lebanon permanently, most of those that survived attribute their ability to sustain their business to LED’s support. Most interviewed business advisors, firms, and LED staff confirmed that because of LED’s interventions, many of the firms had been able to reach their job-creation targets and to even exceed them sometimes, while others had been able to sustain their existing jobs despite the Covid￾19 lockdowns and the recession. This is confirmed by the KPI results collected by LED through early August 2022, which show that 60% of the assisted firms created new jobs as a result of LED’s intervention. A similar result emerged from the KIIs with grantees (market actors), who noted that they had been able to engage more companies and create more jobs than committed. From the beginning, LED’s approach was based on directly identifying and solving business problems that constrained enterprises’ ability to increase their sales, domestically and internationally, and create jobs. LED was also keen to develop market-growth initiatives through partnerships with local business groups, such as one that received a grant to strengthen the franchising capacity of Lebanese firms, which complemented its direct TA to firms. The worsening economic crisis emphasized the criticality of exporting, to at least sustain the business and provide a lifeline of fresh USD revenues to businesses. Import substitution was also a focus to reduce companies’ input costs and help them adapt to the changing market dynamics following the decline in local consumers’ purchasing power. It can be seen from the types of TA delivered that most (56% of deals) related to quality management, access to markets, and sales training, followed by product development and production and franchise development (8% each). This assistance contributes significantly to export preparedness and import substitution for the assisted firms. In addition, just over a half (51%) of the 13 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 grant-funded initiatives provided business advisory and coaching/mentorship services to firms directly to help them survive and sustain their operations (business continuity), scale up, and/or reach new international buyers and markets. EQ 2 CONCLUSIONS  LED’s interventions were very appropriate in terms of helping stabilize enterprises and jobs, as most firms attributed their survival to LED’s support, which they said had come at the right time. The majority were even able to create new jobs thanks to LED. The grants to market actors helped enhance job creation by supporting firms and create employment and entrepreneurship opportunities for individuals.  LED’s support for export and import substitution was more focused during the crises and was critical at a time when businesses needed to find sources of fresh USD and/or control or reduce their increasing production costs due to rampant inflation and more expensive imports of equipment and raw materials. EVALUATION QUESTION 3 (EFFECTIVENESS) EQ 3: What are the primary successes of the LED Activity? How and to what extent has the Activity achieved its planned results? What went well, and what contributed to this success? This question assesses how effectively LED was able to meet its objectives (primarily Objective 1) and achieve its key outcome targets (for jobs, sales, and investment), particularly in light of the challenging and deteriorating economic context during its lifetime. LED’s Key Achievements (Outcomes) Summary Table 1 below summarizes the status of LED’s key outcome indicators through the end of Q3 FY 2021/22 and compares actual results vs. targets. Note that the LED contract only contains the job creation targets (FTE new and continuing). Other targets were developed in LED’s Monitoring, Evaluation, and Learning (MEL) Plan in 2018 (for Years 1-3) and later in the updated MEL Plan 2021 for Years 4 & 5 proposed targets. Table 1: Status of LED's key performance indicators Key Indicator LOP Target** Total Actual* Total Actual (adjusted for avg. LBP/USD rate)* Achievement (%) Achievement (%, adjusted) Number of full-time equivalent (FTE) jobs created 3,626 new; 5,012 continuing15 2,377.5 new; 3,622.5 continuing n/a 66% 72% n/a Value of incremental sales (domestic and $126.91 million $614.77 million $57.91 million 484% 46% 15 Original target was 9,800 new FTEs and 17,743 continuing FTEs. Cumulatively by YR 5. Continuing FTE refers to jobs created in the previous reporting year and maintained for the current year (i.e., 12 months later). 14 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 Key Indicator LOP Target** Total Actual* Total Actual (adjusted for avg. LBP/USD rate)* Achievement (%) Achievement (%, adjusted) export) collected at firm level for enterprises Value of new private sector investment leveraged $19 million $48.84 million $4.60 million 257% 24% Source: LED's FY22-Q3 progress report, LED Revised MEL Plan (of 17th December 2021). * LED converts all LBP reported data at the official rate of 1,507.5 LBP to 1 USD, when in fact – and as LED acknowledges in its reporting to USAID – there are different parallel exchange rates (LBP 3,900, 8,000, Lollar, Sayrafa, and black-market rates) employed by market actors. Thus, the reported USD value is much higher than the real value of the reported sales, since the larger proportion of reported sales is domestic (vs. export). Therefore, to account for the devaluation of the LBP against the USD and to reflect more closely the market exchange rate and compare more like-with-like, the evaluation team adjusted the monetary values (i.e., for sales and leveraged investment) to reflect an average market rate of LBP 16,000 per 1 USD over the last five years16. The resulting adjusted values (‘Actual (adj.)’ column) were then compared to the targets in the last column. ** As reported in LED’s FY22-Q3 progress report, which states that these updated targets (updated from the original contractual targets) were submitted to USAID on December 17, 2021 and approved on April 28, 2022. The evaluation team notes that these target revisions are not reflected in a LED contract modification and were not raised by LED KII participants and thus will assume they are the final life-of-project (LOP) targets for the evaluation of LED’s performance. (Table 2) Table 2: LED’s revised KPI targets (April 2022) Key Indicator Revised target Original target Change from original target Number of full-time equivalent (FTE) jobs created 3,626 new; 5,012 (continuing) 9,800 new (during reporting year); 17,743 continuing -6,174 (-63%) new; -12,731 (-72%) continuing Value of incremental sales (domestic and export) collected at firm level for enterprises $126.91 million $245.0 million -$118.09 million (-48%) Value of new private sector investment leveraged $19.037 million $49.0 million -$29.963 million (-61%) 16 Note that the LBP 16,000/USD average exchange rate is an approximation of the weighted average LBP/USD in the parallel market (weighted by the number of TA deals executed per calendar year). As it was difficult to find the history of the LBP/USD parallel market rate since August 2019 and due to the high volatility of the parallel market we relied on the evaluation team’s estimation of the average rate annually for 2019 to 2022 using one online data source (The Fluctuation of Lebanese Lira Against the US Dollar In the Parallel Market - BLOMINVEST (blominvestbank.com)). 15 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 Source: LED Q3-FY5 quarterly progress report (for revised targets), LED contract (for number of FTE jobs created), LED revised MEL Plan (for original sales and investment targets) Table 1 shows that LED achieved 66% of its created (new FTE) jobs target, 46% of its (adjusted) incremental sales target, and 24% of its (adjusted) leveraged investment target. In addition, for the reasons outlined earlier (under Overview of LED’s Activities and Outputs), LED did not achieve any results for the Objective 2 outcome in relation to BEE policies and regulations. In terms of how many firms were positively affected by LED’s TA, 60% reported to LED that they had employed new staff, 36% that they had increased their sales, and 47% that they had invested in their business as a result of the TA they received. See Table 3 below. Table 3: Number of firms that hired new staff, increased their sales, or leveraged investment thanks to LED’s TA KPI No. of firms creating jobs, or increasing sales, or leveraging investment % of total assisted firms (n=521) New Jobs 311 60% Incremental sales 190 36% Leveraged Investment 247 47% Source: LED KPI Results (through August 9, 2022) An unintended positive effect of LED was that it indirectly helped to sustain and strengthen the capacity of about 160 local business consultants by connecting them with local firms. LED built and utilized a pool of 78 individual consultants and 88 consultancy firms.17 Its role was to facilitate discussions between firms that had a potential for growth and employment creation on one hand and qualified consultants and consulting firms on the other, concerning any services that the former might need to tackle the identified constraints. LED, through its business advisors as relationship managers, fostered trust between the two parties (firms and business consultants) and incentivized linkages between them by sharing the cost of the consultancy with the firm, which aimed to lower the perceived risk of external consultancies to the firms and encouraged business consultants/consultancy firms to work with smaller firms (MSMEs). EQ 3.A: How and to what extent was LED able to adapt its approach to major context changes in the country (financial and economic crisis, COVID-19)? FINDING 3.A.1 LED was quick to adapt its approach to major context changes in the country – while remaining within the confines of the existing Activity design and mandate – by being agile and pivoting in small but numerous ways to implement (non-fundamental) changes to its approach and delivery. It shifted its aim to balance job-creation and job-sustaining interventions, as it intended to support as many firms as possible. 17 It should be noted that it is very common for an individual consultant to have an established consultancy firm. Indeed, a number of the interviewed individual consultants indicated that they had in parallel set up a consultancy firm, either before or during their engagements with LED. 16 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 LED continued to adapt throughout its life, using multiple channels (business advisors, outreach events, MEL) to receive daily feedback/inputs from firms to inform its adaptation efforts. However, per USAID’s wish, there were no major or formal shifts or changes to the Activity design or objectives (although one key informant said, “LED pivoted and adapted much more than expected”). LED’s response to the evolving context was underlined by increased flexibility with its implementation approach. Specifically, LED implemented the following measures and changes that largely reflect its recognition that the economic situation in Lebanon has made it more difficult for firms of all sizes to create new jobs and for many smaller firms to cost share.  LED wanted to assist as many firms as possible to (i) adapt to changing market demand and (ii) sustain and retain existing employees and hire new ones where possible. Therefore, it prioritized firms and sectors with good potential for export and import substitution in sectors such as food production, manufacturing, IT, and services. In addition, LED started accepting more micro and large companies, where they met its selection criteria.  LED adjusted its cost-sharing requirement by decreasing the client firm’s cost-share from around 50% to 5-30% (depending on the size of the firm and other factors).  LED increased the cost per committed job ratio (an indicator used to calculate the budget for a given TA project) from an average of $500 to approx. $1,000 per committed job.  LED paid its business consultants in fresh USD instead of LBP, which made it easier to negotiate lower business consultant rates (to maximum $250 based on USAID’s guidance and/or LOE, depending on the type of TA).  The geographic coverage of LED’s partner BIAT was extended from North Lebanon to include other regions as well (as the number of eligible companies were very limited in the North) and to improve its outreach and capitalize on the business advisors’ wide networks that extended beyond their originally assigned regions,  LED shifted its activities from physical to online events and topic-focused webinars during the Covid-19 lockdowns.  According to interviewed grantees, LED was also flexible when they asked it to revise grant milestones, including to move their planned activities online. LED’s adaptation had a positive effect on the ground as evidenced by the fact it reached more firms with TA in the last three years. As noted in Table 3 above, 60% of the firms reported hiring new Lebanese employees as a result of LED’s assistance. “…When the financial crisis hit Lebanon, it affected the operations of companies. So, LED became more flexible and reduced the cost share by 5 to 10% each time, finally reaching the 20 – 30% commitment level from companies.” — KII Respondent (Business Advisor) “Some companies were investing more because of the fresh dollars flow into their business and this was evident by the end of the program.” — KII Respondent (Business Advisor) 17 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 FINDING 3.A.2 LED re-focused its TA offerings to reflect the changing market demand and the firms’ evolving needs as a result of the Covid-19 measures. It introduced topical webinars and online events when in￾person support was not possible—which the firms perceived as valuable. As a result of these adaptations, LED boosted its outreach to firms and significantly increased the number of assisted firms, in Year 4 particularly. Lebanon was hit by multiple crises during the implementation of LED, each of which brough its own, and in some cases unique, challenges for businesses. The Covid-19 situation required new health and safety measures be implemented by all companies, particularly in factories and industries where remote work was not an option. New grants by LED were given to support firms in implementing hygiene and business continuity measures. LED also shifted its activities (internal and external) to online during the Covid-19 lockdowns and produced – with support from its partner Berytech – a series of nine technical webinars that proved popular among attendees (attended by over 1,500 businesspeople). These webinars covered hot topics, like crises management, how quality assurance programs can help boost exports, how to access new markets, and how to cost and price in an environment with severe currency depreciation and high inflationary market. These events also caused a significant increase in online applications compared to the previous year; for example, the first quarter in Year 4 saw a 142% increase of online applications compared to the same quarter of the previous year. LED recorded all the webinars and posted them on its YouTube channel as a reference.18 One firm participating in an FGD stated that these webinars had been as beneficial as TA. On the other hand, in-person support by business consultants was not possible or severely restricted during the lockdowns, which slowed down the delivery of TA. LED and the business consultants adapted by shifting to online meetings and interactions with client firms, where feasible, or by reducing the number of site visits. 18 LED Year 4 Annual Report “We shifted from niche high-end products to products that are more affordable according to the new living expenses in Lebanon. We also reduced our profit margins to try to help consumers purchase and afford our products. Also, it is important to note that the entire way of doing business in Lebanon has changed from a credit base scenario to only cash on delivery, which tremendously helped in turnover and collection.” — FGD Respondent “We attended training provided by LED during the crisis, on crisis management. One of the major decisions that we took as a result of this training was to sell part of our assets to sustain our business during the crisis. This was based on LED’s advice – a difficult decision but a good one.” — FGD Respondent 18 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 FINDING 3.A.3 Given the ongoing challenges and multi-faceted crises in the country and with the limited changes to the approach permitted by USAID, it was impossible and unrealistic for LED to be expected to achieve its original targets for Year 5. Although the targets were seemingly amended in Q2 FY519 (to reflect end of Year 3 targets originally agreed by USAID), given the ongoing country challenges and their detrimental impact on businesses, and with the limited changes implemented to the approach following the crises, it was impossible and unrealistic for LED to achieve its target results. The high target for job creation was cited as a major challenge by LED’s leadership since the country’s difficult context meant that there was a lot of effort that had to be spent by the LED team to achieve monthly targets and a constant momentum that had to be maintained towards the primary goal of job creation. The consequence, in the LED key informants’ view, was that there was no time for the team to reflect on the results and learnings to feed back into further adaptation of the Activity for maximizing the impact. Thus, it was a missed opportunity to adjust the outcome targets earlier on, to better reflect the new market reality and correct for a more realistic and reasonable target for the KPIs that the LED program team could work more effectively towards. EQ 3.A CONCLUSIONS LED was quick to adapt its approach following the onset of Covid-19 and the economic crises, but in small ways only, as it had to remain within its original mandate and objectives. The main changes that LED implemented related to (i) slightly shifting its focus from purely creating jobs to sustaining them too, (ii) reducing the cost-share for firms to encourage them to prioritize external assistance, and (iii) being more flexible with its criteria for firm selection (to reach as many firms with potential for growth as possible). LED introduced webinars that provided topical and practical knowledge to help firms deal with the crises and become more resilient. These webinars complemented the TA provided through the business consultants. These changes were important to incentivize more firms to apply to and benefit from LED’s TA, with the majority of firms creating new jobs as a result, albeit much lower than the target. This also meant that the already ambitious original job-creation targets were even more unachievable than before, and LED was spending a lot of effort on this KPI, thereby possibly missing other opportunities that could have maximized other growth drivers of the assisted firms, such as sales growth. EQ 3.B: How was the success of LED influenced by contextual factors? What were the supporting factors? What challenges did the Activity face? FINDING 3.B.1 The main challenge to the achievement of LED’s purposes was the fast deteriorating socio-economic and financial situation in Lebanon, which severely restricted enterprises’ economic activities and liquidity, rendering a significant proportion of the assisted firms incapable of achieving their committed job-creation and sales targets. There was consensus among all interviewed stakeholders that the most significant challenges to LED’s success were the multi-dimensional effects of the various crises in Lebanon on the firms’ solvency and operations, particularly MSEs, which are the majority of businesses in the market. As is 19 In its Q3-FY5 quarterly progress report, LED shows updated targets which it states are “updated targets submitted to USAID on December 17, 2021 and approved on April 28, 2022.” 19 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 the case globally, while small businesses are more agile than their larger corporate counterparts, they often lack the deep pockets, resources, and experience to survive under pressure for long periods. Almost all firms that responded to LED’s survey said their sales and investment would be higher if the situation in Lebanon were different. The challenging business environment in Lebanon posed several internal and external challenges for LED-assisted firms, which constrained their ability to implement the TA optimally. According to the KIIs and FGDs, the main challenges included:  Temporary business closures due to Covid-19.  Difficulty retaining staff and keeping them motivated in light of the tough economic situation.  Having to lay off staff for financial reasons.  Difficulty finding skilled workers for all position levels at affordable salaries and across regions (including in remote areas), especially while having to compete with the fresh USD salaries being offered by international NGOs and donor-funded programs.  Lack of basic government services, especially state electricity, and fuel shortages that prevented many public- and private-sector workers from traveling to their workplace  Increasing input costs (worker salaries and transportation, local and imported raw materials, electricity, fuel, etc.) due to soaring inflation.  Steep devaluation of the local currency and a highly volatile exchange rate with more than four parallel (non-official) exchange rates being adopted in the market.  Difficulty receiving or making payments due to the limited availability of banking services, strict limitations on cash withdrawals, high transaction fees, and consumers’ limited purchasing power.  Limited access to liquidity due to lack of external funding and limited internal cashflows prevented firms from investing in their business and making improvements.  Lack of USD to import raw materials or pay for certification licenses, etc.  Lebanon’s negative reputation abroad, the lack of government support, and the lack of investment in infrastructure and export facilitation services hindered Lebanese exports and the integration of Lebanese firms/solutions into global supply chains. Some business consultants went even further and noted that the crises had pushed some firms to delay the start of the TA, not pursue the implementation of the TA at all, or delay certain steps of the TA (e.g., acquiring a certification) until they really needed it (e.g., to export). One beneficiary firm stated that even companies that had been able to grow their business following LED’s TA had not done so to their full potential due to lack of financing, particularly for working capital to increase production and meet client demand. The text box below describes the current situation for businesses that struggle with accessing funding. 20 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 A Cashflow Crisis for Businesses Businesses in Lebanon faced multi-faceted crises that started in 2019 and resulted in a very challenging business environment in which to operate. Many businesses permanently closed down in 2020 due to the detrimental effects of the Covid-19 lockdowns, the Beirut Port explosion, and the worsening economic crisis. Those that survived had to continuously deal with arising problems, many of which related to their lack of access to working capital and growth funding. Lebanese firms started facing difficulties in late 2019 due to a severe shortage of hard currency, which also resulted in a steep devaluation of the local currency – which has lost about 95% of its value to date. People’s and businesses’ savings, especially those denominated in U.S. dollars, were locked (and still are) in a defunct banking system through a de-facto and unregulated capital control applied by the banks. Coupled with a lack of access to finance and credit facilities, this problem led many people and businesses to transact outside the banking system after losing trust in it, resorting instead to using (expensive) money transfer agencies. Many non-exporting SMEs rely on cash from daily sales turnover (in LBP or USD) to cover daily expenses, and where fresh USD is needed (e.g., to pay for imports), they either convert the LBP cash they have on hand to USD at a (volatile) parallel market rate, or - where feasible - they charge their clients in USD or LBP at the parallel market rate. While exporting businesses can receive fresh USD through banks or money transfer companies, the fees are relatively high. As a result, many businesses have had to restrict their operations. Below are some relevant quotes from the FGDs with beneficiary firms: “I pay through Western Union. I prefer to pay commissions, as I do not believe in our banking system… I once received an external transaction; it disappeared in the bank... I do not trust our banking system now. Now I am buying a machine from (…), and Western Union is taking a commission of around $1,000.” “We are working miracles daily. I have the support of my husband. He has to count cash daily to see where it has to be paid. The tools that we have today for our work took us back 100 years. It is difficult for us to find payment solutions even though some [clients] are paying in fresh USD, as the bank may not pay you or sometimes not accept the dollars.” “We sold several pieces of equipment to keep cash flowing in. We are paid in fresh USD. We opened a fresh USD account in a bank, but the monthly withdrawal limit affects our work because we have to delay our purchasing.” “We have a problem with cashflow now. We are growing and need more capital, but all prices are increasing (…) It is not fair for our company because we opened up to new markets and [buyers] are asking us for big amounts of products that we simply can’t make.” “What was not ok, is that they (consultants) helped us develop something we want, but we had to do it all on our own afterwards. For me it was a financial problem because as you know we had the [October 2019] revolution then the Beirut port explosion… We lost everything. Our offices were located in front of the port.” FINDING 3.B.2 Many firms complained that they had not been able to implement the TA without additional follow￾on guidance or assistance from the consultant due to the limited size and/or scope of the TA. Other challenges cited related to issues with the TA provider (consultant) and their delivery, and LED’s slow launch of operations in the initial few years, which lead to delays in the roll-out of TA. 21 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 Many of the interviewed firms and consultants spoke about their inability to implement/continue implementing the TA (e.g., through re-hiring the consultant) after LED’s intervention was over, or to even pay their cost-share due to their tight cashflows (as described in detail under EQ1). Over a third of the interviewed firms and some consultants expressed dissatisfaction with specific stages of the TA (design, implementation, or outputs):  The consultant’s theoretical recommendations and guidance could have had a better impact if there had been a follow-up TA to implement them.  The TA’s SOW was limited and short (insufficient LOE), not very clear, and not focused or customized enough. A second TA was not possible either. One firm pointed out though that firms must be realistic in what to expect from the selected business consultant, as the budget is a limiting factor for the scope of the work that can be provided, particularly for innovation companies.  Guidance was needed from LED to determine more precisely what scope should be worked on and prioritized, especially in areas where the firm lacked experience (such as HR), in addition to guidance on selecting the most appropriate consultant for the final scope.  Long, delayed, and incomplete deliverables (due to Covid-19 or the unprofessionalism of business consultants), or TA recommendations that are not very relevant (in the view of the assisted firm).  Some business consultants also noted that a few firms had not been committed to the TA or had not possessed qualified staff to carry the TA through. Some stakeholders also indicated that in some instances the TA provider (consultant/consulting firm) had not been experienced, competent, or professional enough to deliver quality TA, for example: o It was a challenge to find consultants with experience in specific sectors (e.g., logistics), and LED did not permit hiring international consultants. o There was delay in the procurement process of consultants. o The business consultants had limited experience, competency, and/or professionalism. Some LED staff and business consultants noted some challenges that delayed the implementation and delivery of TA, particularly in LED’s early years:  It took LED about a year to sub-contract its partners, and overall operations were slow in the first two years.  The process of selecting and contracting business consultants was slow, and finding the right candidate was hard.  The Request for Quotation (RFQ) published with the TA’s SOW was not detailed or specific enough. As a result, changes up to 30% (in the specifics, not the type of TA) were made after meeting with the client and understanding their needs. “I felt the TA was theoretical – only producing reports (a booklet). I can’t say I did not benefit, but I can’t say I benefited either. I wanted training and (hands-on) support in hiring over a longer period… I feel bad that maybe I didn’t know what to ask for and I don’t have experience in this. It was my first time working with business consultants.” — FGD Respondent “Unrealistic LOE for at least 50% of the projects [I was involved in]. I wanted to have [more] impact on the clients.” — KII Respondent (Business Consultant) 22 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022  There was a high turnover of business advisors, particularly in the first few years.  Not all business advisors, at least initially, had (i) the skills or experience to carry out appropriate diagnostics of the firms that could not identify or prioritize areas for needed assistance themselves or (ii) the technical specialization to determine what type or scope of support was needed by firms that had already identified their primary gap or problem (for example in HR). This firm-assessment process improved later as LED’s understanding of the market improved.  Negotiating (or amending) some grant agreements took substantial time. Some of the firms and business consultants interviewed in the LED-outsourced USEK impact study also mentioned many of the above reasons. A LED senior manager and some beneficiary firms and grantees also mentioned challenges related to LED’s collection of data on jobs and incremental sales, such as which exchange rate to use to value the sales in USD and the sensitivity of commercial sales data (especially in the tech industry) particularly since the end-beneficiaries did not feel obligated toward LED (as no funding was involved). FINDING 3.B.3 The main positive factor, driven by the major contextual changes in Lebanon, was the shift in firms’ mentality (especially small businesses) with regard to seeking external support. This helped them partly mitigate the direct negative impact of their business. The crises in Lebanon did bring about some positive change for businesses, as most KII respondents commented. The positive shift in mentality that occurred as businesses started to recognize the value that an external third party could bring them, especially since they had to deal with immense business challenges and risks, was a major success factor for LED. This was evidenced by the increased demand for LED TA in 2021. The main points raised by stakeholders were:  Pre-crises, the private sector’s mentality/mindset was that mainly large companies worked with business consultants. Small businesses in particular were not acquainted with the concept of TA and were used to mainly receiving grants.  The crises in Lebanon changed this mindset and made companies, particularly SMEs, more aware and appreciative of the value of bringing in external consultants. As firms started to deal with the repercussions of the crises, they recognized their need for support to manage their challenges and in find opportunities in international markets for new sources of revenue in (fresh USD).  Another positive effect of the crises is that they pushed LED to reach out to new companies and achieve more results.  Some sectors, like IT, benefited from the Covid-19 crisis, as their services were more in demand and their staff were able to continue operating through remote work. In addition, most LED grantees who shifted their capacity building activities to online were able to reach more trainees than anticipated. Below are some relevant quotes from the KIIs: “LED played a major role in raising MSMEs’ awareness of their need to change [their way of thinking] from ‘It’s nice to have TA’ to ‘We must have TA’ in order to achieve their market expansion goals.” –LED Staff “Currently, everybody in the country is desperate. They all want to have a glimpse of hope, to be able to get their brands outside the country so that they can start getting fresh dollars. So, 23 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 whoever can help them do that, they are willing to listen to them, even the ones that were never considering leaving their comfort zone.” –Business Consultant “… Firms started believing in the need for consultancy support.” –Business Consultant FINDING 3.B.4 Other success factors – cited by the firms especially – were that (i) the types of TA offered by LED were wide-ranging, (ii) each TA could be tailored to the specific needs of each firm, (iii) both the consultants and the client firms were committed to making the TA successful, and (iv) LED’s staff were professional and committed. Some business advisors highlighted other success factors, such as LED’s systematic and structured MLA that enabled it to reach firms across many sectors, choose the right TA and consultant, and continuously follow up with the business owner. In addition, as expressed by all consultants and many firms, the business advisors’ professional and supportive relationship with the firms was an important driver. Furthermore, by the time there was a surge of interest in LED by firms (in Year 4) LED’s experience and lessons learned meant that its team was able to put together better needs assessments and deal, and approach clients more effectively. LED and USAID interviewees highlighted the advantage of providing a wide range of TA interventions that were tailored to the specific needs of the firms, in addition to other focused support, such as webinars. In addition, Berytech built synergies with its other enterprise development programs, e.g., Agri-Food Innovation Days (AFID), which connected three LED￾assisted firms to international buyers. Communication was also highly active in order to promote the success of TA with assisted firms. LED’s management stressed the importance of (i) the people involved with the Activity, such as the Chemonics, Berytech, and BIAT teams and the business advisors; (ii) the due diligence and matching process of firms with the right consultants; and (iii) the skills and professionalism of the local consultants. LED’s staff and one grantee also pointed to the flexibility introduced in LED’s operations to mitigate the risks arising from the prolonged crises (as described under EQ 3.A). Below are some relevant quotes from the KIIs: “In March 2022, Berytech organized the Agri-Food Innovation Days-AFID in collaboration with Horeca, with a purpose of matching eight companies with international buyers, since Berytech has a cluster for agri-food. During the event, they managed to conduct 320 meetings and have so far closed five deals (support LED clients with other programs), out of which three were from the LED program.” –Business Advisor “The crisis also ignited something in the team. Since it started, everyone gave their all to support the partner companies just to help them stay afloat and sustain their operations.” –LED Staff “LED’s team was incredible. They delivered the maximum for the benefit of LED’s vision and objectives. All were committed… They wanted to help. I have worked with many contractors. The good thing about it is none of the [LED] team members would ask for a commission. In many other programs, there’s always someone who wants a cut.” –LED Grantee On TA delivery, all consultants and many business advisors noted the commitment of most of the assisted firms (with the level of commitment varying depending on each firm) to successfully complete the TA. Where the firms were not committed, the consultants would push them or ask the business advisor or LED to intervene. Over half of the interviewed firms expressed their appreciation for the commitment shown by the engaged business consultants towards successfully delivering the TA. This was exemplified through the consultants’ regular (and sometimes voluntary) follow-up to provide additional support outside the scope of the TA and their flexibility in accommodating the firms’ inability to pay their share. This can be considered a major supporting 24 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 factor to the shift in firms’ mentality and is important for the future sustainability of the relationship between firms and business consultants. Below are some relevant quotes from the FGDs with beneficiary firms: “The results were so satisfying. The trainer even provided additional modules on top of the scope of work.” “The consultant still follows up with us even though she is in France and not getting paid (beyond LED support). I really love this follow-up.” Many business consultants noted that after the onset of the crises some of them forwent their payments from companies that genuinely could not afford to pay or accepted to be paid in LBP at a rate below the market rate (a few said they had been paid in fresh USD for some LED TA). Having said that, it is important to note that many of the interviewed business consultants said they had not entered into any formal agreement with the client to confirm the price and payment conditions (since they relied on their agreement with LED, although that did not cover the client firm’s cost share). Therefore, these individual business consultants did not have legal recourse in cases where the client was able but unwilling to pay, which highlights the risk taken by the business consultants in such instances.20 In addition, this highlights an area where LED could have intervened, for instance by raising awareness around this risk and how to mitigate it or by building the capacity of individual consultants to manage their consultancy business. EQ 3.B CONCLUSIONS  Undoubtedly, the major challenge to LED’s achievement of its KPI targets was the deteriorating socio-economic situation in Lebanon starting in 2019 with the financial, economic, and Covid-19 crises leading to more acute problems for businesses, many of whom had to close temporarily or permanently. One positive effect of the crises though is that they made business owners look at their businesses through a different lens, search for new opportunities, and more seriously consider seeking external support to help them overcome the new (and existing) challenges. As a result of this change in business owners’ mindset, there was a surge in interest in LED’s assistance, propelled by highly active communications conducted by LED to advertise their success stories.  Another positive influence on LED’s success was the commitment of the implementing partners’ staff, the client firms, and most importantly, of the front-line service providers, i.e., the business consultants. Most beneficiary firms valued the regular follow-up by business consultants (many a time voluntary, outside the scope of the TA and post-TA). The business consultants were also highly flexible in accommodating the inability of many (smaller) firms to pay their cost-share of the TA due to liquidity issues – in some cases the consultants even forwent their payments. This bodes well for maintaining this professional relationship and sustaining future business-development-services demand and supply.  Other challenges that the beneficiary firms faced include (i) their inability to implement the TA on their own afterwards without additional follow-on guidance or assistance from the consultant (due to the limited size and/or scope of the TA), and (ii) the unsatisfying quality of the TA and/or TA provider (consultant), which limited their ability to benefit from the TA and maximize its impact. 20 The reality of the Lebanese judicial system is that even if the business consultants did have legal recourse, the red tape and weak functionality of the system render such legal action time-consuming, expensive, and lengthy. 25 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 EQ 3.C: To what extent has the firm-level, buyer-led/market-led approach been cost-effective in meeting LED’s objectives and KPI targets? FINDING 3.C.1 The firm-level MLA has not been as cost-effective as originally projected in meeting LED’s objectives. As a result, LED significantly under-achieved its contractual KPI targets. As shown earlier, LED did not meet the KPI targets defined in its MEL Plan for Years 1-5. Under this section, the evaluation compares the firms’ actual results with those indicated in their MOU with LED, which can be viewed as more realistic given the external context. Note that firms commit to a different metric than what is formally reported to USAID. Firms report the number of new hires (people) and not new FTEs. Comparing the results for the “direct TA” intervention21 (Table 4) shows that while LED-assisted firms did not achieve the committed targets, the % achievement is better than when comparing to LOP targets for two of the KPIs: 62% of committed incremental sales (vs. 46% of LOP target in Table 1), and 83% of committed leveraged investment (vs. 24% of LOP target in Table 1). Note that for the number of jobs created, we are not comparing the same indicator since Table 4 shows the number of people hired while Table 1 compares the number of FTE jobs created. Table 4: LED KPI results - actual vs. committed to by firms KPI Committed Actual Actual vs. committed (%) Jobs - No. of people Direct TA through consultants 7,695 3,940 51% Grants not available 323 n/a Sub-contracts not available 0 n/a 4,263 Incremental sales Direct TA through BCs $ 726,851,280 $ 448,366,339 62% Grants not available $ 59,306 n/a Sub-contracts (initiatives) not available $ - n/a $ 448,425,645 Leveraged Investment Direct TA through BCs $ 58,909,006 $ 48,861,178 83% Grants not available $ 11,624,278 n/a Sub-contracts (initiatives) not available $ - n/a $ 60,485,456 Source: LED Contract, LED KPI Results by firm (as of August 9, 2022), LED KPI Results by Grantee and Sub-contractors; LED Q3-FY5 Report for FTE actual results 21 For grantees and sub-contractors, no outcome targets were set by LED, as there was no monitoring requirement for them after the completion of the grant/sub-contract. 26 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 It is important to note also that as the impact of the TA delivered in 2022 is unlikely to be captured in full before LED closes, the above results do not show a complete picture of the impact of all the delivered TA, especially on job creation where there is a significant time lag between the provision of TA and the creation of the associated jobs, and due to the 12-month nature of the FTE measurement (particularly for continuing FTE reporting). In terms of the number of firms meeting their commitments (for the number of created jobs, incremental sales increase, and leveraged investment), only 15%, 7% and 23% of the firms met (or exceeded) their committed targets, respectively (Table 5). Table 5: Number of firms achieving their committed targets for the three main LED outcome indicators KPI No. of firms that met or exceeded their committed target % of total assisted firms New Jobs 77 15% Incremental sales 37 7% Leveraged Investment 118 23% TOTAL assisted firms 521 Source: LED KPI Results (as of August 9, 2022) With respect to gender-disaggregated reporting, as of Q3 FY5, 34% of the number of jobs created by all interventions were for women, and women accounted for 42% of those engaged in capacity￾building activities. (Data as of July 29, 2022). FINDING 3.C.2 LED’s cost per actual FTE job created through direct assistance to firms, over the life of the project, was $1,827 – just over four times the average cost per projected job of $443. As part of its firm selection criteria, LED applied a jobs-per-enterprise ratio and a cost-effectiveness (cost per job22) ratio in its assessment of candidate firms. While there were no hard rules for the maximum acceptable ratio – as it was revised annually – it was used to assess the reasonableness of the size of LED’s investment (and the cost-share). To measure the cost-effectiveness quantitatively, we calculate the cost per job attributable to the job metric from the direct technical assistance intervention and use three different definitions of “cost:”  Using LED’s total spend (budget) implies a cost per forecasted FTE job created of $2,478 (total budget divided by 9,800 forecast new FTE jobs), whilst cost per actual FTE job was $10,216 (total budget divided by 2,378 achieved new FTE jobs), i.e., just over four times higher than originally projected.  Using “directly attributable” costs only, i.e., those incurred by sub-contractors (Berytech, BIAT, consultancy firms, and other fixed-price sub-contractor organizations) and short-term technical assistance (individual business consultants), the cost per forecasted (FTE) job is 22 The cost-per-committed-job ratio is calculated based on the direct costs of LED’s assistance (i.e., consultants’ fees and any operational expenses, such as travel) and the number of jobs committed to by the firm. 27 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 $700 (using the $6,856,036 budget line). On the other hand, the “direct” cost per actual FTE job is $3,049 using ($7,249,384 actual spend, i.e., Including LED’s re-allocated savings)23. 24  Using delivery costs, defined as LED’s actual cost-share amount (comprising consultants’ fees plus other TA-related expenses incurred by consultants) – which is the method used by LED and USAID to assess firm applications and monitor cost-effectiveness25 – we have $443 cost per actual FTE vs. $1,827 per forecasted job (based on a total LED investment for all TA of $4,342,760).26 Of course, the value-for-money judgement cannot be based only on the LED project – as these ratios need to be benchmarked against other projects that are in some way comparable. Such data are not easily available from other similar job creation projects and programs. However, a World Bank blog27 has stated that active labor-market-development programs (such as training, job search assistance, wage subsidies, or public works), which connect workers to existing jobs, usually have a cost per job created range of $300-3,000 per job. It also notes that this metric’s value would be higher for promoting private investments to create new jobs in the formal sector. The blog explains that creating new jobs is not cheap, particularly through a sectoral investment that creates a new sector. In the case of LED, the interventions supported the creation of new jobs through the expansion of existing businesses, and thus the cost per job should be higher than that for an active labor-market￾development program but lower than that for projects promoting the creation of new firms, which, in Tunisia for example, is an average of $30,000 per job created following a sectoral investment.28 Therefore, in theory, the cost per job should be between $3,000 and $30,000. This suggests that the original projected cost per job of $2,478 based on total cost was quite low and therefore unviable for Lebanon at the time of LED’s project design (especially as Lebanon, pre-crises, was considered an upper-middle income country). It is even more unviable now in the current context of economic challenges (including a decreased supply of skilled labor force due to increased emigration), hyperinflation, and financial risks. Thus, while it might appear that LED’s approach was not cost-effective, as its ex-post cost-per-job ratio exceeded its projections, this conclusion is inaccurate since it should factor in the economic costs and risks resulting from the major context changes in the country. More importantly, without contrasting it with comparable projects in similar contexts, no conclusive judgment can be made on the approach’s true cost-effectiveness or overall value for money. 23 The difference is less pronounced when using the number of hires/jobs created from the direct technical assistance intervention: $891 per projected job (using committed no. of jobs) vs. $1,840 actual. 24 Note that an amount of $241,455 was spent on four initiatives implemented by sub-contractors in Year 5, for which no jobs had been created by the date of receiving the KPI results from LED. However, their effect on the ratio calculation would be negligible. 25 The cost-per-job ratio that is included in LED’s assessment of the firm’s application is calculated based on the expected direct costs of LED’s assistance (i.e., LED’s share of the consultants’ fees and any operational expenses, such as travel, per diem, etc.) and the number of jobs committed to by the firm. 26 The difference is less pronounced when using the number of hires/jobs created: $564 per projected job vs. $1,102 actual (which is just above the average cap of $1,000 cost per committed job set by LED). 27 How much does it cost to create a job? (worldbank.org), David Robalino, World Bank, February 15, 2018 28 ibid 28 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 FINDING 3.C.3 With respect to the other two outcome KPIs, LED-assisted firms generated incremental sales that were 9.7 times its investment, while for leveraged investments by firms the ratio to LED’s investment was 1.1 times. When compared to LED’s investment (i.e., cost-share contribution), the other KPI results are positive – for every $1 of TA invested, firms generated $9.70 (on average) as incremental sales (we refer to it as “incremental sales multiple”) and invested $1.10 (on average) into their business (excluding their cost-share) (we refer to it as “leveraged investment multiple”). However, for both KPIs, the actual multiple achieved was much less than originally implied via the targets. See Table 6 below. It is worth reminding the reader here that these results were achieved by 36% and 47% of the assisted firms, respectively, as shown in Table 3 at the beginning of section EQ3. Table 6: Incremental Sales and Leveraged Investment KPI actual results vs. targets (in terms of USD and implied multiple). Incremental Sales (Value and Multiple) Leveraged Investment (Value and Multiple) LED investment in TA Target Actual* Target Actual* $ 4,342,760 $ 245,000,000 $ 42,244,516 $ 49,000,000 $ 4,603,639 56.4x 9.7x 11.3x 1.1x * As done in previous calculations in this report, the incremental sales and leveraged investments were adjusted by the estimated average LBP/USD exchange rate over the five years of LBP/USD 16,000. Incremental sales multiple = incremental sales value divided by LED investment value. Leveraged investment multiple = leveraged investment value divided by LED investment value. Source: LED KPI Results by Firms (as of August 9, 2022); LED Cost-share data; LED Contract FINDING 3.C.4 Factors that contributed positively to LED’s cost-effectiveness include the use of local ecosystem partners’ networks and the flexibility in the firms’ selection process (which helped to improve the efficiency of outreach and approvals), as well as the competitive selection of business consultants. LED also considers the savings re-allocated into TA as contributing to the project’s cost￾effectiveness. Our KIIs highlighted a number of internal factors that would have contributed, positively or negatively, to the cost-effectiveness of LED’s interventions by influencing the efficiency and effectiveness of LED’s processes/operations. Cost-effectiveness of the selection process of beneficiary firms  Beneficiary firms were selected through LED’s local partners’ (Berytech and BIAT) business advisors, who used their own databases, connections, and networks for outreach, particularly as the focus was mainly on reaching eligible micro and small businesses.  However, as noted by a LED senior manager, the delay in sub-contracting the local partners 29 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 (contracted seven months into the first financial year29) also led to a delay in hiring business advisors and enterprise engagement. Cost-effectiveness of the selection process of business consultants LED procured the consultants on a competitive tender basis by posting all RFQs on Daleel-Madani, a local platform for development jobs and project tenders. LED involved the client in the technical evaluation and had a special committee for doing the final evaluation of the technical and financial proposals. Although the due-diligence and vetting procedures were slow to kick off, the process became more efficient two years into the project, particularly as the consultants’ roster grew and due diligence and vetting became clearer and quicker. Some interviewed firms complained that the evaluation of the shortlisted business consultants had taken up a lot of their time and could have been organized more efficiently by LED. Budget savings LED used all the savings it was able to make (mainly from saved LOEs for business advisors who were hired later than planned or from vacancies that remained unfilled for a while) to support as many businesses as possible and invest in more TA. FINDING 3.C.5 A significant proportion of consultants struggled to price their offers and compete with the lower prices expected by LED after the crises hit. The majority of them noted that LED’s rate had been below the market rate, and almost half said they were dissatisfied with the expected lower rates. All business consultants and some business advisors indicated that initially the TA value had been $10,000 - $20,000 per TA before the crises and that LED had reduced it to an average of $8,000 - $10,000 afterwards. (See table 7 showing the average total30 TA size per financial year – from Year 1 to Year 5 there was a 40% reduction in the average size.) According to the KIIs with LED staff, this was due to the devaluation of the local currency and the reduction/capping of individual consultants’ daily rates (benchmarked to the local market by LED). LED was able to lower the rates because it started to pay the business consultants in fresh USD (i.e., physical USD bills). USAID’s guidance was to set an average rate of $250 in fresh USD for individual consultants (compared to $350 previously). However, not everyone was happy with the reduction in their USD daily rate. While ten consultants who took part in the KIIs said they were satisfied with this lower daily rate, six others said they were not. (As mentioned previously, many business consultants were paid the client’s cost-share at 29 LED Year 1 Annual Report 30 Total TA value meaning the total of LED and client’s cost share. Year Avg. Total TA size (USD) FY17/18 14,707 FY18/19 15,817 FY19/20 10,873 FY20/21 9,033 FY21/22 8,783 Overall 10,936 Table 7: Average total technical assistance value 30 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 below market rate or not paid at all in some cases.) To some consultants, this was compensated for by the volume of projects they got from LED. They were motivated by their desire to build their credentials and long-term relationships with the clients and by the fresh USD pay they were receiving from LED. This is why they often went willingly below their standard rate (by 10-50%) or reduced the LOE (following negotiations with LED) to win projects. A few consulting firms said they had to compete with individual consultants in terms of pricing. In addition, some consultants with senior experience felt they were being priced out by more junior consultants, particularly for certain types of TA, which forced them to undercut their standard rate and/or reduce their LOE significantly. Several consultants contrasted LED’s daily rates with those offered by other donors (e.g., the European Bank for Reconstruction and Development (EBRD) at $500) and the private sector $300-700 in fresh USD. Below are some relevant quotes from the KIIs with business consultants: “Actually, only LED lowered the rate of consultants by at least 50% and sometimes even more. Other projects kept paying the same rate and in fresh dollars. But still, the price can be considered within the market rate during the crisis. However, the prices now [in the market] have stabilized and are getting better.” “I used to be paid two or three times more, even during the crisis, but because LED gave me many projects and had a competitive approach, I had to accept a lower price. Getting paid in fresh dollars during the crisis made me take on this work.” “I am satisfied, but it should be higher. I am earning $200 with LED (was $250 previously) in a more competitive market and with a duration that is relatively short.” EQ 3.C CONCLUSIONS  Based on the achieved results, which are much below the KPI targets, LED’s approach does not appear to have been as cost-effective as originally projected. In addition, the reported monetary results (sales and investments) should be assessed with caution as the reported USD amount does not reflect the market value given the local currency situation and the financial and economic risks in Lebanon.  Using LED’s cost-per-job metric definition, LED’s cost per actual FTE job created through direct assistance to firms, over the life of the project, was $1,827 – about four times the average cost per projected job of $443. The most likely explanation for this is that the original cost per forecasted job was set too low in the first place, although this under￾estimation is partly offset by the fall in the real USD value of local goods and services following the devaluation of the local currency (which LED tried to take advantage of by reducing the business consultants’ rates and paying in fresh USD).  Furthermore, even after adjusting for the market exchange rate, any conclusion on the cost￾effectiveness of LED’s approach will be inaccurate, as it does not factor in the economic costs resulting from the major context changes in the country. More importantly, without benchmarking the metric with comparable projects (i.e., in similar contexts too), no final judgement can be made on the cost-effectiveness of the approach.  Overall, for firms that were able to increase their sales as a result of LED’s TA, the total incremental sales were 9.7 times LED’s investment. For firms that leveraged investment, the invested amount was just above the value of LED’s investment. This demonstrates the attribution to LED of the positive effects on these assisted firms’ sales and the latter’s willingness to invest further into their business to maximize the impact of the TA.  Where possible, LED leveraged its local partners and implemented measures and budget savings to improve the cost-effectiveness of its operations.  One dissatisfaction expressed by a significant proportion of interviewed consultants related 31 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 to the reduced TA pricing that LED expected after capping its USD fee rates, which were considered by most consultants as below market (including other donors’) rates. These reduced rates could create a challenge for business consultants wanting to price their future services to MSMEs at market/commercial rates. EVALUATION QUESTION 4 (EFFECTIVENESS) EQ 4: Which LED interventions and technical support have been the most and/or least beneficial, productive, and cost-effective? FINDING 4.1 In terms of beneficial interventions, most firms and many business consultants benefited from the direct-TA interventions as mentioned in Finding 2.1. Other non-TA beneficial and productive interventions mentioned by some stakeholders include the webinars and the grant-funded capacity￾building activities, in which results exceeded targets. In addition to the direct TA’s benefits reaped by assisted firms (as described under Finding 2.1), all grantees and some LED staff talked about the achievements of their other (non-TA) initiatives, such as the webinars, the online capacity-development trainings, and other support (mentorship) that they offered to groups of consultants, startups, and/or scale-up businesses. The grantees achieved or exceeded targets and are now able to scale these initiatives. Additionally, some grantees made significant cost contributions (around 50%) toward the grant-funded project. In the Self-Evaluation Survey for Client Enterprises, all respondents noted at least one of the following benefits of the TA provided by LED: access to new markets, access to professional consultants, improved food safety and hygiene, better management/structure, improved quality/productivity/sales new products or improved product development process, preparation for and more confidence to export/certification, etc. Five business consultants who took part in the KIIs said that they had gained new skills, improved existing skills (e.g., business development skills), or acquired new knowledge as a result of working with new clients across, for example, new sectors. Importantly, they mentioned that a valuable experience for them had been dealing with difficult situations and working with struggling companies. Below are some relevant quotes from the KIIs: “Twelve companies committed to the program, while the initial target in the grant was six. The target number of beneficiaries in the agreement was 120, yet 198 completed the program, many of whom benefited from more than one training. The number of newly-employed people set in the agreement was 70, yet they were able to secure 109 new full-time jobs after finishing the program.” –LED Grantee “There is always learning when working with any type of company, especially when dealing with a new sector. One example is working with a cosmetics company, which I had no previous experience with.” –Business Consultant “We did a cost-sharing of 52.5% (…) vs. 47.5% by LED. For staff, we contributed a percentage of our salaries.” –LED Grantee FINDING 4.2 According to the interviewed business advisors and consultants, the most productive and cost￾effective type of TA was the one that enabled firms to export and access new international markets. In terms of achieving LED’s KPI targets, the most cost-effective TA was HR & Restructuring followed by Knowledge Management. The last on that list was Supply Chain Management. 32 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 TA is deemed productive if it leads to increased sales and/or lower production costs for a firm. There was consensus among many of the interviewed business advisors and business consultants that the most productive interventions were in export readiness, ICT and digital transformation, access to markets, franchising, digital marketing, research and development, product development (especially for food industries), quality management/certification, and salesforce development. Cost-effectiveness of LED’s TA As seen in Table 8 below, the most effective type of TA (measured by the TA’s contribution to each total KPI result) can be inferred as follows:  In terms of the highest percentage of new jobs created, it was mostly through Quality Management, followed by Franchise Development and Production & Process Management.  In terms of the highest incremental sales achieved, it was mostly through Sales Management followed by Quality Management TA  In terms of leveraged investment, it was mostly through HR Training & Restructuring TA. Table 8: Percent contribution of each type of TA, by KPI. Data source: LED KPI Results by Firms (as of August 9, 2022) and LED Cost-share data On the cost side, we see somewhat different indications of the most cost-effective type of TA. Table 9 shows the calculated cost-effectiveness ratios using the formulas shown below it. Type of TA No. of jobs created (%) Incremental Sales (%) Leveraged investment (%) Access to finance / financial management 2% 3% 3% Access to market 5% 2% 1% Franchise Development 15% 1% 4% HR and Restructuring 11% 5% 60% Knowledge management 9% 13% 2% Marketing 1% 1% 0% Product design or development 5% 2% 2% Production & Process Management 15% 3% 9% Quality management 27% 19% 16% Sales management 10% 52% 2% Supply chain management 1% 0% 0% Other 0% 0% 0% TOTAL 100% 100% 100% “LED pushed the majority of companies to [get a] certification and implement the system to improve production... so they would implement the improvements and be ready when there was a need to have a certificate.” — KII Respondent (Business Consultant) 33 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 Table 9: Cost-effectiveness ratios, by KPI and type of TA Definitions of table headings: Cost per job = LED investment value divided by no. of jobs created. Incremental sales multiple = incremental sales value divided by LED investment value. Leveraged investment multiple = leveraged investment value divided by LED investment value. Data Source: LED KPI Results by Firms (as of 9th August 2022) and LED Cost-share data * Since LED’s investment was in (fresh) USD, and in order to have comparable and more accurate ratios, where monetary amounts are involved, we have adjusted the reported sales and investment values to reflect more closely actual market exchange rates (rather than the LBP/ USD 1,507.5 official rate) – as was done in previous analysis in this report – using an average parallel market exchange rate of LBP/USD 16,000. The green-highlighted cells in the table above show the most successful results for each ratio, while the red ones indicate the least successful, using these criteria: Color coding Cost per job Incremental Sales Multiple Leveraged Investments Multiple Green =5.0x >=1.0 Red >US$ 2000 <1.0x <1.0 We note the following from Table 9:  ‘Cost per job’ was lowest for Production & Process Management ($613), followed by HR and Restructuring ($777), Quality Management ($891), and Knowledge Management ($942). It was highest for Marketing ($3604), followed by Access to Market ($2850).  The incremental sales multiple was highest for Sales/Salesforce Development (42x), followed by Knowledge Management (17x), Quality Management (8x) and HR and Restructuring (6x). It was lowest for Supply Chain Management (0.7x).  Leveraged investment multiple was highest for HR and Restructuring (8x) and lowest for Supply Chain Management (0.04x). Thus, the analysis indicates that cost-effectiveness is highest for HR and Restructuring, as it was positive across all three cost-effectiveness ratios that were calculated, followed by Knowledge Management and Quality Management, both of which were positive across the first two ratios. Interestingly, both HR & Restructuring and Knowledge Management were perceived by several business advisors as being cost-ineffective. Moreover, two of the interviewed firms noted that the HR & Restructuring TA they had received was not really that effective. Type of TA Cost per job (USD) Incremental Sales multiple* Leveraged investment multiple* Access to finance / financial management 1,502 9.19x 1.06x Access to market 2,850 1.29x 0.11x Franchise Development 994 0.91x 0.34x HR and Restructuring 777 6.13x 8.19x Knowledge management 942 16.99x 0.26x Marketing 3,604 3.30x 0.12x Product design or development 2,132 2.03x 0.24x Production & Process Management 613 3.43x 1.14x Quality management 891 8.14x 0.77x Sales management 1,385 42.41x 0.19x Supply chain management 2,080 0.67x 0.04x Other 1,694 0.00x 0.00x TOTAL 1,102 9.73x 1.06x 34 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 Note that when asked about cost-effective interventions, KII respondents interpreted this as those types of TA that created less expensive, lower-salaried jobs. For example, a business advisor noted that knowledge management was not so cost-effective, as the salaries of IT specialists in Lebanon are generally higher than salaries in other professions. Below are some relevant quotes from the KIIs and FGDs: “ISO 22000 created 82 new employments at (company name).” –Business Advisor “Through product-development support to the bakery, we were able to extend shelf-life without (…) changing the packaging or paying the $15,000 that another consultant had requested. So, through product reformulation and without paying extra expenses, five products’ shelf-lives were extended.” –Business Consultant “The sales for sure increased because of LED, due to the expansion we made. The FSSC [certification] allowed to reach new clients faster.” –FGD Respondent “LED helped us digitize this system through TA. The person who created the system did not start it from scratch. He already had a blueprint to work with.” –FGD Respondent “When the program ended, we felt like we had accomplished a very big milestone and achieved what we wanted. But later on, when we hired an HR [person]… our perspective changed a bit, because when the HR [person] reviewed the company and the outcome of the consultancy, we found that a lot of the stuff that was done was kind of secondary or unnecessary and that a lot of improvements were not done the right way.” –FGD Respondent FINDING 4.3 The most cost-effective firm size in LED’s interventions was large firms, and this is normally the case since big companies are in a better position to create more jobs, improve sales, and invest significant amounts in their business. The most cost-effective sectors were oil and gas, tourism (particularly food services), power and water, and healthcare, while the least cost-effective sector was recycling and environment. In terms of regions, the Beqaa was the most cost-effective in LED’s interventions, followed by Baalbek and Nabatiyeh. Mount Lebanon had the highest sales multiple but a medium level for cost per job. The least cost-effective region was South Lebanon. The analysis of LED interventions’ cost-effectiveness with other variables of firm characteristics shows the following:  Firm size: Not surprisingly, large companies were the most effective, followed by medium, small, then micro firms (the latter two were unable to leverage investments). Table 10: Cost-effectiveness ratios, by size of assisted firms Cost per job = LED investment value divided by no. of jobs created. Incremental sales multiple = incremental sales value divided by LED investment value. Leveraged investment multiple = leveraged investment value divided by LED investment value. Data Source: LED KPI Results by Firms (as of 9th August 2022) and LED Cost-share data.  Sector: The most cost-effective sectors were oil and gas (driven by a large firm), tourism (particularly food services), power and water (one firm), and healthcare. KII respondents Cost per job (USD) Incremental Sales multiple Leveraged investment multiple Large, 100+ FTE 363 51.8x 3.9x Medium, 51-100 FTE 853 9.6x 2.5x Small, 11-50 FTE 1,645 3.8x 0.3x Micro, 1-10 FTE 2,346 1.0x 0.4x TOTAL 1,102 9.7x 1.1x 35 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 considered agri-food/food manufacturing and ICT as the most productive sectors (since they were high-potential export sectors). Table 11: Cost-effectiveness ratios, by sector of assisted firms Cost per job = LED investment value divided by no. of jobs created. Incremental sales multiple = incremental sales value divided by LED investment value. Leveraged investment multiple = leveraged investment value divided by LED investment value. Data Source: LED KPI Results by Firms (as of 9th August 2022) and LED Cost-share data.  Region: The Beqaa leads the way with a low cost per job and good sales and investment multiples, followed by Baalbek-Hermel and Nabatiyeh (due to a few large firms, although they did not generate incremental sales). Mount Lebanon generated the highest sales multiple but also a medium level for cost per job. The least cost-effective was South Lebanon due to the low number of jobs created there. Table 12: Cost-effectiveness ratios, by location (governorate) of assisted firms Cost per job = LED investment value divided by no. of jobs created. Incremental sales multiple = incremental sales value divided by LED investment value. Leveraged investment multiple = leveraged investment value divided by LED investment value. Data Source: LED KPI Results by Firms (as of 9th August 2022) and LED Cost-share data. EQ 4 CONCLUSIONS  Beneficiary firms and other market actors perceived all LED interventions (direct and indirect TA and capacity building) to be beneficial.  LED’s business advisors and consultants perceived the TA that enabled firms to export their products and access international markets as the most productive and cost-effective TA. Industry Cost per job (USD) Incremental Sales multiple Leveraged investment multiple Agriculture 1,637 2.4x 0.6x Education and Training 1,304 0.6x 0.3x Healthcare 765 0.7x 15.2x Industry 1,174 6.9x 1.1x Knowledge Economy 1,138 5.8x 0.2x Oil & Gas 667 49.1x 0.1x Other 1,417 5.2x 0.2x Power and Water 563 1.3x 0.2x Real Estate and Construction 1,082 5.0x 0.1x Recycling & Environment 13,425 0.1x 0.0x Renewable Energy 1,553 6.2x 0.5x Services 1,065 2.9x 0.8x Tourism 760 4.7x 0.5x Trade 1,683 118.9x 0.5x Transport & Logistics 1,543 27.3x 0.1x 9.7x 1.1x 1,102 Cost per job (USD) Incremental Sales multiple Leveraged investment multiple Akkar 1,810 0.0x 1.0x Baalbek-Hermel 352 3.5x 0.1x Beirut 1,207 9.3x 0.4x Beqaa 686 3.9x 2.8x Mount Lebanon 1,298 12.9x 1.0x Nabatieh 88 0.0x 1.2x North Lebanon 1,372 9.3x 0.5x South Lebanon 4,182 0.5x 0.2x Grand Total 1,102 9.7x 1.1x 36 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022  Quantitative analysis shows slightly different results when assessing cost-effectiveness in relation to all three outcome KPIs (jobs, sales, and investment). The most cost-effective TA was HR & Restructuring (best results across all three cost-effectiveness ratios), followed by Production & Process Management, then Knowledge Management and Quality Management. The least cost-effective was Supply Chain Management.  According to various key informants, knowledge management, access to markets, franchising, and digital marketing were the most cost-effective type of TA. However, access to markets and digital marketing had very high cost-per-job ratios, which suggests that the definition of “cost-effective” for these stakeholders was not always linked to quantitative indicators but rather to the assumption that all TA that supported export was cost￾effective. These results were largely influenced by the context in which the businesses operated and should therefore not be considered as indicative of potential results for future programs.  The most cost-effective firm size in LED’s interventions was large firms (unsurprisingly), as big companies can create more jobs, generate more sales, and invest significant amounts in their business. Micro and small businesses trail far behind the larger firms, most likely because they were affected more heavily by the crises.  The most cost-effective sectors were the ones that received smaller investments, such as oil and gas and power and water, as they created new jobs, in addition to tourism (particularly food services) and healthcare. Trade and transport and logistics were mainly cost-effective in terms of incremental sales. The least cost-effective sector was recycling and environment, which is a possible indication of the under-development of this sector in Lebanon.  In terms of regions, the Beqaa was the most cost-effective, followed by Baalbek-Hermel and Nabatiyeh, although Mount Lebanon and the North had the highest sales multiple and showed potential for improving cost per job. The least cost-effective region in LED’s interventions was South Lebanon, across all three ratios, likely because of the high prevalence of informal enterprises there. EVALUATION QUESTION 5 (SUSTAINABILITY, SCALABILITY, ADDITIONALITY) EQ 5: To what extent are the successes of LED likely to be sustainable and scalable and demonstrate additionality? EQ 5.A: Which LED interventions and results show the most promise of being sustained, and why? FINDING 5.A.1 Of LED’s different results, those achieved through direct TA and grants show the most promise of being sustained. The TA approach reshaped the beneficiary firms’ mindset regarding business growth and encouraged them to explore new growth avenues, target new markets and buyers, and implement better practices by engaging competent business consultants. Similarly, for grantees that piloted market-led support initiatives, the success of their pilot projects enabled them to attract further funding for scale-up. As described under EQ 3.a, the stakeholders highlighted the positive impact that LED’s interventions had had on the firms’ mindset, increasing their awareness about how external experts can assist them in improving their business effectively and more efficiently than if they were to do it on their own. Specifically, business advisors noted that LED’s TA had greatly benefited businesses by building a stable foundation for export, creating relationships with international buyers, and introducing the idea of partnering with external consultants. All of these changes and factors are enablers for sustaining the results of the TA. They also align with what most of the business consultants mentioned, as many reported that they remained in contact with the companies through informal follow-ups. This is also a positive sign of the potential sustainability of the TA’s effects and is supported by what the assisted firms mentioned too. 37 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 According to LED’s survey, 34% of the firms said they had already re-hired the same consultant and are paying them from their own funds. The same data show that the majority of large firms (54%) and a significant percentage of small and medium-sized firms (39% and 32%, respectively) have re￾hired the same consultants with whom they had previously worked. However, only 24% of micro￾enterprise respondents said that they had done so, which could be related to an affordability issue. (See Finding 5.a.2 below) Additionally, beneficiary firms and other stakeholders noted that LED had:  Assisted them in assessing and responding to market and buyer needs, in addition to being aware of their strengths and weaknesses, which enabled them to continue monitoring the TA results and identifying evolving business needs to address in the future.  Built their capacity and experience in selecting and procuring business-consulting services and the right consultants.  Assisted them in strengthening their position in the local and/or international markets and earning fresh USD, thereby helping to sustain their business and retain their staff.  Improved their resilience and capacity to continue dealing with the current economic and financial risks and manage and adapt to future ones quickly. Furthermore, some firms invested their own funds (i.e., leveraged investment) using grants from other donors or family and friends to implement the TA they received from LED, e.g., to purchase new machinery, renovate their factories, etc.—which also supports the sustainability of the TA’s results. All four grantees confirmed the high prospects of scaling up their pilot initiatives following their successful interventions with firms at the startup and scale-up stages of development. An important aspect of LED, built into the project’s design from the beginning, is that by working with and through local suppliers and market actors (Berytech, and BIAT—but also other partners), it contributed to building their capacity and assets so that they can continue offering business￾development services to MSMEs. LED also helped build the capacity of local business consultants. This improved capacity ensures that firm-level, market-led business development can be continued, after LED closes, by those that have the capacity to do so (as can be seen already from Berytech’s recent involvement with a USAID project and other donor-funded enterprise development projects). Below are some relevant quotes from the KIIs and FGDs: “This [LED TA] is sustainable because now the companies have the foundation to continue exporting and also because LED helped them establish links with buyers abroad who can sustain their operations in Lebanon.” –KII Respondent (Business Advisor) “LED has contributed immensely to changing the way beneficiaries look at consultancy. Within the first month and a half of the technical assistance, they (i.e., beneficiary firms) saw the difference, and this is thanks to the LED project.” –KII Respondent (Business Consultant) “We are now better organized and have better procedures. The quality department can follow up on everything to help us better organize our work with the clients.” –FGD Respondent FINDING 5.A.2 The challenges to sustaining the effects of LED’s interventions after the project concludes are mostly linked to Lebanon’s economic crisis and beneficiary firms’ limited ability to finance and continue implementing the TA without external assistance. 38 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 Almost all firms that responded to LED’s survey said that the TA’s benefits were sustainable, as they had acquired the necessary capacities to ensure that. This is likely to hold true for those firms that are exporting and/or were able to improve their productivity. The primary challenge to the sustainability of the LED TA’s impact is the continuation (or worsening) of the economic crisis and its negative effects on beneficiary firms’ operations (e.g., high turnover of skilled staff), financial sustainability, ability to invest in their business, and hire consultants again in the future. Some business advisors stated that the economic crisis greatly reduced the ability of MSEs to hire technical consultants in the absence of external funds. This observation was corroborated by one USAID/Lebanon staff and several business consultants who said that many firms would likely need third-party support. Another challenge (as described under Finding 3.b.2) is that many firms are not technically capable of implementing the TA on their own and will need external support. For example, some beneficiaries complained that there had been no follow-up by LED to ensure the sustainability of their intervention, given the limited timeframe, budget, and SOW of the TA. They said they would have benefited greatly from regular follow-up (e.g., over a period of six months after the TA) by the business consultant to support and guide them in implementing the acquired knowledge and recommendations. Moreover, a small number of firms were not satisfied with the TA consultants that were assigned to them and said they would not consider re-hiring them in the future. On another note, one key informant mentioned the risk of other donors continuing to provide free business development and consulting services, thereby distorting the business consultancy market’s open competitiveness and the transparency and efficiency that LED has helped to instill in the market. A LED senior manager raised a similar point about the importance of other donors building on the success of LED’s approach (and learnings) rather than developing a new approach from scratch. Below are a couple of relevant quotes from the KIIs and FGDs: “It’s a bit tough without LED’s support since the cost-share was the main attractive point to working with the consultant, especially after the crisis hit, and the MSMEs currently prioritize their overhead expenses.” –KII Respondent (Business Advisor) “We asked [LED] for a minimum of one hour of help or training just to teach us. [By comparison] We make agreements [as a contractor] with the UN, and they do not agree to work with us unless there is (…) at least three to six months of follow-up [by our company after completing the work].” –FGD Respondent FINDING 5.A.3 A risk to the sustainability of the TA after LED concludes is the business consultants’ inability to offer their services to MSMEs at a lower rate compared to the market, in addition to smaller firms’ inability to hire consultants at the market rate. As mentioned under Finding 3.c.5, many consultants felt forced to accept lower rates than the market’s to win work from LED, especially as they were trying to sustain themselves in times of hardship. It was an exceptional situation for many of them, but it is not sustainable. Therefore, this raises a question and an uncertainty about whether the market linkages created between business consultants and MSMEs (at least MSEs) are sustainable on the whole, also given the inability of small businesses to afford the market rates of business consultants after LED concludes. A few consultants “Yes, if they (the firms) are willing to pay a decent price, I don’t mind [assisting them]. But due to the current situation in the country, I prefer to get paid through donor- funded projects, as companies are struggling and cannot afford TA fees.” — KII Respondent (Business Consultant) 39 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 said they had been re-hired for follow-up by some of their LED clients at their standard rate in fresh USD, but these were mainly larger firms. EQ 5.A CONCLUSIONS  LED’s key results, including jobs created and incremental sales achieved, are likely to be sustained after the project concludes. Sustainability also depends on the type of TA delivered. Evidence shows that the most implemented type of TA was strategic (e.g., access to market, new product development) or operational (e.g., quality management, sales training) leading to changes that enable the firms to reach and build connections with buyers in new markets more strongly and confidently. The TA (and also the webinars) supported the beneficiary firms in increasing their exports or improving their export-readiness and/or production capacity.  LED’s interventions also had some milder effects on the assisted firms, such as improved resilience, increased ability to identify business issues and gaps, increased capacity and confidence to hire business consultants in the future, etc., which are also likely to be sustained beyond LED’s lifetime.  The most significant enabler for the sustainability of LED’s impact on businesses is the change in the mindset of business owners and managers who now recognize (through their own experience) the importance of looking at new and more productive ways (such as seeking external advice) to run and grow their business.  There is uncertainty regarding the sustainability of the market linkages established through LED between business consultants and MSEs, particularly without third-party support (financial and non-financial), notwithstanding the severe impact of the crises on the businesses’ financial situation, making them unable to afford external support. EQ 5.B: Which particular aspects of the LED Activity have resulted in the growing replication of technical assistance at declining marginal costs? Has there been any particular aspect of the Activity that ended up being more “one-off,” i.e., not scalable? FINDING 5.B.1 Analysis suggests that LED has replicated the TA at declining marginal costs, as it ramped up the number of closed deals after the first year. Efficiencies in its firm-selection process and the potential for standardizing certain types of TA largely contributed to declining marginal costs. The evaluation team conjectures, based on the evidence showcased under EQ 3.C (cost￾effectiveness), that the scalability of LED’s interventions at declining marginal costs would have likely been achieved after Year 1 once the business advisors’ productivity had improved and the firms’ selection process had become more efficient. In addition, the team can surmise that certain types of TA that are quite standardized, such as those designed to help the firms acquire international certifications (ISO, FSSC, etc.) and those that offer franchise-development support for enterprises in the same sector, would have also benefited from “economies of scale,” at least during the TA design phase (RFQ, consultant selection, etc.) and possibly in the implementation phase (to some extent). Similarly, a LED senior manager noted that the business consultants’ use of standardized tools and material might have contributed to TA implementation efficiencies (although this would probably have not trickled through to the consultants’ fee, i.e., LED’s cost). Given that all TA interventions are tailored to the firm in question, it is hard to precisely identify patterns of declining marginal costs when looking at the average total cost (or LED plus client’s costs) per TA (i.e., total cost of TA / no. of TA for each type of TA). Table 13 below shows a tendency for falling average values overall, for all TA, from 2020 (i.e., as efficiencies kicked in) and for specific TA areas, such as franchise development, HR and restructuring, quality management, and sales training from the first few years. 40 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 Table 13: Percentage change in average size/value of TA, by type of TA Source: LED Assisted Firms data; LED cost share data A factor that makes this analysis incomplete is that the average size of TA overall was reduced by LED during the crises. As such, some of this reduction in average size is unrelated to efficiencies. However, a falling trend over several years does point to efficiencies in the TA implementation process. To quantitatively measure the magnitude of the declining marginal costs would require a separate costing exercise outside this evaluation’s scope. FINDING 5.B.2 The future replicability of LED’s interventions is possible due to (i) the positive experience and learnings acquired by the assisted firms, which encourage them to re-hire business consultants, and (ii) the scalable design of the direct assistance projects/initiatives implemented through many grantees and sub-contractors. In addition, LED has indirectly helped to build the scale, capacity, and commercial viability of local business consultants and consultancy firms in Lebanon, thereby increasing supply and competition. The replicability of LED’s interventions in the future, after LED concludes, is enabled by some of the factors discussed previously in this report (such as the positive experience of the assisted firms with the consultants and the TA, the efficiencies gained by business consultants, etc.) in addition to the replicable design of the LED grant and sub-contractor initiatives. As stated previously, LED’s effect on the business consultancy market was unintended but highly impactful, given the large number of deals (539) implemented over the last five years, with the majority occurring during challenging times in the market for business consultants. By increasing the competitiveness of local business consultants (through increased supply of and demand for consultancy services, and indirectly through supporting relevant partners’ activities), LED’s interventions have contributed to widening the market, which is also beneficial to other donor￾funded programs looking for local business experts. Furthermore, LED allowed consultants to (i) build long-term relationships with clients, especially in a new segment (MSMEs), (ii) boost their profile and visibility in the market, and (iii) strengthen their credentials through new knowledge and experience. These are enabling factors for the sustainability of the business consultancy market. It should be noted, though, that some interviewed business consultants said the abundance of freshly￾graduated, unexperienced consultants could be a challenge to the scalability of quality business consultancy services. Below are a couple of relevant quotes from the KIIs: “I think the use of consultancy is scalable to a large extent because it could be recurring. Companies will always need external support and good consultants, not just in technical aspects, but also in managerial aspects, in export promotion and certification, and so on. So, there is a lot of potential in scalability.” –KII Respondent (Business Advisor) Change in avg. size per TA FY18/19 FY19/20 FY20/21 FY21/22 Financial management / Access to finance 38% -10% Access to market 39% -26% 11% Franchise Development -34% -8% -54% -7% HR and Restructuring 15% -41% -25% -5% Knowledge management -84% 75% -9% Marketing -17% Product design or development 1% -33% -19% Production & Process Management 252% -56% 18% 13% Quality management -49% -26% -12% 11% Sales training 147% -25% 0% -29% Supply chain management -12% 20% Other TOTAL 8% -31% -17% -3% 41 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 “USAID knows now that we (Lebanon) have a lot of potential in consulting. They don’t need to bring international consultants for their programs. (…) We (the firm) had enough local consultants available. There was no need to bring in international consultants.” –KII Respondent (Grantee) FINDING 5.B.3 The aspects of LED’s interventions that are not scalable relate to (i) the business advisor’s role as a facilitator and trust-builder that also helped manage TA quality, (ii) the components or tasks of the TA design / SOW that are tailored to the client’s specific needs, and (iii) some firms re-hiring more junior consultants (at lower cost) to implement the TA. In addition, one grantee mentioned that reporting on jobs in highly competitive sectors (e.g., technology startups) was an issue due to the sensitivity of data there. EQ 5.B CONCLUSIONS  Increasing efficiencies in LED’s firm-selection process along with the use of “standardizable” types of TA are likely factors that have led to declining marginal costs when replicating TA generally and also specific types of TA. This observation is based on a qualitative assessment, as there are few data available to make a full quantitative assessment.  In the future, after LED concludes, it is feasible to replicate and scale LED’s main interventions (firm-level support, capacity building for entrepreneurs and business consultants), by building on LED’s success in creating positive experience for assisted firms (which encouraged some of them to re-hire their business consultants) and beneficiaries of LED grantee projects. Additionally, the increased supply of TA due to a larger, more viable, and more competitive business-consultancy market, along with the market dynamics established between business consultants and firms (acknowledged by LED and USAID as an unintended positive outcome of LED), are key enablers to ensure the sustainability of quality business development services that address firms’ needs.  The process will also have to consider possible delivery challenges arising from not being able to replicate other aspects of the LED Activity, such as the business advisor’s role as a facilitator of the trust relationship between the consultant and firm, overseeing the design of the TA, tailoring the TA’s SOW, supporting firms in the selection of professional and credible business consultants, and ensuring quality TA deliverables. EQ 5.C: How likely is it that beneficiary firms would have been able to effectively research, procure, purchase, monitor, and achieve similar results had USAID’s assistance not been available? FINDING 5.C.1 According to the interviewed firms, it is unlikely they would have searched for and procured business consultancy services, as it was the first time most of them had used an external consultant. In LED’s survey, 70% of respondents said it had been their first time working with an external consultant. More specifically, 79% of micro enterprises, 71% of small enterprises, 56% of medium enterprises, and 46% of large enterprises said it had been their first time. All interviewed stakeholders appreciated USAID/LED’s assistance because it encouraged and improved the capacity of (i) the assisted firms to procure and pay for business consultancy services again in the future; (ii) the business consultants to approach and build relationships with new clients (particularly MSMEs); and (iii) the grantees, who have already approached new donors and funders to scale up their initiatives. Prior to engaging any TA provider, LED tried to involve the beneficiary firm in the process. However, it was soon revealed that many firms did not know how to enlist the services of 42 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 professional and appropriate technical experts or did not feel confident doing so.31 Subsequently, LED modified its process for selecting business consultants, and attempted to mirror common practice in the commercial world, i.e., when a firm is (independently) seeking to procure business or management consultancy services and has to evaluate available prospects, in order to build the capacity and confidence of the firms to hire external consultants. Through LED, the consultants accessed a pipeline of projects with a variety of clients, in a variety of sectors. LED introduced (or in some cases strengthened) the culture of firms directly procuring and paying for consultancy services. This was especially important for smaller firms that had not worked with consultants previously. These firms gained valuable experience in how to procure, deal, and manage consultants’ work and deliverables. In addition, the association with USAID as a funder of LED served as a good reference for business consultants, assisted firms, and grantees/sub-contractors when approaching new clients, buyers, or donors. All LED grantees stated that USAID’s assistance had provided more credibility, enabling the beneficiaries to acquire funds from other donors. In addition, USAID’s assistance through LED helped create a pool of highly qualified local consultants, which other donors can leverage. This aligns with a key informants’ testimony that LED acted as a catalyzer to help companies invest, grow, and become more visible. Most LED beneficiaries stated that LED had given them the technical and financial push they needed to organize their firms internally, increase their production, and grow their business locally and internationally amidst Lebanon’s economic crisis and limited cash flows. In LED’s survey, some beneficiaries stated that they would have started their projects even without LED’s support, but it would have taken them much more time and resources. Below are some relevant quotes from the FGDs with beneficiary firms: “[Without LED’s assistance] We would not have franchised. Now, we have learned how to float but not how to swim.” “We are lucky, given the business we are in. But without LED, we wouldn’t even be thinking about exporting services. We probably would have moved to another country.” “Because we didn’t have money for investments, we weren’t planning to develop any new products. But LED encouraged us to hire a consultant (…) and we are now in the process of closing an [external] investment. Plus, we now have confidence in our products when showing our portfolio to investors.” “Rebranding [our business] and [building] the website needed a huge amount of cash. At that time, and even now with everything that’s happening in Lebanon, we had a cash flow problem. (…) LED took a huge burden [off our shoulders by covering a part] of that huge budget.” EQ 5.C CONCLUSIONS  There is overwhelmingly positive feedback from the various stakeholders interviewed about the role of USAID/LED’s assistance in helping firms sustain and grown their business in a crisis situation. Specifically, the TA was instrumental in helping them do things differently to achieve tangible growth, which they would likely have not been to achieve on their own. The timing of LED’s interventions was critical, as many firms required external help from experts, but the culture was such that it was mostly large firms that had the courage and/or 31 LED Year 1 Annual Report – page 28 43 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 could afford to hire external consultants.  The fact that business owners participated in selecting the business consultants to provide TA to their firms and worked closely with them, along with the fact that they contributed to the TA itself through cost-share, increased their buy-in of the process and significantly contributed to their companies achieving the intended results. This also made them more confident that not only can they hire business consultants in the future, if needed, but that they can also hold them accountable for not generating quality outputs that lead to results. 44 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 RECOMMENDATIONS The following section includes a set of recommendations devised by the evaluation team after analyzing the above findings and conclusions in addition to some recommendations made by the KIIs and FGD respondents when asked how they thought future USAID/Lebanon Activities similar to LED could be improved. JOB CREATION TARGET AND MEASUREMENT Recommendation 1: USAID/Lebanon should re-assess the relevance and practicality of its focus on job creation, including in terms of indicators and targets, especially in light of the major changes that occurred in LED’s context.  USAID should consider more closely the reasonableness and achievability of the job creation target for future BLA, MLA, and other firm-level support Activities. Following the multiple crises in Lebanon (economic, financial, and Covid-19), the Chemonics team requested amending the contractual target (originally proposed by Chemonics in the bid stage and now considered unrealistic), particularly in light of the massive negative impact of the crises on firms’ hiring decisions. However, as the evaluation team understood from LED’s leadership, USAID did not approve this request, although some downward revision to the KPI targets seems to have been approved in April 2022 as indicated in the latest LED quarterly progress report. Even if this was the case, it came at a late stage in LED’s lifetime. While relaxing the firm-eligibility criteria following the crises helped LED target more firms with job creation potential, it was still not enough for LED to move a lot closer towards its contractual targets.  USAID should consider removing the “Continuing FTE jobs created” disaggregation from the indicator “Number of FTE jobs created” given the long time needed (12 months) for jobs to be considered “continuing” Alternatively, USAID can opt for “Number of net FTE jobs” or a simple “Number of jobs created” indicator (i.e., number of people that benefited from additional jobs, or number of full-time employees).  USAID should consider shifting the focus from creating new jobs to increasing businesses’ incremental sales, which would lead to the creation of new (direct and indirect) jobs over time. This objective is more direct and easier to influence by TA and to assess and manage by an implementing partner (IP). Recommendation 2: Partners implementing Activities similar to LED should regularly conduct “value for money” assessments at the project level for the timely identification and implementation of improvements and adaptations to the project design, objectives, or implementation processes.  If USAID is to continue using “cost per job” as a cost-effectiveness metric in future Activities, it should develop a more accurate (or encompassing) definition of what costs should be included (direct costs only or direct costs with an allocation of indirect costs) and conduct a deeper costing analysis to accurately measure cost-effectiveness at Activity level. USAID should also consider applying this process at each output level. In addition, such a “value for money” assessment should be regularly (for example, annually) conducted and benchmarked internally to the baseline and annual progress of the metric, as well as externally to other comparable projects. FIRM-LEVEL APPROACH Recommendation 3: In future enterprise-development Activities, USAID/Lebanon should consider whether or not other/additional types of support can result in a higher impact and value for money compared to offering firm-level TA only. Examples of other types of interventions include focusing on specific sectors or value chains, strengthening the capacity of key market actors (including 45 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 business consultants and business support organizations), and providing other types of firm-level assistance (such as financing/access to finance and facilitating business linkages).  A more effective firm-level intervention by USAID should consider both non-financial (technical) and financial (cash or in-kind grants, correlated to firm size) assistance to allow the assisted firms to execute the TA recommendations more effectively and comprehensively, especially given the liquidity issue faced by businesses in Lebanon. Examples suggested by the interviewed firms include: o Cover all or part of the certification costs (license, investment in required business changes, etc.) o Website, software, and app development. o Cover exhibition fees for a cluster of assisted companies.  While LED’s TA was wide-ranging and provided support to firms across many sectors, a sectoral (or value chain) focus, such as working with groups or clusters of producers/suppliers in specific sectors or value chains, might have had larger impact and led to higher value for money. (This is similar to the cluster approach followed in the four sub￾contracted initiatives funded by LED, albeit in the final year.)  USAID and the IP should consider also offering other types of support that are more directly focused on helping firms to identify and expand into new international and regional markets (usually an intensive and expensive path to undertake by small firms). This could include business promotion initiatives and B2B linkages, etc., that aim to identify and connect the companies with potential buyers. Such interventions could have a more profound impact on the firms’ growth and performance, and attributing USAID’s support to the reported results would be more assertive.  In future Activities similar to LED, USAID should consider working with fewer firms and increasing the number of repeat TA interventions allowed and/or size of each TA intervention (e.g., $2,000+) to provide more elaborate support and more opportunities for engagement. Additional TA would particularly benefit those firms that have successfully completed the first TA and require additional related support to execute its recommendations (e.g., TA on certification after an export strategy).  USAID should allow enough time for the impact to materialize, during the Activity’s lifetime and beyond, in order to measure the full impact. The results of the four initiatives commenced and completed by LED in Year 5 should be considered for post period-of￾performance (PoP) measurement and monitoring to understand their impact on the performance of the assisted firms if such initiatives are to be replicated.  USAID should capitalize on LED’s positive impact on the business consultancy market by adding a distinct stream/component in the project to systematically strengthen the linkages between local business consultants (business service providers) and SMEs and develop the capacities of the former by working with local organizations.  USAID should explore other, potentially more cost-effective approaches than tailored TA for supporting micro and small firms, e.g., virtual or hybrid incubation or acceleration, especially since this evaluation’s findings indicate that the LED approach’s cost-effectiveness decreased with the firm’s size.  The interviewed firms suggested additional areas of TA or other types of support, including: labeling for businesses; focusing on tourism as a sector that can create jobs quickly; supporting factories in accessing renewable energy to reduce operational costs and overcome electricity challenges. They also recommended that the IP use international experts where needed, e.g., for highly specialized areas like agri-food. 46 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 PROJECT IMPLEMENTATION AND PROCESSES Recommendation 4: Partners implementing Activities similar to LED should take into consideration the lessons learned by LED and its partners that could contribute to increased efficiencies in implementation processes and an increased efficacy of TA outputs, i.e., in the TA design phase (for higher relevance and applicability of TA SOWs), during TA delivery (for quality outputs), and post-TA (for continuous program improvements).  Partners implementing Activities similar to LED should strengthen their firm-diagnosis process by improving the diagnosis skills of business advisors (where needed) or engage experienced management consultants for the diagnosis phase of each TA, so that they (internal or external business advisors) can guide the client firms better on the SOW before these firms start evaluating business consultants for TA implementation. This would also allow for a deeper assessment of the client’s needs, which would minimize any changes needed to the SOW following the first engagement/visit of the interested business consultants.  The IP should require the client firm to conduct an evaluation of the consultant’s work at the end of the TA (plus midterm evaluations for long TA interventions). For added transparency, the IP should also consider putting in place a rating system for client firms to rate the consultant’s profile and credibility following each TA.  As recommended by business consultants, the IP should streamline the role of the business advisor, as having many layers and parties involved (implementing partner, business advisor, business consultant) can lead to confusion for the client and/or the consultant and less transparency around the role of each party.  USAID should review the consultants’ selection process adopted by the IP to optimize the quantity and quality of matched consultants. Having the client select the final consultant (like EBRD’s model) can also be considered. The IP should reduce the number of consultants that the client has to meet to a reasonable number, as this is time consuming for the business. This could be through a clearer RFQ to guide business consultants on the LOE and budget in their application bid and through increased communication between the IP and prospective consultants during the bid stage. Where relevant, the IP should enhance the consultant profile requirements and, accordingly, the projected TA fees to bring on board the best consultants for more advanced/specialized TA. As mentioned by some firms, the IP should widen the selection where there are only a few qualified consultant applications and do the vetting prior to proposing to the client in order to avoid a re-tender.  As noted by some interviewed consulting firms, the IP should rationalize the procurement time between the launch of the RFQ and the start of the TA by, for example, implementing long-term contracts for a smaller number of consultancy firms, depending on specialty (say 15 firms for five types of TA or expertise required at the same price) (cf. the now closed UKAID-funded Lebanon Enterprise and Employment Programme-LEEP design). While it is important to ensure pricing is competitive (for the same level of quality of outputs), some additional indication or guidance on the expected LOE to interested bidders on the RFQ’s scope (e.g., range or maximum number of days expected) would ensure that those that are best qualified and have the appropriate level of resources can apply, thereby saving time during the evaluation stage.  As recommended by some assisted firms, the IP should strengthen its outreach communications efforts to reach companies in remote areas and attract a wider pool of business consultants.  The IP should conduct internal program evaluations (like the LED Entity Self-Evaluation Survey) mid-term rather than at the end of PoP so that the lessons learned can be applied early on. 47 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022  The IP should improve the categorization of types of TA and the sectors used for data collection. The categories should be better defined and broken down into consistent categories, which can better aid analysis of results. GENERAL Recommendation 5: For future interventions, USAID should consider the suggestions provided by the interviewed assisted firms and stakeholders, mainly in relation to stakeholder engagement and sharing and building on the lessons learned.  As noted by a key informant, USAID and other donors should continue using cost-share mechanisms in future firm-level or business service provider development projects so as to not distort the market with free TA that could adversely impact the market linkages, dynamics, and transparency that were built by LED for business consultants and firms.  When international contractors of USAID and other donors bring their international experience to Lebanon, it is important that they take more time in the inception phase of a project to speak to key stakeholders in the market about the implementation approach, as this can help them adapt to the local market quickly.  Future TA should focus on supply chain management in the form of capacity building for factories, based on LED’s successful grant-funded initiative.  More grant management orientation and support should be provided to first-time or less experienced grantees, particularly on HR requirements for M&E and reporting activities.  Organize networking opportunities among assisted firms and with other firms to build linkages and partnerships for business development.  Increase beneficiary firms’ and business consultants’ awareness of USAID projects generally as part of continuous outreach efforts, as relevant, as some interviewed firms said that people in many areas, as well as some business consultants, were not aware of LED’s existence and work. 48 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 ANNEXES Annex I: Evaluation Scope of Work Annex II: LED Results Framework and Indicators Annex III: Evaluation Methods and Limitations Annex IV: Data Collection and Analysis Tools Annex V: Sources of Information Annex VI: Disclosure of Any Conflicts of Interest Annex VII: Characteristics of LED Deals / Assisted firms 49 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 ANNEX I: EVALUATION SCOPE OF WORK LED Final Evaluation SOW April 2022.docx 50 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 ANNEX II: LED RESULTS FRAMEWORK AND INDICATORS 51 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 ANNEX III: EVALUATION METHODS AND LIMITATIONS The evaluation team used a mixed-methods approach to answer the evaluation questions and conducted interviews with key project stakeholders (or key informants) and FGDs with the businesses that received LED support. The team additionally supplemented the qualitative data collection with a document/desk review and an extensive examination of the available quantitative data. The review of LED’s key documents and the meetings held with the LED team and COR indicated that LED had carried out extensive quantitative data collection and a survey with assisted firms, which the evaluation used for further quantitative analysis to inform this evaluation. Thus, the evaluation team did not carry out primary quantitative data collection. Data collection began on June 22nd and ended at the end of July 2022. The evaluation team piloted all the data-collection tools before the fieldwork. The final data collection instruments are presented in Annex IV. The data collection instruments are specifically tailored for each respondent group, as appropriate, to gather information on the respondents’ perspectives. The team facilitated data comparison by asking similar questions to different respondent groups when possible and relevant. DESK REVIEW The evaluation team reviewed existing documents, including LED’s contract agreement and related modifications, workplans, and progress reports (annual, quarterly, other). The team also reviewed other external studies of Lebanon’s SMEs sector and economic growth to understand the additional context to the Activity’s relevance, effectiveness, and sustainability. . Annex V lists the documents reviewed by the evaluation team. QUANTITATIVE DATA ANALYSIS OF LED MONITORING DATA The evaluation team received monitoring data from LED and analyzed them extensively through various forms of disaggregation to determine trends over time and progress toward program goals, using SPSS software for statistical analysis and cross-tabulation. The team also examined all LED’s key performance indicators (KPIs) in relation to firm-related variables for meaningful insights that address the evaluation questions, by examining the main LED KPIs under Objective 1 (number of jobs created, value of increase in sales, value of leveraged investment) vis-à-vis the type of firms assisted (size, sector, location) and type of technical assistance. The evaluation team also analyzed the results of the LED Self-Evaluation Survey (LED Self-Evaluation Questionnaire for Enterprise Clients) conducted by the LED team from November 2021 to August 2022, as part of LED’s internal evaluation. The LED MEL team contacted 470 firms who had received and completed the technical assistance at the time of initiating the survey, of which 37732 firms agreed to proceed with the interviews by the LED team and respond to the survey questions. KEY INFORMANT INTERVIEWS (KIIS) The evaluation team designed a structured interview guide (Annex IV) and conducted virtual KIIs with key stakeholders, including:  USAID/Lebanon Activity management,  LED management, 32 The 377 survey respondents comprise 72% of the total population of 521 directly assisted firms. 52 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022  LED subcontractors’ (Berytech and BIAT) management,  LED subcontractors’ business advisors  project-affiliated business consultants (individuals and firms),  LED grantees (market / ecosystem actors such as business associations, academic institutions, local NGOs, business advisory service providers); and  Beneficiary firms (micro, small, medium and large size companies) that received TA from LED. See Table 14 below for the number of interviewees by type of stakeholder. The evaluation team conducted interviews in Arabic and English, depending on the interviewees’ preferences. Discussions focused on how LED’s outreach, TA design and implementation went and how LED identified and addressed the beneficiary firms’ needs, what challenges they (LED) faced and what successes they achieved. Firm and business consultant participants were invited to talk about their successes and challenges with respect to the LED TA intervention and to suggest ways to improve activity outcomes and achievements in future USAID/Lebanon programming in this sector. The KIIs were semi-structured – about one to one and half hour long. They were conducted by the Team Leader and Team Member with assistance from a note-taker. All interviews were recorded, and the recordings were used to complete the interview notes at a later stage. The KII sampling in each respondent category was as follows: USAID/Lebanon and LED staff: The evaluation team conducted KIIs with the LED COR and LED senior management (COP, DCOP/Director of Operation and Finance, MEL Director, Director of Business Promotion, and Communications Manager). The KIIs aimed to provide an in-depth understanding of project achievements, factors contributing to the project results, and implementation successes and challenges. They also allow the evaluation team to identify patterns and differentiation among the perceptions of various individuals. Business advisors, independent consultants, consulting firms and grantees: The evaluation team interviewed seven out of nine of the project’s Business Advisors and Business Advisor Managers (from Berytech and BIAT). The remaining two business advisors were not available during the data collection phase. The business advisors provided critical information about the successes and challenges of LED, including on selecting firms and business consultants and monitoring the delivery of TA. LED’s direct service providers—the independent consultants (88 in total) and consulting firms (106 in total)—were sampled by selecting those that served the highest number of MSMEs. For instance, the selected nine consultants (out of the 88) served over one-third of the total MSME pool and also included two consultants that served a small number of MSMEs. Similarly, the selected seven of the 106 consulting firms served another 34% of the MSMEs. The types of technical assistance delivered by these seven individuals and seven firms are diverse and represent the majority of services offered to the beneficiary firms. LED awarded grants to 14 entities, of which the evaluation team interviewed four organizations that directly served LED’s objectives of creating jobs and improving the business enabling environment. FOCUS GROUP DISCUSSIONS WITH LED BENEFICIARIES The evaluation team conducted nine FGDs with representatives of 36 beneficiary firms (~7% of the total beneficiary pool of directly assisted firms) using a FGD moderator guide (Annex 4). The FGD participants included male and female respondents, each representing a LED-assisted firm, with an average of three to four participating firms in each session. In some cases, the evaluation team conducted KIIs with firms that were unavailable or unable to have a representative attend the scheduled FGDs. 53 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 The team used LED’s beneficiary list to select participants in a way that represented diverse types of TA received, the two types of service providers that assisted them (firm or independent consultant), firm size and geographic location. The evaluation team’s assumption entering these FGDs was that each participant will be the owner of the sampled assisted firm. That was the case for most interviews; however, the team deviated from this rule occasionally when the owner was unavailable and had sent a co-founder/senior manager (with knowledge about the LED TA) to represent them. Table 14: Breakdown of interviewees by type of stakeholder Stakeholder Title Number of interviews Key Informant Interviews USAID LED COR 1 Interview LED Staff COP / Director of Operations and Finance DCOP/ Deputy Director Director of MEL Director of Business Promotion Communications Manager 5 Interviews LED Sub-contractor Berytech Business Advisor Director Business Advisors 5 Interviews BIAT Sub-contractors Service Providers Business Advisor Manager Business Advisor 2 Interviews LED Consultants Independent Consultants Consulting firms 16 interviews LED Grantees Grantee Management /Representative 4 Interviewees TOTAL KIIs 33 interviews Focus Group Discussions MSMEs Assisted Firms 36 firms (FGDs and KIIs) LIMITATIONS AND BIASES The evaluation team is aware of several noteworthy biases/limitations in this kind of research: Recall Bias: This is expected, especially with beneficiaries who received the services and support more than a year ago. They may also respond with answers that blend their experiences into a composite memory, and those who participated in other similar activities may not separately distinguish their LED experience. In interviews, the evaluation team used references to related projects or historical events to assist in accurate recall and mitigate this bias. Response Bias: Response bias is the risk that key informants may be motivated to provide responses that would be considered socially desirable or influential in obtaining donor support. For example, an owner of a client firm may provide positive remarks about LED because they would like 54 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 to receive similar support in the future or because they want to please the interviewer. To mitigate such bias, the evaluation team clarified at the start of every interview that the responses will not influence the possibility of obtaining future assistance while emphasizing the confidential nature of the interview and the importance of collecting factual information. Selection/Availability Bias: In KIIs, there is a risk that the evaluation team will only reach the most active, responsive, or engaged stakeholders. To mitigate the risk of selection bias, the team has relied on random selections wherever possible and taken only minimal input from the implementers and partners in selection. On the monitoring-data analysis, there is a comparable risk that the low-performing firms may not have reported full data or reported inaccurately. The team attempted to fill any data gaps where possible and explained any resulting limitations in the report. 55 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 ANNEX IV: DATA COLLECTION AND ANALYSIS TOOLS Title Tool Questions to LED team and USAID/Lebanon Business Partner Questionnaire_LED Team - Copy.docx Questions to LED Business advisors Business Partner Questionnaire_Business Advisor .docx Questions to Consulting Firms and Independent Consultants Individual and firm consultant Questionnaire.docx Questions to MSMEs Beneficiaries questionnaire.docx 56 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 ANNEX V: SOURCES OF INFORMATION Below is a list of documents made available to the evaluation team by USAID and LED:  USAID/Lebanon Country Development Cooperative Strategy (2014 – 2021)  USAID/Lebanon Country Development Cooperative Strategy (2022 – 2026)  LED’s original contract agreement  LED contract modifications (1 to 6)  LED annual workplans (six in total, for Y1, Y1.5 (revised), Y2, Y3, Y4, and Y5)  LED quarterly progress reports (15 in total, Y1 through Y5-Q3)  LED annual progress reports (Y1, Y2, Y3 and Y4)  LED MEL Plans (FY2018 and FY2021 (revised))  LED Self-Evaluation Survey for Enterprise Clients, raw data  USEK Study of LED Impact on Assisted Enterprises Since Inception (commissioned by LED)  LED Grantees List  LED KPI Results (as of 9th August 2022)  Other LED-related documents deemed relevant to the evaluation questions  External studies of Lebanon's MSMEs and macro-economy that may lend additional context to the project's relevance, effectiveness, and sustainability. 57 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST Hoda Salman [redacted] Maya Trad [redacted] 58 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 ANNEX VII: CHARACTERISTICS OF LED DEALS / ASSISTED FIRMS Figure 2: Percentage of Closed Deals, by Industry (n=521) Figure 3: Percentage of Closed Deals, by Location (n=521) Agriculture 7% Education and Training 1% Healthcare 1% Industry—Food Processing 36% Industry— Manufacturing 18% Industry—Other 7% Knowledge Economy 6% Oil & Gas 0% Other 0% Power and Water 0% Real Estate and Construction 3% Recycling & Environment 1% Renewable Energy 1% Services 6% Tourism - food services 7% Tourism - other 2% Trade 3% Transport & Logistics 1% Akkar 1% Baalbek-Hermel 1% Beirut 15% Beqaa 17% Mount Lebanon 46% Nabatieh 1% North Lebanon 16% South Lebanon 3% 59 | LED – FINAL PERFORMANCE EVALUATION USAID.GOV FINAL REPORT – SEPTEMBER 2022 Figure 4: Percentage of Closed Deals, by Firm Size (n=521) Figure 5: Percentage of Closed Deals, by Type of Technical Assistance (n=521) Large, 100+ FTE 8% Medium, 51-100 FTE 12% Micro, 1-10 FTE 30% Small, 11-50 FTE 50% Financial management and Finance 3% Access to market 13% Franchise Development 8% HR and Restructuring 6% Knowledge management 7% Marketing 2% Product design or development 8% Production & Process Management 8% Quality management 31% Sales training 12% Supply chain management 2% Other 0%