March 2020 (Revised) This publication was produced for review by the United States for International Development (USAID). It was prepared by International Business Technical Consultants, Inc. (IBTCI), under AID-280-TO-17-00001. EVALUATION FINAL PERFORMANCE EVALUATION OF LIBYA ECONOMIC EMPOWERMENT PROGRAM MONITORING AND EVALUATION FOR TUNISIA AND LIBYA (METAL) ACTIVITY LIBYA ECONOMIC EMPOWERMENT (LEE) PERFORMANCE EVALUATION March 2020 (Revised) International Business & Technical Consultants, Inc. (IBTCI) In the U.S.: 8618 Westwood Center Drive Suite 400 Vienna, VA 22182 USA Tel: +1.703.749.0100 In Tunisia: 1er Étage, Blue Bloc Vital Cube Building 4 L’Ile de Tasmanie Les Berges du Lac 2 1053 Tunis, Tunisia AID-280-TO-17-00001 Cover: Tripoli, Libya; Two Libyan women walking through the streets near the 'souk' at the old Medina Market (John Warburton-Lee Photography / Alamy Stock Photo). DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of IBTCI and do not necessarily reflect the views of USAID or the United States Government. i TABLE OF CONTENTS ACRONYMS............................................................................................................................III 1. ABSTRACT ..........................................................................................................................1 2. EXECUTIVE SUMMARY .....................................................................................................2 2.1. Key Findings ...................................................................................................................................................... 2 2.2. Recommendations........................................................................................................................................... 3 3. EVALUATION PURPOSE AND OBJECTIVES ..................................................................5 3.1. Description of the Evaluation Activity......................................................................................................... 5 4. EVALUATION DESIGNS AND METHODOLOGY ...........................................................7 4.1. Data Collection and Analysis Methods...................................................................................................... 7 4.2. Sampling Frame and Data Gathering .......................................................................................................... 7 4.3. Gender............................................................................................................................................................... 8 5. FINDINGS ..........................................................................................................................10 5.1. Quantitative Results..................................................................................................................................... 10 5.1.1. Document Review..................................................................................................................................... 10 5.2. Participant Survey Results........................................................................................................................... 11 5.3. Qualitative Results (narrative responses to Evaluation Questions based on all survey and document review data)................................................................................................................................ 14 6. CONCLUSIONS AND RECOMMENDATIONS...............................................................18 6.1. Conclusions .................................................................................................................................................... 18 6.2. Recommendations......................................................................................................................................... 20 ANNEX 1: SCOPE OF WORK..............................................................................................21 ANNEX 2: INCEPTION REPORT.........................................................................................43 ANNEX 3: FINAL TIMELINE................................................................................................75 ANNEX 4: COMPILED ACTUAL TEXT RESPONSES........................................................76 ii LIST OF FIGURES Figure 1: LEE Results Framework (updated November 2019) 5 Figure 2: Positive Aspects of the LEE Program 11 Figure 3: Negative Aspects of the LEE Program 12 Figure 4: Sustainability (Ongoing Benefits) 12 Figure 5: Sustainability (Ended Benefits) 13 Figure 6: Additional Service Provision Resulting from the LEE Program 14 LIST OF TABLES Table 1: Surveyed Respondents 8 Table 2 Case Study Participants 9 Table 3 Delivery Performance Against Output Targets 10 iii ACRONYMS BPC Business Plan Competition CA Cooperative Agreement EQ Evaluative Question GOL Government of Libya IBTCI International Business & Technical Consultants, Inc. ICT Information & Communication Technology LEE Libya Economic Empowerment (Activity) METAL Monitoring and Evaluation for Libya and Tunisia (Activity) MEDA Mennonite Economic Development Associates MELP Monitoring, Evaluation and Learning Plan M&E Monitoring and Evaluation SLEIDSE Support to Libya for Economic Integration, Diversification, and Sustainable Development (European Union) SME Small to Medium Enterprise SOW Scope of Work USAID United States Agency for International Development VBS Virtual Business Space 1  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 1. ABSTRACT The Libya Economic Empowerment (LEE) activity supported economic entrepreneurial opportunities targeting women and youth to help promote economic growth and stability. LEE was implemented through a Cooperative Agreement (CA) with the Mennonite Economic Development Associates (MEDA). The activity consisted of three phases, with a period of performance from October 2012 to March 2020 and total estimated cost of $4.5 million. USAID commissioned a final performance evaluation of the activity to gather lessons learned and inform decisions and future programming in the economic growth sector. As defined by the USAID/ Libya statement of work (SOW), this evaluation specifically focused on Phase III of LEE’s implementation, which was implemented from 2017 to 2019 and had a total estimated cost of $1.35 million. MEDA adjusted the program through six amendments that eliminated the components of increasing women’s access to finance through loans/grants from the Government of Libya (GOL) and private institutions, helping women entrepreneurs access local, regional and international markets, and partnering women owned small businesses with successful entrepreneurs, and focused instead on mainly training women and youth to develop and strengthen their business skills, and also issued financial prizes for business plan competitions and small grants to youth to start a business. These changes were made over time as MEDA began to consider training and the small financial provisions a more optimal use of the limited funds available. This evaluation focused on the output targets identified and on interviews with participants of the program’s training and other activities. 2  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 2. EXECUTIVE SUMMARY The demand for economic empowerment programming in Libya remains high. Under the Gadhafi regime, a highly centralized economy dominated by the petrochemical industry left little room for entrepreneurship. More recent instability, including violent conflict, has exacerbated this shortfall. The USAID $4.5 million Libya Economic Empowerment (LEE) activity responded to this need from 2012 to 2019 by supporting economic entrepreneurial opportunities targeting women and youth to help promote economic growth and stability. Other donors have also stepped in to meet the need. These range from relatively large programs such as the Support to Libya for Economic Integration, Diversification, and Sustainable Development (SLEIDSE)1 to more strategic, but comparatively smaller interventions such as Phase III of the LEE Activity. The current Phase III (Cooperative Agreement No. AID-OAA-A-12-00074-MEDA MOD #06) under evaluation covers the period from October 2017 to September 2019. The budget for Phase III was $1.35 million of the entire $4.5 million cooperative agreement. 2.1. Key Findings The objectives of the initial CA with MEDA (No. AID-OAA-A-12-00074), entitled Libya Women’s Economic Empowerment Program: Business Training and SME Development, included “increasing women business owners’ access to finance” and to “local, regional and international markets.” Although these objectives remained in future amendments to the initial CA, operational and budget constraints forced LEE to eliminate large-scale access to finance as an activity, which was reflected in the subsequent work plans. Consequently, LEE became mainly a training activity with additional networking events and a small amount of funds provided for business plan competition financial prizes, and youth innovation grants. USAID and MEDA appear to have worked cooperatively in agreeing on ways forward. One area that MEDA and USAID overlooked was to develop protocols for USAID preapproval of LEE’s Business Plan Competition, which at times was promoted externally without referencing LEE.2 Nonetheless, for the most part, activity planning and execution was transparent and cooperative. The evaluation found that MEDA adjusted the program through six amendments that eliminated the components of increasing women’s access to finance through loans/grants from the Government of Libya (GOL) and private institutions, helping women entrepreneurs access local, regional and international markets, and partnering women owned small businesses with successful entrepreneurs, and focused instead on mainly training women and youth to develop and strengthen their business skills, and awarding financial prizes for business plan competitions and small grants to youth to start a business. These changes were made over time as MEDA began to consider training and the small financial provisions a more optimal use of the limited funds available. All these changes were incorporated into workplans that USAID approved. One challenge faced by MEDA with the series of CA amendments was that the MELPs did not keep pace with the updated work plans, as they continued to contain indicators that were no longer relevant to the most recent scope of the Activity. As a result, the evaluation relied on the indicators and targets reflected in the work plans for which data had been collected regularly by MEDA. MEDA selected participants for the program on a nondiscriminatory basis through open advertising of training slots to ensure adequate demand and to mitigate against the perception that the program showed any bias in its selection of participants. 1 The SLEIDSE program received 7.8 million euros from the EU over 4 years. 2 https://www.libya-businessnews.com/2017/02/28/business-plan-competition-for-libyan-smes/ 3  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 At both output and outcome levels, the program has met or exceeded its targets. In addition, the vast majority of women and youth participants contacted were able to provide concrete examples of how the training had helped them further their careers or businesses. In cases when individuals participated in a series of linked trainings, such as under the Business Plan Competition (BPC) activities, the cumulative benefits of multiple trainings were consistently reported. Some participants expressed dissatisfaction with the results of the business plan competition because the criteria for awarding grants and the qualifications of the judges were not made known to participants. “Lighter-touch” activities such as business networking events were difficult to evaluate, primarily because of the difficulty in tracking down attendees and their difficulty in attributing specific benefits to such events. MEDA has also used its relationships with similar but larger programs of other donors (particularly training) to complement the activities of LEE with others such as SLEIDSE. Most high-level stakeholders, including those from MEDA, agree that before any new economic empowerment activity is designed, the current context must be well known and factored into the design, and that the implementing mechanism must be drafted in a way to ensure maximum flexibility to quickly adapt to the fluid Libyan operating environment. For example, the effect of the imposition of Islamic banking rules on some financial institutions has severely affected the liquidity of all organizations to operate in Libya, which has far-reaching impacts on the business sector, civil society organizations, and donor programs. 2.2. Recommendations The following recommendations are intended to inform future USAID programming to promote the economic empowerment of key groups. Recommendation 1. USAID should conduct an independent study to decide whether to invest in a new economic empowerment activity and, if so, what its size, objectives, components, and implementing mechanism should be. The remaining recommendations are premised on USAID funding a new economic empowerment activity. Recommendation 2. The activity design should allow for greater flexibility in scope to obviate the need for successive amendments to the award agreement. Recommendation 3. USAID needs to ensure that all key documentation, including work plans, MELPs, and result frameworks, remains consistent throughout the life of the activity. Recommendation 4. If the new activity includes training, monitoring efforts following each training session should be improved to use qualitative questions such as “Please list up to three things that you learned from this training” and (linked to the first answer) “Please state how you have used this new knowledge.” These would replace the more commonly used satisfaction scales regarding the quality and relevance of the training and would bring monitoring data more in line with USAID data quality standards. 4  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Recommendation 5. Key USAID oversight of the program should be moved to Tunisia (and potentially Libya itself, if circumstances ever allow) to ensure easier communication with and direction from USAID, especially given Libya’s rapidly changing environment. Recommendation 6. There must be clearer protocols and approval mechanisms for partnerships between the new activity and other donor programs. Recommendation 7. Any new activity component that issues awards following a competition must have transparent judging criteria and relevant qualifications for individuals to be appointed as judges. This would reduce the level of participant dissatisfaction with the award process. 5  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 3. EVALUATION PURPOSE AND OBJECTIVES 3.1. Description of the Evaluation Activity The Libya Economic Empowerment (LEE) program was developed as an amended extension of the Libya Women’s Economic Empowerment Program: Business Training and Small to Medium Enterprise (SME) Development (Cooperative Agreement No. AID-OAA-A -12-00074), which began in October 2012, at a total value of $4.5 million. Phase III (Cooperative Agreement No. AID-OAA-A -12-00074-MEDA MOD #06) covers the final period of the activity from October 2017 to September 2019 and covers $1.35 million of the total. The implementing partner for the entire period of the activity was the Mennonite Economic Development Associates (MEDA). Phase III had two primary objectives: 1) increase business performance of participating women and youth; and 2) reduce barriers to entry for enterprise development for participating women and youth. These were supported by three sub-objectives and seven activities. The results framework of Phase III is illustrated below in Figure 1. Figure 1: LEE Results Framework (updated November 2019)3 USAID/Libya requested Monitoring and Evaluation for Tunisia and Libya (METAL) to conduct a final performance evaluation of Phase III of the LEE activity. The purpose is to identify the extent to which LEE has achieved its intended results and the effectiveness and relevance of the approaches used to achieve USAID/Libya objectives, and garner any lessons learned or best practices. The results of the 6  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 evaluation will be used to inform future USAID/Libya programming in the economic growth sector. The primary audience for this evaluation is USAID/Libya, particularly the Economic Growth Office. The evaluation questions (EQs), as defined in the SOW (see Annex 1), are as follows: EQ 1: To what extent was the design of the project approach relevant to the economic empowerment of Libyan women and youth? ▪ EQ 1.A: To what extent was the LEE approach (mix of interventions, beneficiary targeting, and partnerships), properly aligned to achieve the project’s objective? ▪ EQ 1.B: Were there any internal factors (e.g., management capacity, resources, local staffing, etc.) or external factors (e.g., local partnerships, host government, regulatory environment, etc.) that affected the implementation of the project? How did these factors affect implementation? ▪ EQ 1.C: To what extent was the LEE approach relevant/flexible/manageable throughout implementation in the shifting Libyan context? How effective were the steps taken to adaptively manage the activity in light of the operating context and other constraints? EQ 2: To what extent did LEE achieve its intended objective of improving economic empowerment of women and youth in Libya? ▪ EQ 2.A: How and to what extent did the LEE interventions contribute to increased business skills and business performance of the beneficiaries? ▪ EQ 2.B: How effective were the grants, rewarded as a result of a business plan competition, in enabling entrepreneurs to begin businesses? ▪ EQ 2.C: How and to what extent did the LEE interventions increase beneficiaries’ access to financial resources for enterprise development (excluding grants awarded in the business plan competition)? ▪ EQ 2.D: How and to what extent did the LEE interventions increase the enterprise development￾related skills and capacity of participating beneficiaries? ▪ EQ 2.E: How and to what extent did the LEE interventions increase formal or informal business linkages and contribute to enterprise development? EQ 3: To what extent are the LEE results sustainable? What steps could have been taken to enhance the sustainability of the results? 7  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 4. EVALUATION DESIGNS AND METHODOLOGY 4.1. Data Collection and Analysis Methods This evaluation uses a targeted, mixed-method approach to research. This includes document review, participant surveys, case study investigations, and key informant interviews. All research is designed to gather data that can help answer the evaluative questions posed by the SOW. Phase III of LEE is mainly a training program targeting women and youth to impart knowledge and develop their skills to facilitate their employment or enable them to become entrepreneurs. It is clear from review of the LEE objectives (see Section 3.2) that determining their achievement will require some measurement of knowledge use among the training participants. In addition, the indicator data collected to measure results is largely output-level or perceptional. It does not actually measure the acquisition of knowledge or provide information on the use of that knowledge. This evaluation gathered data from a representative sample of participants and asked for responses that validly tested knowledge and requested qualitative examples4 of knowledge/skills used. Primary data was collected through semi-structured, one-on-one surveys, or interviews conducted via phone (or in person, when feasible). Instruments used for this purpose have been translated as required. The data collection instruments are contained in Annex 2. The evaluation also collected data from eight MEDA staff (4F/4M) and three USAID staff (2F/1M). Secondary data was collected and reviewed from program documents such as the CA and amendments, work plan, MELP, and quarterly and annual reports. Other documentation reviewed included both official reports and correspondence of any kind, and any unofficial publications such as print or electronic media articles or social media posts. Qualitative data from semi-structured interviews was translated, coded, and aggregated in line with standard qualitative research approaches. Development of primary codes was done deductively from the interview responses, rather than being predetermined. Secondary codes were also applied to each response, including codes indicating relevance to each evaluation question and relevance to cross-cutting considerations, such as gender-related issues. Synthesis of coded responses is included in both narrative summaries and relevant quantitative statistics. In addition, all instruments gathered relevant header data to allow for disaggregation of data along various lines (including by gender) and comparison of responses from various respondent types. 4.2. Sampling Frame and Data Gathering All available documents were reviewed and additional references (usually web-based) were also identified. The only type of stakeholders for which valid sampling was initially deemed feasible were the direct participants in LEE activities. This was because MEDA was able to supply the evaluation team with complete participant lists, including contact details. Other stakeholders were pre-identified by either USAID or MEDA, therefore “sampling” of those individuals was purposive. From a total of approximately 1,100 Phase III participants, it was planned to randomly sample 65 individuals, which was representative at the 90 percent confidence level and 10 percent margin of error, and representative only at the whole-of-program level (it was not representative of participants within 4 This use of “qualitative examples” as raw data follows the methodology of the Tactical Conflict Assessment Framework developed by USAID for application in difficult operational environments such as Afghanistan. 8  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 each different type of LEE sub-activity which would have required sampling more individuals). This number of participants was considered manageable enough to allow the evaluation team to collect sufficient data within the limited time frame.5 Limitations Unfortunately, even with a limited sample of 65 persons, the team was unable to obtain access to all the respondents. After over 300 attempts to contact individuals from a randomized list of 1,100, only 34 respondents were available and willing to participate in the survey. MEDA assisted greatly by providing the precise (translated) titles of each of LEE’s sub-activities to facilitate the interviews. The lack of success in meeting our sample goal was associated with a number of factors: ongoing sporadic conflict in the country, renewed push to take Tripoli by forces loyal to General Khalifa Haftar, outdated contact details of participants, and reluctance of respondents to be interviewed by strangers. In order to improve this situation, and complete the interviews within the allotted time, an early compromise was made which allowed interviewers to interview fewer of those with lesser involvement in the program because these were the most likely to have provided problematic contact details or were the least motivated to participate if successfully contacted. This resulted in two persons interviewed who had attended business networking events. A separate and additional group of ten participants who had participated in a “pathway” of BPC trainings was also contacted and interviewed using a different instrument, which allowed the team to construct a small case study of cumulative benefits of multiple, sequenced activities. 4.3. Gender All data gathering allowed for gender disaggregation, and included questioning that provided respondents opportunities to raise gender-related issues. Relevant issues raised are highlighted in the findings. Sample Make-Up Of the 34 participants surveyed, 26 were women and 8 were men. Fifteen respondents completed Springboard Leadership Training (women-focused), eight completed ICT and Business Training (including Step-up Training), two attended media booths at networking events, and one attended the BPC Awards Ceremony. The small case study that was done focused on participants who had participated in multiple activities. Table 1: Surveyed Respondents Activities Total Men Women Training Springboard Leadership 15 0 15 ICT 8 3 5 5 A bigger sample could have easily been proposed, but it was unrealistic to expect to be able to access larger numbers of participants because the evaluation had already been advised by the implementer that contacting participants was problematic due to factors such as conflict, changing contact details, and a general reluctance to be contacted and questioned by strangers. 9  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Activities Total Men Women Navigator Leadership 8 2 6 Networking Business Networking Events 2 2 0 Other BPC Awards 1 1 0 For the case study, 10 persons were interviewed who had attended a number of different events. The gender breakdown of each interviewee and the mix of training and events that each participated in follows in table 2. Table 2. Case Study Participants S. No. Gender Events Attended 1 Female Business Plan Competition 2018 General Business Skills Training Pitch Training-BPC Step Forward 2 Male Competition Awards Ceremony Pitch Training-BPC Step Forward 3 Female General Training - BPC Step Forward Pitch Training -BPC Step Forward Business Plan Competition 2018 4 Male Competition Awards Ceremony Pitch Training – BPC Step Forward General Business Skills Training 5 Female General Training - BPC Step Forward Pitch Training - BPC Step Forward Competition Awards Ceremony 6 Male Innovation Award Business Networking Event 7 Male General Business Skills Training Competition Awards Ceremony 8 Male Business Networking Event (BNE) Business Plan Competition 9 Female Pitch Training - BPC Step Forward General Training – BPC Step Forward Business Plan Competition 2018 10 Female General Training - BPC Step Forward Pitch Training -BPC Step Forward Competition Awards Ceremony 10  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 5. FINDINGS 5.1Quantitative Results 5.1.1. Document Review Work plans and the Monitoring Evaluation and Learning Plan (MELP) are reported on in MEDA quarterly reports. They are both heavily focused on output-level indicators and set output targets to be met during delivery. Because work plans were continuously updated and the MELP is static, this created two competing sets of benchmarks for assessing output delivery. This issue was recognized at the inception stage of this evaluation and clarification was sought from USAID regarding which plan to measure results against targets. USAID advised that the work plan targets were the most appropriate to use. This seemed appropriate since the MELP had a number of indicators for areas included in the CA that the program ultimately did not focus on, as illustrated in figure 1. A final compilation of relevant work plan targets and delivery achievements is provided in Table 3. Table 3. Delivery Performance Against Indicator Targets Phase III Totals Latest update data supplied by MEDA November 19, 2019 Activities Targets Actual Total Actual (Female) Actual (Male) Component 1: Training Assessments 2 4 Navigator Leadership Training – men only 120 6 0 6 Springboard Leadership Training – women only 100 168 168 0 Virtual Business Space (VBS) portal 300 63 47 16 ICT Training 160 (60 F, 100 M) 177 76 101 Component 2: Networking Business Networking Events 27 Events (190 F, 40 M) 27 (332) 176 256 Component 3: Access to Finance Business Development Training (workshops + part of competition) 160 (100 F, 60 M) 245 124 121 Finance Awards (prizes/grants) 6 14 10 4 Youth Innovation Grants 1 (3) 0 3 As shown above, in the majority of cases targets have been met or exceeded, with the notable exceptions of very limited delivery of Navigator Leadership Training (male-focused) and low levels of use 11  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 of the VBS portal. The shortfall in delivery of Navigator Training can be associated with Phase III of LEE being the first phase that opened to male youth, so it was necessary to develop new approaches to providing relevant training (including translation of all teaching materials) before any training could take place.6 MEDA was attempting to remedy issues with the VBS portal, as evidenced by the fact that the MEDA-funded Youth Innovation Grant was provided for this purpose. 5.2. Participant Survey Results As noted in Section 4.2, the evaluation team was able to contact only a limited number of participants engaged in LEE activities (34 of over 1,100). Therefore, the following results should be taken at face value only, with no statistical representativeness inferred. Reporting of percentages is omitted to prevent misinterpretation of the representativeness of the data. Question 1. Positive Aspects of the LEE Program All respondents were able to identify one or more positive aspects (benefits) of the program. The qualitative answers they provided suggested four categories of responses. The breakdown of contextual responses using these categories is provided below in Figure 2. Actual contextual responses are tabulated in Annex 4. From the responses, interviewers also consistently noted that respondents enthusiastically supported the benefits provided by LEE. Figure 2: Positive Aspects of the LEE Program Question 2. Negative Aspects of the LEE Program Twenty-four respondents were able to identify one or more deficiencies of the program. The qualitative answers they provided suggested five categories of responses. The breakdown of responses is provided below in Figure 3. 6 The existing Springboard Training was developed to be women-specific, so it could not immediately be applied to men. 26 21 4 3 3 8 1 3 4 1 Total respondents New skills acquired Improvements to existing business practices Improvements to existing networking practices Increases to participant income Women Men 12  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Actual contextual responses are tabulated in Annex 4. It should be noted that the most common response from both women and men was a request to continue the training, which is not necessarily a complaint, but rather a recommendation. Figure 3: Negative Aspects of the LEE Program Question 3. Sustainability (Ongoing Benefits/Ended Benefits) Ongoing Benefits: All respondents were able to identify one or more ongoing benefits that they continue to experience or use, as a result of participation in the LEE program. The qualitative answers they provided suggested three categories of responses. The breakdown of responses are provided below in Figure 4. Figure 4: Sustainability (Ongoing Benefits) Actual contextual responses are tabulated in Annex 4. 26 3 12 5 1 1 8 6 1 Total respondents Lack of training in legal aspects of business The limited duration or extent of training/activity The lack of contextual examples used in training Rigid style of trainer Gender balance of trainees Women Men 26 13 2 13 8 5 2 3 Total respondents New business skills New networking skills New personal skills (including leadership￾related skills) Women Men 13  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Ended Benefits: Only five respondents were able to identify one or more benefits of the program that they no longer continue to experience or use. The qualitative answers they provided suggested three categories of responses. The breakdown of responses using these categories is provided below in Figure 5. Actual contextual responses are tabulated in Annex 4. No men respondents provided examples of ended benefits. Figure 5: Sustainability (Ended Benefits) Question 4. Skills Acquisition and Use Women: All women respondents were able to provide program-targeted examples of things they learned from LEE training, and all could also provide a credible contextual example of how they have applied this knowledge either in their business or personal life. Men: Five of eight men respondents were able to provide program-targeted examples of things they learned from LEE training and a credible example of how they have applied this knowledge either in their business or personal life. Actual contextual responses are tabulated in Annex 4. Question 5. Network Acquisition and Use Data for this question is severely limited. Only three respondents (all men) were involved in networking-specific activities. All three were able to provide examples of networks they acquired through LEE and were also able to also provide a credible example of how they have used these networks to assist their business goals. Question 6. Additional Service Provision Resulting from the LEE Program Twenty-six of the thirty-four respondents were able to identify one or more credible examples of services they have been able to provide to others (or improve for their own business) as a result of the program. The qualitative answers they provided suggested four categories of responses. 26 2 2 1 8 Total respondents New business skills New networking skills New personal skills (including leadership￾related skills) Women Men 14  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 The breakdown of responses using these categories is provided in Figure 6. Actual contextual responses are tabulated in Annex 4. Figure 6: Additional Service Provision Resulting from the LEE Program I. 5.3. Qualitative Results EQ.1: To what extent was the design of the project approach relevant to the economic empowerment of Libyan women and youth? Under a cooperative agreement, the design elements provided are often provided as “indicative” or “illustrative,” rather than fixed requirements, and this was clearly the case with the initial LEE design. In fact, the list of Illustrative Tasks in the CA includes “adaptation of the training program to local needs and context through an embedded program design element.” The design of LEE has therefore been an adaptive and iterative process. The final design is best reflected in the most updated Results Framework provided in Figure 1. As seen in Figure 1, and as reflected in the approved Phase III work plan, the final version of the LEE design removed the component to deliver broader access to finance or markets to participants, and instead focused mainly on delivering training. The limited responses from the participant survey conducted by the evaluation (see Section 4.2) suggest that this training was effective, highly appreciated, and relevant to improving participant business and leadership skills of women and youth. As part of the ongoing adaptation of the LEE design, MEDA decided to leverage the program’s limited resources by partnering with larger programs to leverage resources implemented by other organizations working in the same field within Libya, such as SLEIDSE. This was documented in work plans and represents a way to extend the effects of the program beyond its own resource limitations. Cooperation with SLEIDSE was primarily within the BPC activity, which also provided a number of staged pathway trainings leading to the possibility of being awarded a prize at the end (for a very limited number of participants). Based on survey results and the small case study conducted of the BPC’s 26 17 1 2 1 8 3 1 1 Total respondents Skill transfer (passing on training) Networking assistance to others Business expansion (increased Econ. activity) Provision of employment to others Women Men 15  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 activities, it was found that the training and the BPC were effective means of strongly motivating participants and delivering a more complete range of skills for economic empowerment. As one BPC participant stated: “I think it is necessary to have more of these kinds of activities, especially the competition, because the competitive factor made the participants push further to learn and to win.” However, when partnering with other donor programs, MEDA needs to ensure it does not lose its visibility in the process (see response to EQ 1.B). Another noted deficiency of the program was the lack of training content of local, business-related legal and regulatory frameworks, and a number of surveyed participants highlighted the need for such guidance in establishing businesses. EQ 1.A: To what extent was the LEE approach (mix of interventions, beneficiary targeting, and partnerships) properly aligned to achieve the project’s objective? Again, it is important to note that the LEE design, through the six amendments to the original cooperative agreement eliminated a number of components and instead focused on delivering a mix of business, ICT and leadership training, and small financial prizes and grants to youth to start a business. Beneficiary targeting is limited to relevant gender and age groups, with selection of beneficiaries based on responses to advertisements of training and other opportunities. This approach is appropriate because it avoided perceptions of bias and ensures a basic level of participant motivation. The participant survey data suggest all LEE trainings have been effective in imparting appropriate knowledge and that this knowledge has been regularly used to improve the business activity of participants (see Section 5.2, Question 4). Feedback from respondents on BPC activities suggests that the staged pathways of trainings provided by BPC and the incentive of eventual prizes, albeit to only a very limited number of participants, was especially effective in attracting and motivating participants. EQ 1.B: Were there any internal factors (e.g., management capacity, resources, local staffing, etc.) and external factors (e.g., local partnerships, host government, regulatory environment, etc.) that affected the implementation of the project? How did these factors affect implementation? LEE is a relatively small program with limited resources and staff operating in a difficult environment. Relevant staffing on the USAID side is equally, if not more, limited. Oversight has been provided remotely, often from Washington rather than Tunisia, and as a small part of USAID staffs’ overall responsibilities. However, neither the limited number of MEDA staff nor remote USAID oversight of the program appear to have created any significant implementation challenges. The one area that USAID has found problematic was the lack of alternative contacts provided by MEDA for use when senior staff are unavailable. Partnerships have been used to successfully leverage impact of the program. Overall, these partnerships have been beneficial, but partnerships do come with some risks if not administered carefully. A key risk is that LEE may not be presented in accordance with USAID branding and marking requirements when partnering with other donors’ programs. One example of this was the experience of the BPC activity. In partnering with SLEIDSE, MEDA was able to more than double the number of prize winners (and other funding recipients) for this activity. This was a very positive result. However, not all SLEIDSE affiliates were aware of LEE’s core role in the activity, and publicity about the LEE’s BPC was released without mention of either LEE or MEDA. This suggests the need for greater guidance and oversight, 16  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 both from USAID and MEDA, around protocols and approval procedures for establishing and administering such partnerships. The ongoing conflict in Libya routinely creates accessibility issues for both participants and staff. Significant administrative challenges, such as access to funds and ability to make payments through banks that have severe transactional restrictions in place (and often huge, time-consuming queues) have regularly limited choice of material or service providers to those that are willing to accept checks at a point when cash is the most trusted option. In spite of the operational challenges, MEDA was able to meet or exceed all of its performance targets, except for the Navigator Leadership Training, which only initiated this training during Phase III of the program and, therefore, lacked sufficient time to train large numbers of male youth. Feedback from MEDA staff also indicated that the short planning horizon impeded development of complementary activities, and some of the male youth suggested that improving the ability of the program to provide more staged training leading to creating and managing businesses (now limited to the BPC activity) would have been beneficial. EQ 1C: To what extent was the LEE approach relevant/flexible/manageable throughout implementation in the shifting Libyan context? How effective were the steps taken to adaptively manage the activity in light of the operating context and other constraints? Please see responses to EQs 1A and 1B. EQ 2: To what extent did LEE achieve its intended objective of improving economic empowerment of women and youth in Libya? As discussed in responses to the preceding EQs, Phase III of LEE largely focused on providing training. This was a mix of business-specific trainings (associated with the BPC activity) and more generic leadership and ICT trainings. The survey data demonstrate that the trainings were relevant and effective in imparting appropriate skills and knowledge, and ultimately resulted in these new skills and knowledge being used. Participants of all trainings were able to provide specific and credible examples of how they had put their new skills to productive use. See also responses to the following sub-EQs. EQ 2.A: How and to what extent did the LEE interventions contribute to increased business skills and business performance of the beneficiaries? The survey data suggest that all the trainings have been effective in imparting appropriate knowledge, and that this knowledge has been regularly used to improve the business activity of participants (see Section 5.2, Question 4). Feedback from respondents to the case study on BPC activities suggest that the staged pathways of trainings provided by BPC and the incentive of eventual prizes, albeit to only a very limited number of participants, was especially effective in attracting and motivating participants. One shortcoming of existing business-related training that was raised by a number of participants was the lack of guidance on local regulatory and legal frameworks for business creation and operation. This is potentially an area for future improvement. 17  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 EQ 2.B: How effective were the grants, rewarded as a result of a business plan competition, in enabling entrepreneurs to begin businesses? Feedback from respondents obtained through the BPC case study suggested that the additional funding provided through the grants allowed them to register their new businesses and create websites to promote their businesses and the services and/or products they provide. EQ 2.C: How and to what extent did the LEE interventions increase beneficiaries’ access to financial resources for enterprise development (excluding grants awarded in the business plan competition)? As discussed under response to EQ1, increasing beneficiaries’ access to financial resources for enterprise development beyond the BPC was not included in Phase III of the program. The only activity that could be considered as increasing beneficiary access to financial resources was the Youth Innovation Grant provided to redevelop the VBS Portal, and funding for this grant was supplied directly by MEDA. EQ 2.D: How and to what extent did the LEE interventions increase the enterprise development-related skills and capacity of participating beneficiaries? Because LEE Phase III primarily delivered training, the response to this EQ is largely equivalent to the response to EQ 2.A. The remaining LEE activities, such as the VBS portal, and assessments did not directly address enterprise development-related skills of participants. EQ 2.E: How and to what extent did the LEE interventions increase formal or informal business linkages and contribute to enterprise development? Anecdotal feedback (see Section 5.2, Question 5) suggests that business networking events were effective in creating links between relevant stakeholders, and that these links were subsequently used to improve business performance. However, it should be noted that insufficient data is available to respond to this question with any level of confidence. The number of users of the pre-phase III VBS portal was low, attaining only 21 percent of the target (see Table 1). However, LEE attempted to remedy this problem during Phase III by providing a MEDA￾funded Youth Innovation Grant specifically for its redesign. EQ 3: To what extent are the LEE results sustainable? What steps could have been taken to enhance the sustainability of the results? Because LEE Phase III primarily provided training, participant survey data (see Section 5.2, Question 4) show that nearly all respondents were able to provide credible contextual examples of skills they acquired from the training that they continue to use in their business or professional life. Five of 34 respondents indicated that they had ceased to use program-provided skills; in most cases, this was due to a lack of opportunity to apply the skill rather than to a loss of the skill itself. In addition, the examination of the services participants are now able to provide as a result of training (see Section 5.2, Question 6) revealed that 20 of 43 respondents are passing on these skills to others. 18  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Additional observations A number of other points deserve mention in this qualitative discussion: ● From the case study of BPC participants, a number of them suggested that the competition could be improved by provision of a clear set of criteria on which entries would be judged, as well as the criteria for selecting the judges, which should focus on their relative expertise in the areas they judge. ● There is a need to ensure that monitoring and evaluation (M&E) design keeps pace with adaptive program design to avoid creating competing implementation benchmarks where different program documents provide varying output indicators and targets, as appears to have been the case with the MELP and work plans. ● Other improvements to M&E are also possible. While the MELP attempted to include outcome indicators and MEDA collected data to inform them, many of these indicators were perceptional and therefore failed to meet the “verifiable” requirement of USAID Data Quality Standards. These could be corrected quite simply and without significant additional workloads by moving from collecting perceptional measures from training participants to collecting specific examples of their learnings and skills applied. 6. CONCLUSIONS AND RECOMMENDATIONS 6.1. Conclusions MEDA has been a strong partner in implementation and has provided a sound local knowledge base for adjusting the program, as needed, which resulted in Phase III primarily focusing on training women and youth selected on a nondiscriminatory basis through open advertisements of the program. Survey data indicates that the leadership, ICT, and business planning training provided through LEE were both highly demanded and considered relevant, and training participants reported that they are applying their skills in their business and professionally. As the latest in a round of short extensions to a CA signed in 2012, Phase III deviated from some of the illustrative content of that initial agreement. However, the successive amendments to the work plan were necessary to adjust to the changing operating environment and were approved by USAID. Because the MELP did not reflect the Phase III work plan, this evaluation relied on the indicators and targets contained in the Phase III work plan. Participants also mentioned several areas that could be improved: ● The business skills training should have been sequenced more logically to prepare them for launching a business. ● The Navigational Leadership Training was not provided a sufficient number of times during Phase III. As a result, only 6 of the 120 targeted male youth were able to attend. ● It would have been helpful if the training had included information on the legal and regulatory framework for business creation and operation. 19  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 ● Criteria for awarding prizes and grants to participants and for selecting judges needs to be transparent and shared with participants to reduce allegations of unfairness with the process. While MEDA’s partnering with other donor initiatives was an effective way of leveraging the impact of LEE, USAID and LEE were sometimes removed from visible recognition and branding at the activities LEE conducted with SLEIDSE and its affiliates. 20  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 6.2. Recommendations Recommendation 1. USAID should conduct an independent study to decide whether to invest in a new economic empowerment activity and, if so, what its size, objectives, components, and implementing mechanism should be. The remaining recommendations are premised on USAID funding a new economic empowerment activity. Recommendation 2. The activity design should allow for greater flexibility in scope to obviate the need for successive amendments to the award agreement. Recommendation 3. USAID needs to ensure that all key documentation, including work plans, MELPs, and result frameworks, remains consistent throughout the life of the activity. Recommendation 4. If the new activity includes training, monitoring efforts following each training session should be improved to use qualitative questions such as “Please list up to three things that you learned from this training” and (linked to the first answer) “Please state how you have used this new knowledge.” These would replace the more commonly used satisfaction scales regarding the quality and relevance of the training and would bring monitoring data more in line with USAID data quality standards. Recommendation 5. Key USAID oversight of the program should be moved to Tunisia (and potentially Libya itself, if circumstances ever allow) to ensure easier communication with and direction from USAID, especially given Libya’s rapidly changing environment. Recommendation 6. There must be clearer protocols and approval mechanisms for partnerships between the new activity and other donor programs. Recommendation 7. Any new activity component that issues awards following a competition must have transparent judging criteria and relevant qualifications for individuals to be appointed as judges. This would reduce the level of participant dissatisfaction with the award process. 21  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Annex 1: Scope of Work STATEMENT OF WORK Summative Performance Evaluation Libya Economic Empowerment Project (LEE) II. PURPOSE OF THE EVALUATION USAID/Libya requested Monitoring and Evaluation for Tunisia and Libya (METAL) to conduct a summative (or final) performance evaluation of the Libya Economic Empowerment (LEE) activity. The purpose is to identify the extent to which LEE has achieved intended results, and assess the effectiveness and relevance of the approaches used to achieve USAID/Libya objectives. The evaluation will identify successes, best practices, lessons learned and challenges (internal and contextual factors) that affected the implementation of the activity. The final report will propose recommendations based on the findings that would help inform USAID/Libya’s programming strategy and decision-making. The primary audience for this evaluation is USAID/Libya, particularly the Economic Growth (EG) Office. III. SUMMARY INFORMATION IV. BACKGROUND The post-revolution period is fraught with the challenges of reconstructing a country and developing an economy. The economic system under Gaddafi was highly centralized and focused on the oil industry. Private sector endeavors and entrepreneurship were not valued or fostered. The business climate should improve as the country stabilizes, but the private sector has historically been weak, so there is a need for targeted support such as entrepreneurship mentoring, and improving access to finance, and in particular for women and youth, who face societal barriers to credit and market systems. A. Description of the Problem and Context LEE was designed to promote the economic potential of entrepreneurship among women and youth, and thereby contribute to economic growth and stability. There is historically a lack of business support networks for entrepreneurs, and promising entrepreneurs fail to commercialize promising ideas or start-up job-creating businesses. Libyan women face cultural barriers in accessing finance and markets, and Libyan youth, particularly at-risk males, often have limited viable economic opportunities aside from militias. The October 2017 award builds on two previous awards comprising five years of experience in supporting entrepreneurship among women. This award has an added focus of male youth who are at-risk of being drawn into extremism. B. Description of the Intervention to be Evaluated and Theory of Change Activity name Libya Economic Empowerment (LEE) Activity USAID office USAID/Libya Implementer Mennonite Economic Development Associates (MEDA) Cooperative agreement/Contract # No. AID-OAA-A-12-00074 Total estimated ceiling of the activity (TEC) $4,989,536 Life of activity October 2017- September 2019 Active geographic regions Libya Development objective(s) Increased opportunity for licit economic growth and participation Required evaluation? Yes External or internal evaluation? External 22  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 LEE Theory of Change: If the Libyan non-oil sector has knowledgeable and competitive business-women and youth who operate successful and job-creating enterprises, and they have a better business support environment, adequate training, networking, and market linkages, then the private sector will be strengthened and the economic empowerment of Libyan women and youth will be improved. The program provides basic business skills training to women and youth at the start-up level, and intermediate to advanced skills training and market linkages activities to existing or aspiring entrepreneurs. Below is a high-level Results Chain or Logic Model linking project activities to their stated objectives. Overall objective: Improved economic empowerment of women and youth in Libya 23  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 INTRODUCTION Monitoring, evaluation, and learning of the USAID/Libya portfolio is an ongoing, collaborative process with the participation of Mennonite Economic Development Associates (MEDA), USAID, and USAID’s third-party monitoring contractor under the Monitoring and Evaluation for Tunisia and Libya (METAL) contract, and other partners. Due to its status as a non-presence operating unit (see ADS 201.3.2.4), USAID/Libya does not have a Country Development Cooperation Strategy (CDCS). USAID/Libya does have a scenario-based strategic plan that serves to articulate USAID’s highest purpose in Libya and links closely to the U.S. Government’s Integrated Country Strategy for Libya. USAID/Libya’s activities are authorized under a single PAD, so the project-level M&E plan is the same as the country-level MEL plan. All AMELPS, including this one, link to USAID’s country-level MEL plan. The USAID/Libya Results Framework (RF) is shown in Figure 1 below. The RF is the graphical representation of the USAID’s country-level development hypothesis for Libya. Figure 1: USAID/Libya Results Framework USAID/Libya Overarching Goal Strengthened foundations of a more unified Libyan state Purpose Libyan institutions and communities advance national stability & self-reliance 'Stability' means minimum levels of predictability in economy, governance 'Advance self-reliance' because Libya has educated populace, oil. Assistance unlocks potential Obj. 1: Institutions of governance more accountable to and better represent the interests of citizens Obj. 2: Increased opportunity for licit economic growth and participation Obj. 3: Libyan communities have enhanced ability to address drivers of instability and conflict Theory of Change: Improved governance and service delivery will increase citizen support for, and stability of, national and local institutions. Theory of Change: If economic reintegration improves, economic reforms advance, and key institutions have more revenue, Libyan self￾reliance will improve, as will national stability. If the drivers and impacts of conflict and extremism diminish, grievances will be less severe and less exploitable by spoilers, improving prospects for national unity and stability. 24  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 The Libya Economic Empowerment (LEE) AMELP supports the overall aims of USAID’s strategic plan and is designed to support achievement of RF Objective # 2, by improving economic empowerment of women and youth in Libya through increased business performance, increased business skills and by reducing the barriers to entry for enterprise development for participating Libyan women and youth BACKGROUND AND ACTIVITY CONTEXT ACTIVITY DESCRIPTION The LEE project was designed specifically to unleash the unrecognized economic potential of women’s and youth’s entrepreneurship and thereby contribute to Libya’s economic growth and stability. In Libya, business support networks for entrepreneurs, and for women and at-risk youth in particular, are weak. As a result, promising entrepreneurs in need of mentoring and specific business expertise are failing to commercialize promising ideas and start-up job-creating businesses at a time of heightened economic dislocation in the country. Libyan women, while highly educated, encounter challenges related to access to finance and markets, including cultural barriers. Libyan youth, particularly at-risk males, are often left with limited viable economic opportunities aside from militias and other violent extremist activities. Launched in 2012, the first phase of the project was women-focused and aimed to grow Libya’s private sector through business training, networking and mentorship. Building upon these successes, LEE’s second phase worked with women-owned and managed businesses and business support organizations to increase the number of women able to engage in entrepreneurship throughout Libya. In October 2017, a third phase was awarded by USAID with a focus on programming for youth. Although women will still benefit from project activities, phase three will focus add-on male youth who are at-risk of being drawn into extremism. Therefore, improvements in the local economy particularly among male youth could create jobs but also provide an alternative to militias and thus counter violent terrorism. A vibrant economy with a private sector that has strong, knowledgeable, and competitive business players who understand the growth potential of win-win commercial relationships, will be a critical element of a healthy Libyan future. The LEE Theory of Change implies that if the Libyan non-oil sector is equipped with strong, knowledgeable and competitive business women and youth, who operate successful and job creating enterprises, and women and youth are better supported with a strengthened business support environment, and women and youth entrepreneurs receive adequate trainings, networking, and market linkages, then the Libyan private sector will be strengthened and the economic empowerment of Libyan women and youth will be improved. The LEE project activities include business skills training and market linkage activities directed toward women and youth with a broad range of skill levels and business acumen. The program provides basic business skills training to women and youth at the start-up level and intermediate to advanced skills training and market linkages activities to existing or aspiring entrepreneurs. Below is a high-level Results Chain or Logic Model linking project activities to their stated objectives. Please refer to Annex 3 for a complete Performance Indicator Reference Sheet (PIRS). 25  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Overall objective or goal: Improved economic empowerment of women and youth in Libya. Objective #1: Increased business performance of participating Libyan women and youth. Objective #2: Reduced barriers to entry for enterprise development for participating Libyan women and youth. Sub-objective #1.1: Increased business skills of participating women and youth entrepreneurs. Sub-objective 2.1: Increased access to finance by participating women and youth entrepreneurs. Sub-objective 2.2: Increased leadership capacity of participating women and youth. Sub-objective # 2.3: Increased formal/informal business linkages established among participating women and youth and market actors. 1.1.1: Business capacity building events organized for women and youth. 1.1.2: Virtual business capacity building platform developed for women and youth. 1.1.3: Co-working location created for women and youth entrepreneurs with embedded BDS services and networking. 1.1.4: ICT skills training conducted for women and youth. 2.1.1: Youth Innovation Grants (matching) provided to youth for business creation. 2.1.2: Business Plan Competition awarded to women entrepreneurs. 2.2.1: Leadership skills training conducted for women and youth Navigator Leadership training for youth and Springboard Leadership training for women). 2.3.1: An assessment conducted to identify unique barriers to entrepreneurship for youth in Libya. 2.3.2: Business Networking Events organized for women and youth entrepreneurs. LEARNING PLAN The LEE project uses results-based project management, monitoring, evaluation and learning approach to ensure that activities are in line with the intended objectives. We utilize a mixed￾methods design that collects and analyses both qualitative and quantitative data to determine project impact, measure results against prescribed targets. For the most part, monitoring of the project is conducted by field M&E officer with support form MEDA HQ M&E Specialist. The LEE project team has created a Performance Measurement Plan (PMP) to track progress of project implementation and achievement of results. The PMP provides information about objectives or results, performance indicators, targets, and data gathering tools or methods, as well as frequency and staff responsible. The PMP is open to modifications given the difficulties of managing LEE activities under conditions of uncertainty. Below is a description of key qualitative and quantitative tools and methods that are part of the PMP reporting. 26  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Surveys with Women and Youth Entrepreneurs MEDA will develop and administer surveys to capture data related to key project objectives from project clients (i.e. women and youth entrepreneurs). The survey, which will collect information from women and youth entrepreneurs to determine the extent to which the project activities produced results identified in the results chain. In particular, client survey will respond to the following indicators: • Percent of participating clients (disaggregated by sex and age) reporting increased economic wellbeing (e.g. increased income or profit, overall sense of improve quality of life, etc.) • Percent of clients (disaggregated by sex and age) who are self-employed (running a business) • Percent of clients (disaggregated sex and age) reporting improved business performance (e.g. more customers, sales, inventory) • Percent of clients (disaggregated by sex and age) reporting improved business skills • Percent of participating clients (disaggregated by sex and age) reporting increased access to finance to operate their businesses • Percent of participating clients (disaggregated by sex and age) reporting increased leadership skills • Percent of participating clients (disaggregated by sex and age) reporting increased linkages with businesses/market actors Interviews with Women and Youth Entrepreneurs In addition to the client survey, individual and/or group interviews are conducted with women and youth entrepreneurs, in particular business competition award winners, to provide in-depth information about their business skills, business performance and overall economic well-being. In particular, these interviews will provide information about the enabling environment for women and youth to operate their own enterprises. Reporting LEE’s field team produces quarterly reports about completion of activities (i.e. outputs) and their effectiveness, including the level of project clients’ satisfaction with these activities. Each quarterly report includes: i. Executive summary: a narrative summary of all activities and overall achievements ii. Key activities conducted during the quarter iii. Key achievements during the quarter, including any additional narrative regarding variance in achieving desired results iv. Key challenges and lessons learned during the quarter v. Project outreach and feedback activities (e.g. social media) Client Success Stories Client (i.e. women and youth entrepreneurs) success stories are collected on an ongoing basis. Inspiring client success stories not only showcase project’s impact in terms improving capacity and performance of participating women and youth entrepreneurs, these stories also help in recruiting more clients for the project. We expect to gather up to 2-4 client success stories each year. The 27  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 LEE field team gather success stories. At minimum, the success stories address the following: a) demographic information of clients; b) situation prior to MEDA intervention; c) description of MEDA support that client received; d) impact or success; e) factors that contributed to success; and f) sustainability—what client is planning to do next. Evaluation and Learning Strategy A final internal evaluation and learning study will be conducted that will identify the extent to which: • The project activities are relevant to achieving project results (Relevance) • The project activities are implemented as intended. (Relevance) • The project immediate and intermediate outcomes have been achieved. (Effectiveness) • The project resources have been used in the best possible way. (Efficiency) • The project contributed towards meeting its objectives. (Impact) • The project has created (or creating) enabling environment that would continue to yield positive results for women and youth entrepreneurs beyond the life of the project. (Sustainability) In addition, the evaluation and learning study will compile learnings related to implementing economic development project in an insecure environment. Some of the key learning questions will include: • What kinds of entrepreneurs (i.e. demographic characteristics) are most likely to benefit from project support in a conflict-ridden context? • What are the most effective ways for an economic development project to acquire women and youth clients in a conflict-ridden context? • What monitoring, and evaluation methods are most appropriate to ensure transparency and accuracy? • What are the challenges that women and youth entrepreneurs and businesses face in a conflict-ridden context? • What additional services may be offered that benefit women and youth clients to run their enterprises? • Is there an opportunity for investment component (e.g. private equity?) to be added to the project? • Which products and delivery channels are best suited to the distribution of grants? For example, is a granted asset a more beneficial and lower-risk option to a mobile transfer, or pre-loaded smart card? • What is the profile of the clients (e.g. location, business size, sector) most likely to benefit from a grant? • What are their barriers to establishing or re-establishing their business? • What are the attitudes of participating youth about available life path choices? • Are there any changes in their attitudes as a result of participating in the project? 28  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Other questions may be added as the project progresses. The evaluation and learning study will make use of both secondary and primary data. Secondary data will include project reports, interviews and data from relevant government departments and civil society organizations about business performance of women and youth entrepreneurs. The primary data will be collected by conducting field visit (dependent on security situation), interviews and, if deemed necessary, surveys. Findings of the evaluation and learning study will be disseminated to all project stakeholders. MEDA HQ M&E Specialist will lead the evaluation and learning study. CONTEXT MONITORING Libya presents several factors that make it a challenging environment in which to operate. Some of the key factors impacting the implementation of development projects include: • The Libyan communication infrastructure is dominated by state-controlled entities. The quality and reliability of the service is often poor, which makes domestic and international calling in Libya very difficult, more so with periodical recurring of power cuts and fuel shortages. • The security situation continues to be fluid and relatively unstable. This can impact delivery of program activities as attendance is affected when the security deteriorates. • NGO registration in Libya and general foreign NGO matters continues to be burdensome and confusing whether in the east or west of the country. The commission of civil society, which grants permits to operate, seems to be internally divided into more than two entities in Tripoli and one in Benghazi. Every year we must submit and complete file as we are newly registering while our documents are supposed to be in the archive. • Due to the unique security situation in Libya, the banks stopped selling foreign currency to the public since November 2014, meaning that USD 1.00 which used to be equal to LYD 1.31 at that time became extremely high and unstable in the parallel market. The rate of exchange has been steadily increasing over time. Inflation has risen alongside prices of goods. DATA QUALITY USAID/Libya will conduct Data Quality Assessments (DQAs) of all indicators that will be reported to Washington or used publicly. METAL will lead the implementation of DQAs for USAID review and approval. LEE will collaborate with METAL as required. All data gathered by the LEE field team will be inputted into standardized forms and stored in the project’s secure web-based collaborative platform, called SharePoint (SP). Only secured users, such as LEE project staff, will have access to the SP folders and documents. Microsoft Excel will frequently be used to capture data findings and tracking client numbers. Data cleaning will also be utilized to ensure accurate reporting and reduce duplicate recording. Both quantitative and qualitative findings will be stored in the project’s secure online platform. In addition to Excel, the MEDA HQ M&E Specialist will use a customer-relationship management (CRM) platform to house client information and connect project activities to unique project clients (such as trainings and events attended.) Excel and CRM will be used in unison for data analysis that will inform quarterly reports. MEDA’s CRM systems is secure with editing access granted only to secure project users. As an additional data analysis tool, LEE will use Microsoft’s Power BI. Power BI transforms excel spreadsheets into interactive visualizations with data disaggregating capabilities. End users can create 29  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 visual reports to illustrate project progress against intended objectives and outcomes. Visualizations will be used to project reports to depict data in a visual, more approachable manner. Below is a flow chart which depicts the data analysis process for LEE: MANAGING THE MEL PLAN Roles & Responsibilities Implementing the AMELP The LEE CoP and MEDA M&E Specialist will be the primary persons responsible for implementing the AMELP reported here. They will also responsible for building the capacity of program staff, reporting and supervising general monitoring and evaluation approaches, practices, and tools. The M&E Specialist will also cooperate closely with the METAL project on the technical level, ensuring that indicators and information are reported as needed to USAID/Libya and the AOR/COR. The MEDA HQ M&E Specialist is in charge of producing M&E reports on time and at high quality manner. The Chief of Party (COP) will have responsibility for overseeing M&E, assuring that the work of the M&E Specialist meets overall project needs and responds to USAID requests for information. The MEDA HQ Specialist will also receive support from MEDA’s M&E Director and other MEDA M&E Specialists as needed. Timeline Please see Annex 1 for a Gantt chart highlighting LEE’s planned activities, their frequency, those responsible for implementation. Budget The MEL plan is implemented by the LEE team with support from MEDA HQ M&E specialists and data analyst. In addition, MEDA trains and hires local enumerators or interviewers as needed to gather information from project clients. When needed, subject specialist consultants are contracted to carry out specific studies that not only inform project design but also contribute to MEL activities. Excel • Data collection: LEE field staff • Data Analysis: MEDA M&E Specialist • Responsibility: MEDA M&E Specialist CRM • Data collection: from Excel, imported to CRM • Data Analysis: MEDA M&E Specialist • Responsibility: MEDA M&E Specialist Power BI • Data collection: from Excel • Data Analysis: MEDA M&E Specialist • Responsibility: MEDA M&E Specialist 30  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 This budget may be revised based on the project requirements. However, it should be noted that in addition to the dedicated M&E professionals, the CoP and HQ Project Manager also contribute to MEL tasks, including providing input in developing data gathering tools. The table below provides key MEL tasks and expense categories for each task other than MEDA personnel (i.e. MEDA HQ Specialist, MEDA HQ Data Analyst and LEE M&E Officer). MEL Task Objectives of the task Expense categories Client (i.e. women and youth) surveys (baseline, annual) To gather information about potential clients in terms of their business knowledge and intention to establish a new or strengthen an existing business. Enumerators Translators Stationery Phone charges Rental space Occasional surveys (e.g. needs assessment for virtual platform) To seek perspective of entrepreneurs about topics such as understanding the use of smart phones, internet, technology for business purposes Translators Enumerators Consultants Stationery Evaluation of events and trainings To assess the effectiveness of LEE activities Enumerators Stationery Success Stories To gather client stories that demonstrate project impact Phone charges Software (CRM and PBI) To ensure seamless entry of data at a secure platform Software fee Final evaluation and learning study To capture project results and lessons learned Enumerators Consultant Translators Due to the iterative contracting nature of the LEE project, the current Phase III budget for M&E activities is provided below. Based on this current phase budget, an estimate of total M&E activities cost for all three phases of the project is provided. However, as noted above, the M & E budget is subject to change based on project requirements. Labor North American Project Manager $1,191 Field Project Manager $9,322 M&E Specialist (HQ) $13,389 M&E Officer (Field) $107,932 Other Direct Costs M&E Activities $26,948 Total M&E Costs $158,782 Total Phase III Contract Value $2,097,327 % of Contract Value 8% Total Contract Value $4,989,536 Total M&E Costs $399,163 COMPLIANCE WITH USAID POLICIES GENDER EQUALITY AND EMPOWERMENT (ADS 205) MEDA’s experience in similar projects to LEE have demonstrated the importance of conducting gender awareness and sensitization activities for partners, project participants and their families (in particular men) and the broader community. Men and the community members in general should be sensitized by holding gender community meetings and workshops on gender issues, including using 31  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 the media to spread the message to the public. The LEE team will explore possibilities of holding media events in collaboration with its partners. Social media will also be used to showcase success stories and highlight challenges that women entrepreneurs face and strategies that can be used overcome those challenges Wherever possible, all data collected for the LEE project will be sex disaggregated. This includes client registrations for trainings and events, as well as client surveys. Sex disaggregated data will be analysed with a gender-lens to identify project activities where women participation is lacking, and strategies will be developed to encourage greater women participation. ENVIRONMENT (22 CFR 216 and ADS 204) MEDA applies an Environmental Management System (EMS) to all its field projects which includes screening of assessment tools designed to mitigate environmental and climate change risks associated with our work, while supporting entrepreneurs in climate change adaptation and improving local environment conditions. In addition, the EMS guides field offices in adhering to their own national environmental regulations as well as donor standards relating to environment and climate change. Environmental considerations will be reviewed when analysing project data to identify project activities where environmental hazards could exist. Strategies for mitigation will be adapted when needed with support from MEDA’s HQ Environmental team. ANNEXES Annex 1 – Timeline of MEL Tasks Annex 2 – Indicator Tracking Table (external Excel sheet) Annex 3 – Performance Indicator Reference Sheets (PIRS) Annex 4 – Performance Measurement Plan (PMP) for LEE 32  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 ANNEX 1 – TIMELINE OF MEL TASKS 33  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 ANNEX 2 – INDICATOR TRACKING TABLE (ITT) ANNEX 3 – PERFORMANCE INDICATOR REFERENCE SHEETS (PIRS ANNEX 4: PMP---LIBYA ECONOMIC EMPOWERMENT (LEE) ) Indicator Baseline (Sep 2017) Target (LOP) Data gathering tool or method Frequency/Responsibility Overall objective or goal: Improved economic empowerment of women and youth in Libya Percent of participating women/youth reporting increased economic wellbeing (e.g. increased income or profit, improved recognition within and outside of the household, overall sense of improve quality of life, etc.) na 60% Interview or surveys with participating women and youth Annually/LEE M & E officer Objective # 1: Increased business performance of participating Libyan women and youth Percent of youth self-employed (running a business) disaggregated by sex 0 70% Interview or surveys with participating women and youth Annually/LEE M & E officer Percent of women self-employed (running a business) disaggregated by age 0 70% Interview or surveys with participating women and youth Annually/LEE M & E officer Percent of youth reporting improved business performance (e.g. more customers, sales, invetory) 0 70% Interview or surveys with participating women and youth Annually/LEE M & E officer Percent of women reporting improved business performance (e.g. more customers, sales, inventory) 0 70% Interview or surveys with participating women and youth Annually/LEE M & E officer Objective#2: Reduced barriers to entry for enterprise development for participating Libyan women and youth Perception of women and youth entrepreneurs reporting reduced barriers to entry for enterprise development (e.g. supportive business networks, ease of doing business, etc.) 0 Majority of women youth report that they face fewer barriers to operate their business Interview with participating women and youth Annually/LEE M & E officer 34  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Sub-objective # 1.1: Increased business skills of participating women and youth entrepreneurs Percent of women/youth reporting improved business skills 0 80% Interview or surveys with participating women and youth Semi-annual/LEE M & E officer Sub-objective 2.1: Increased access to finance by participating women and youth entrepreneurs Percent of participating women reporting increased access to finance to operate their businesses 0 80% Interview or surveys with participating women and youth Semi-annual/LEE M & E officer Percent of participating youth reporting increased access to finance to operate their business 0 80% Interview or surveys with participating women and youth Semi-annual/LEE M & E officer Sub-objective 2.2: Increased leadership capacity of participating women and youth Percent of participating women reporting increased leadership skills 0 80% Interview or surveys with participating women and youth Semi-annual/LEE M & E officer Percent of participating youth reporting increased leadership skills 0 80% Interview or surveys with participating women and youth Semi-annual/LEE M & E officer Sub-objective # 2.3: Increased formal/informal business linkages established among participating women and youth and market actors Percent of participating women reporting increased linkages with businesses/market actors 0 80% Interviews or surveys with participating women and youth Semi-annual/LEE M & E officer Percent of participating youth reporting increased linkages with businesses/market actors 0 80% Interview or surveys with participating women and youth Semi-annual/LEE M & E officer 1.1.1: Business capacity building events organized for women and youth Number (and type) of business capacity building organized for women/youth 0 TBD Project records Ongoing/LWEE M & E officer Number of women/youth who attended business capacity building events 0 TBD Project records Ongoing/LWEE M & E officer Level of satisfaction of women/youth about capacity building events na 90% Capacity building evaluation forms Ongoing/LWEE M & E officer 1.1.2: Virtual business capacity building platform developed for women and youth Number of business capacity platforms developed for women/youth 0 1 Project records Ongoing/LWEE M & E officer Number of women/youth reporting benefiting from the business capacity platform 0 TBD Project records Ongoing/LWEE M & E officer 35  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Level of satisfaction of women/youth with the business capacity platform NA 90% BDS platform evaluation forms Semi-annual/LEE M & E officer 1.1.3: Co-working location created for women and youth entrepreneurs with embedded BDS services and networking Number of women/youth reporting benefiting from the co￾working location 0 TBD Project records Ongoing/LWEE M & E officer Level of satisfaction of women/youth with co-working location na 90% Co-working location evaluation forms Semi-annual/LEE M & E officer 1.1.4: ICT skills training conducted for women and youth Number of ICT skills training organized for women/youth 0 9 (5 for youth and 4 for women) Project records Ongoing/LWEE M & E officer Number of women attending ICT skills training 0 40 Project records Ongoing/LWEE M & E officer Number of youth attending ICT skills training 0 80 Project records Ongoing/LWEE M & E officer Level of satisfaction of women/youth with ICT skills trainings na 90% ICT training evaluation forms Ongoing/LWEE M & E officer 2.1.1: Youth Innovation Grants (matching) provided to youth for business creation Number of Youth Innovation Grants 0 2 Project records Ongoing/LWEE M & E officer Value (in USD) of Youth Innovation Grants 0 10,000 Project records Ongoing/LWEE M & E officer 2.1.2: Business Plan Competition awarded to women entrepreneurs Number of Business Plan Competition awarded to women 0 10 Project records Ongoing/LWEE M & E officer Number of women winning Business Plan Competition 0 140 Project records Ongoing/LWEE M & E officer Value (in USD) of Business Plan Competition 0 TBD Project records Ongoing/LWEE M & E officer Number of Business Plan Competition for women/youth for start up or growth 0 6 (50% for women) Project records Ongoing/LWEE M & E officer Value (in USD) of Business Plan Competition for start up or growth 0 TBD Project records Ongoing/LWEE M & E officer 2.2.1: Leadership skills training conducted for women and youth Navigator Leadership training for youth and Springboard Leadership training for women) Number of Leadership Skills Trainings organized for women/youth 0 7 (4 for youth, 3 for women) Project records Ongoing/LWEE M & E officer Number of women who attended Leadership Skills training 0 60 Project records Ongoing/LWEE M & E officer Number of youth who attended Leadership Skills training 0 80 Project records Ongoing/LWEE M & E officer Level of satisfaction of women/youth with Leadership Skills trainings na 90% Training evaluation forms Ongoing/LWEE M & E officer 2.3.1: An assessment conducted to identify unique barriers to entrepreneurship for youth in Libya Number of assessment reports produced 0 1 Project records First year of the project/LWEE M & E officer 2.3.2: Business Networking Events organized for women and youth entrepreneurs 36  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Number of Business Networking Events organized for women 0 19 Project records Ongoing/LWEE M & E officer Number of women who attended Business Networking Events 0 190 Project records Ongoing/LWEE M & E officer Number of Business Networking Events organized for youth 0 8 Project records Ongoing/LWEE M & E officer Number of youth who attended Business Networking Events 0 40 Project records Ongoing/LWEE M & E officer Level of satisfaction of women/youth with Business Networking Events na 90% BNE evaluation forms Ongoing/LWEE M & E officer V. EVALUATION QUESTIONS The Evaluation Team Leader (TL) and USAID will review the evaluation questions (EQ) below and adjust if needed prior to the finalizing of the evaluation design. Relevance: EQ.1: To what extent was the design of the project approach relevant to the economic empowerment of Libyan women and youth? EQ.1.A: To what extent was the LEE approach (mix of interventions, beneficiary targeting, and partnerships), properly aligned to achieve the project’s objective? EQ.1.B: To what extent was the LEE approach relevant/flexible/manageable throughout implementation in the shifting Libyan context? EQ.1.C: Which elements of LEE were the most effective and which were least effective in the current context? Effectiveness: EQ.2: To what extent LEE achieved its intended objectives in improving economic empowerment of women and youth in Libya? EQ.2.A: To what extent and how did the LEE interventions increase the business skills and increased the business performance of the beneficiaries? EQ.2.B: To what extent and how did the LEE interventions increase access to finance by the beneficiaries and contributed to enterprise development? EQ.2.C To what extent and how did the LEE interventions increase leadership capacity of participating beneficiaries and contributed to enterprise development? EQ.2.D To what extent and how did the LEE interventions increase formal/informal business linkages and contributed to enterprise development? Efficiency: EQ.3: To what extent LEE was able to achieve the intended results with optimal use of resources? EQ.3.A: To what extent the USAID’s investment and the leveraged local resources were commensurate with the results achieved (this section should be supported with the targets achieved under the LEE indicators)? EQ.3.B: What were the internal (including; management capacity, resources, local staffing, etc.) and external (including; local partnerships, host government, regulatory environment, etc.) factors that affected the implementation of the project and how did they affect the implementation? Perception and Value: EQ.4: How was the project perceived and valued by the stakeholders? EQ.4.A: To what extent was LEE perceived by the stakeholders to be an effective contributor to women and youth economic empowerment? EQ.4.B: Which elements of LEE have been the most valued by the project’s stakeholders, which ones were the least valued, and why? Negative Effects: 37  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 EQ.5: Were there any unintended outcomes from LEE that had a negative impact? Who was affected? How and to what extent? How they could have been avoided? Sustainability: EQ.6.A: How are the LEE results sustainable and to what extent (to the extent possible this evaluation should address how the LEE contributed to the Journey to Self-Reliance)? What can be done to enhance the sustainability of the LEE results? VI. EVALUATION DESIGN AND METHODOLOGY This performance evaluation will utilize a mixed-methods research design, employing both quantitative and qualitative methods to strengthen the validity of the findings and provide room for data triangulation. METAL will describe and document the methodological approach that will be used and this follows USAID Evaluation best practices. The model will include an evaluation framework and assessment tools for each evaluation question, and highlight the conceptual model(s), specifying the measurement criteria to be used to respond to each question. It will discuss any risks and limitations that may undermine the reliability and validity of the evaluation results. In order to ensure the maximum value for learning and use, a description of the proposed evaluation methodology will include the following: • Review of the existing baseline relevant in data analysis. • Methods of data collection. • An evaluation design that shows how sampling will be done and appropriate sample sizes required to ensure scientific rigor. • Use of primary and secondary data in data analysis and a plan for analysis. Evaluation Design Matrix Template Questions Suggested Data Sources Suggested Data Collection Methods Suggested Data Analysis Methods 1.[Evaluation Question] 2. [Evaluation Question] 3. [Evaluation Question] VII. DELIVERABLES AND REPORTING REQUIREMENTS 1. Evaluation Work Plan: Within two weeks of the evaluation kick-off call with USAID, METAL will present a draft work plan to include: • Draft schedule and logistical arrangements; • Members of the evaluation team, delineated by roles and responsibilities; • Evaluation milestones; • Anticipated schedule of evaluation team data collection efforts; • Locations and dates for piloting data collection efforts, if applicable; • Proposed evaluation methodology including selection criteria for comparison groups, if applicable; and • Evaluation Report outline. METAL will update the COR on any updates (interviewees, survey participants, schedule) as needed. 38  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 2. Evaluation Design: Within two weeks of approval of the work plan, METAL will submit an evaluation design, to include: • Detailed evaluation design matrix that links the Evaluation Questions from the SOW to data sources, methods, and the data analysis plan; • Draft questionnaires and other data collection instruments or their main features; • List of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (will include sampling methodology and methods, including a justification of sample size and any applicable calculations); • Limitations to the evaluation design; and • Dissemination plan (designed in collaboration with USAID). The data analysis plan will clearly describe the TL approach for analyzing quantitative and qualitative data, including proposed sample sizes, specific data analysis tools, and any software proposed to be used, with an explanation of how/why these selections will be useful in answering the evaluation questions for this task. Qualitative data will be coded as part of the analysis approach, and the coding used should be included in the appendix of the final report. Gender, geographic, and role (beneficiary, implementer, government official, NGO, etc.) disaggregation must be included in the data analysis where applicable. All dissemination plans will be developed with USAID and include information on audiences, activities, and deliverables, including any data visualizations, multimedia products, or events to help communicate evaluation findings/conclusions/recommendations. 3. Inception Report: The inception report will describe the conceptual framework the evaluator will use to undertake the evaluation and the justification for selecting this approach. It will detail the evaluation methodology (i.e. how each question will be answered by way of data collection methods, data sources and sampling). The report will also contain a work plan, which indicates the phases in the evaluation with key deliverables and milestones. USAID/Libya will review this report and the evaluation team will submit to COR for approval before it begins implementing the evaluation plan. The inception report will clearly document and discuss how gender analysis will be integrated into the design of the evaluation. 4. Preliminary Presentation for Recommendations Development: The TL must debrief USAID on the preliminary findings of the evaluation. This meeting will provide a summary of any analytical results; discuss challenges, successes and way forward. The TL will deliver an oral presentation of the evaluation findings, conclusions and recommendations for each question to USAID, prior to finalizing the draft evaluation report. The team leader will be required to routinely update the evaluation point of contact on the progress of the evaluation. 5. Final Presentation: In coordination with USAID, and in support of Agency collaboration learning and adaptation methodologies, METAL will organize a learning event and produce a learning report. The focus of the event is to generate varied and diverse learning points including useful and actionable suggestions or proposals for addressing recurrent development challenges (based on the specific activity evaluation) in the Libyan context with the end goal being to enhance achievement of USAID objectives. The specific objectives of the learning event are: ● To disseminate findings and recommendations from the evaluation; ● To review in-depth the key lessons and their implication for future programs; and ● Most importantly, to engage stakeholders on the evaluation topic, to share lessons learned, barriers, successes, discuss recommendations and to generate a dialogue that captures stakeholder input, thoughts, and ideas on the technical approach used to achieve activity results as presented in the evaluation. 6. Final Evaluation Report: The TL will take five working days to respond to and incorporate final draft evaluation report presentation comments from USAID/Libya. METAL will then submit the final report to the COR. 39  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 The TL will submit a final evaluation report that is based on analyzed facts and evidence and fully addresses all the evaluation questions. The report will be 15-20 pages in length (excluding annexes) and comply with the Checklist for Assessing USAID Evaluation Reports (see annexes). After taking into account all the new information and feedback provided on the final oral briefings and draft evaluation report, METAL will submit 10 hard-bound copies and an electronic version of the report to the Mission. METAL will also submit an electronic version in an appropriate media including all tools and products of the evaluation, including instruments and data in data formats suitable for reanalysis. METAL ensure that Appendix I of the USAID Evaluation Policy – Criteria to Ensure the Quality of the Evaluation Report is followed. This includes: ● The evaluation report will represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not and why; ● Evaluation reports will address all evaluation questions included in the scope of work; ● The evaluation report will include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer; ● Evaluation methodology will be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report; ● Evaluation findings will assess outcomes and impact on males and females; ● Limitations to the evaluation will be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.); ● Evaluation findings will be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings must be specific, concise and supported by strong quantitative or qualitative evidence; ● Sources of information will be properly identified and listed in an annex; ● Recommendations will be supported by a specific set of findings; and will be action-oriented, practical and specific, with defined responsibility for the action. ● All quantitative data collected by the evaluation team will be provided in machine-readable, non￾proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data will be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. ● All modifications to the required elements of the SOW of the contract/agreement, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisions should be updated in the SOW that is included as an annex to the Evaluation Report. The final evaluation report must contain the following sections: ● Abstract: This section will be no more than 250 words and will describe what was evaluated, evaluation questions, methods, and key findings or conclusions. ● Executive Summary: This section will be 2-5 pages in length and will summarize the purpose, project background, evaluation design and methodologies including main evaluation questions, key findings, conclusions, and recommendations and lessons learned from the evaluation. ● Background: This section will provide a brief description of the project that highlights the scope of the project, project development hypothesis, activities undertaken in the project, key impact indicators of the project and impact areas of the project. Other activities that complemented the project activities directly or indirectly in the intervention districts must also be highlighted. ● Methodology: This section will detail the methodology and related research protocols undertaken in conducting the evaluation, data collection, analysis, selection criteria/sampling, and related constraints or limitations encountered during the project implementation and evaluation. ● Findings: Empirical facts collected during the evaluation: This section will present findings from the evaluation. The evaluation findings will be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. The evaluation findings 40  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 must assess key outcomes and impacts as structured around the organizational framework of the evaluation questions. The findings will be specific, concise and supported by strong quantitative and qualitative evidence analyzed through scientifically plausible methodologies. Sources of information used in arriving at the findings must be properly acknowledged and listed in an annex. ● Conclusions (Interpretations and judgments based on the findings): Evaluation conclusions will be presented for each key finding. The Conclusions will logically follow from the gathered data and findings. These will be explicitly justified. If and when necessary, the evaluator will state his/her assumptions, judgments and value premises so that readers can better understand and assess them. ● Recommendations (Proposed actions for management): This section will precisely and clearly present recommendations that must be drawn from specific findings. The recommendations will be stated in an action-oriented fashion, must be practical, specific, and with defined responsibility for the requisite action. The recommendations presented in this section will follow the evaluation questions as the organizational framework. ● References: This section will include all documents reviewed, including background documentation and records of technical data application and decision-making. ● Annexes: These will include, but not limited to: o Evaluation SOW (updated, not the original, if there were any modifications); o Evaluation methods; o All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; o All sources of information or data, identified and listed; o Statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team, if applicable; o Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of or describing existing conflicts of interest; and o Summary information about evaluation team members, including qualifications, experience, and role on the team. 7. Submission of Dataset(s) to the Development Data Library: Per USAID’s Open Data policy (see ADS 579, USAID Development Data) METAL will submit to the COR and the Development Data Library (DDL), at www.usaid.gov/data, in a machine-readable, non￾proprietary format, a copy of any dataset created or obtained in performance of this award, if applicable. The dataset should be organized and documented for use by those not fully familiar with the intervention or evaluation. 8. Submission of Final Evaluation Report to the Development Experience Clearinghouse: Per USAID policy (ADS 201.3.5.18) METAL will submit the evaluation final report and its summary or summaries to the Development Experience Clearinghouse (DEC) within three months of final approval by USAID. VIII. EVALUATION TEAM COMPOSITION IX. Description of team members: The team should consist of a team leader, a technical specialist, and local evaluation support staff. The team will possess the skills and experiences below: Team Leader ● Demonstrated 10 years’ experience with economic empowerment program evaluation experience in the Middle East/North Africa and in conflict contexts. ● Advanced degree (Master's or above) or equivalent in development economics related field or in a field related to an area of expertise required for evaluations. ● Solid experience, quantitative and qualitative evaluation methods. ● Solid understanding of the political environment in Libya and the Middle East. 41  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 ● Demonstrated knowledge of USAID’s policies and priorities for evaluation ● Excellent communication, team management, and leadership skills. ● Ability to work with diverse international teams and excellent interpersonal skills. ● Excellent oral and written communication skills in English, including the ability to facilitate groups and present complex material. ● Note American contracted staff are not allowed to travel into Libya Technical Specialist ● Demonstrated 8 years’ experience with economic empowerment program evaluation experience in the Middle East/North Africa and in conflict contexts. ● Advanced degree (Master's or above) or equivalent in development economics related field or in a field related to an area of expertise required for evaluations. ● Knowledge of quantitative and qualitative evaluation methods including key informant interviews, focus group discussions and statistical analysis of survey findings. ● Solid understanding of the political environment in Libya and the Middle East. ● Demonstrated knowledge of USAID’s policies and priorities for evaluation ● Excellent communication skills. ● Ability to work with diverse international teams and excellent interpersonal skills. ● Excellent oral and written communication skills in English, including the ability to facilitate groups and present complex material. ● Note American contracted staff are not allowed to travel into Libya Local Evaluation Support Staff ● Non-Americans in order to travel into Libya ● Demonstrated 5 years’ experience with economic empowerment program evaluation experience in Libya. ● Demonstrated 5 years of in depth knowledge of quantitative and qualitative evaluation methods including key informant interviews, focus group discussions and statistical analysis of survey findings. ● Solid understanding of the political environment in Libya and the Middle East. ● Excellent communication skills. ● Ability to work with diverse international teams and excellent interpersonal skills. ● Excellent oral and written communication skills in English and Arabic. X. EVALUATION SCHEDULE The evaluation will start on May 15, 2019 Performance Evaluation Schedule Estimated LOE in days by activity for a team of three. 13-17 20-24 28-31 3-7 10-14 17-21 24-28 1-5 8-12 15-19 22-26 29-2 5-9 12-16 19-23 26-30 3-6 9-13 16-20 23-27 Desk review of relevant award info/ documents and mid-term evaluation Inception report drafted Inception report submitted Inception report approved Evaluation team meeting with USAID - discussion of detailed work plan, methodology, and evaluation design Detailed work plan Detailed evaluation design Evaluation team develop data collection tools and evaluation design Data collection tools and evaluation design finalized and approved by USAID Field coordinators/interviewers trained on tools Evaluation team meeting with USAID to discuss fieldwork schedules, list of interviewees and tools Finalization of list of interviewees and site visit schedule Introductory meetings with MEDA Fieldwork conducted in Tunisia and remotely with respondents outside of Libya and Fieldwork conducted in Libya (KIIs and face-to-face survey) Data and findings analyzed Key findings presented orally to USAID Preliminary Presentation for Recommendations Development Draft report finalized and submitted Final Presentation USAID feedback provided, incorporated and final evaluation report submitted Final Evaluation Report LEE Activity Evaluation Schedule August September Deliverable Inception Report July Task May June 42  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Tasks Evaluation Advisor/ Team Lead Economic Growth Specialist Evaluation Support Staff 1)Preparation - In briefings, understanding of assignment and documentation, Desk Review, Tool Instruments development and Inception Report 21 19 20 2) Field Work - Data Collection in DC 29 29 15 3)Data Analysis including oral debriefing, presentation, and preparations and submission to USAID for review and comments 15 15 15 4)Draft Report to USAID for Review and Comments 5 5 5 5)Final Draft Report to USAID 5 5 5 6)Final Report for sharing 1 1 Total LOE 76 73 61 XI. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure quality. ● Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity; ● Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly; ● The Executive Summary should present a concise and accurate statement of the most critical elements of the report; ● Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID; ● Evaluation methodology should be explained in detail and sources of information or data properly identified; ● Limitations to the evaluation should be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.); ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions; ● Conclusions should be specific, concise, and include an assessment of quality and strength of evidence to support them supported by strong quantitative and/or qualitative evidence; ● If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females; and ● If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. See ADS 201mah, USAID Evaluation Report Requirements and the Evaluation Report Checklist and Review Template from the Evaluation Toolkit for additional guidance. 43  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 ANNEX 2: INCEPTION REPORT CONTENTS 1.0 Introduction........................................................................................................................................................... 43 2.0 Program Description............................................................................................................................................. 44 2.1 Overview........................................................................................................................................................... 44 2.2 Program Characteristics that Primarily Determine Evaluation Approach ................................................... 44 3.0 Generic (Modal) Considerations Relevant to the Evaluation ............................................................................ 47 3.1 Educational Initiatives....................................................................................................................................... 48 3.2 Capacity-Building (including Institutional Strengthening) Initiatives.............................................................. 49 3.3 Implications for LEE Evaluation........................................................................................................................ 50 4.0 Data Gathering and Analysis................................................................................................................................. 50 4.1 Sampling Considerations.................................................................................................................................. 50 4.2 Data Gathering Modalities............................................................................................................................... 52 4.3 Analysis............................................................................................................................................................. 52 5.0 Evaluation Design Matrix Template ..................................................................................................................... 54 Annex 1. Scope of Work............................................................................................................................................ 57 Annex 2 Draft Instruments......................................................................................................................................... 65 Annex 3. Draft WorkPlan........................................................................................................................................... 74 Update Note: The evaluation team has conducted some initial testing of participant contact data supplied by MEDA. This testing appears to show that there will be significant challenges in randomly selecting any sample of participants, because any attempts at random selection will likely be overwhelmed by a bias towards taking those participants who can (or are willing to), be contacted. This “contactability bias” could hide many important contributing factors such as locational bias, age bias, technical capability bias and other issues. This issue may require some pragmatic adjustments to the approaches used to contact participants. The evaluation team will proceed with the same sample size, however this bias will be declared in any results. 44  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 1.0 Introduction As required by the Scope of Work for this evaluation, this inception report aims to: “…describe the conceptual framework the evaluator will use to undertake the evaluation and the justification for selecting this approach. It will detail the evaluation methodology (i.e. how each question will be answered by way of data collection methods, data sources and sampling). The report will also contain a work plan, which indicates the phases in the evaluation with key deliverables and milestones.” These requirements will be met while maintaining a primary focus on the stated purpose of this evaluation: “The purpose is to identify the extent to which LEE has achieved intended results, and assess the effectiveness and relevance of the approaches used to achieve USAID/Libya objectives.” 2.0 Program Description 2.1 Overview The Libya Economic Empowerment Program (LEE), is a modest initiative developed as an amended extension of the Libya Women’s Economic Empowerment Program: Business Training and SME Development (Cooperative Agreement No. AID-OAA-A -12-00074), which began in October 2012. The current Phase III (Cooperative Agreement No. AID-OAA-A -12-00074-MEDA MOD #06) covers the period from October 2017 to September 2019 and shares a Total Estimated Ceiling (TEC) for the Cooperative Agreement of USAID funding of USD4,989,536.7 From the initial 2012 start date, the Implementing Partner for this program has been the Mennonite Economic Development Associates (MEDA). As the SoW for this evaluation states: “LEE was designed to promote the economic potential of entrepreneurship among women and youth, and thereby contribute to economic growth and stability. There is historically a lack of business support networks for entrepreneurs, and promising entrepreneurs fail to commercialize promising ideas or start-up job-creating businesses. Libyan women face cultural barriers in accessing finance and markets, and Libyan youth, particularly at-risk males, often have limited viable economic opportunities aside from militias.” 2.2 Program Characteristics that Primarily Determine Evaluation Approach Output-level Results Delivery of this program is limited to a small set of activities conducted within the difficult operating environment of Libya. These activities are summarised in Table 1. This table also provides the associated output-level targets and actual results associated with each activity. 7 Note this is the TEC reported in the SOW, but it is for the entire program since October 2012, the incremental amount applied to Phase III appears to be USD1.35M. 45  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Table 1. Phase III Totals Data supplied by MEDA 19/11/2019 Activities Targets Number Actual Actual (Females) Actual (Males) Component 1: Training Assessments 2 4 Navigator Leadership Training 120 Males 6 Springboard Leadership Training 100 Females 168 VBS portal ( D2L Brightspace + SANAD) 300 (M or F) 47 16 ICT Training 160 (Females 60, Males 100) 76 101 Component 2: Networking Business Networking Events 27 Events (Females 190, Males 40) 27 176 256 Component 3: Access to Finance Business development Training (workshops + part of competition ) 160 (Females 100, males 60) 124 121 Finance Awards (prizes/grants) 6 14 10 4 Youth Innovation Grants 1 1 3 46  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 The information in the above table therefore represents the initial, output-level benchmarks that the evaluation will need to use to assess delivery performance of the program.8 These planned outputs will be compared to other final output-level results reported by MEDA, including against relevant output-level indicators their Monitoring, Evaluation and Learning Plan (MELP), with appropriate consideration given to any explanations of variations. It should be noted that the program has evolved since the original Cooperative Agreement was signed. The original agreement was focussed on exclusively on women and included ‘illustrative tasks’ of: Phase III of LEE has been expanded to also cover youth (including male youth). Tasks associated with increasing access to finance and markets have become strictly limited to a small number of ‘prizes’ associated with a Business Planning Competition (BPC).9 These changes in focus have been transparently pre-empted by various drafts of proposed Phase III Workplans and Results Frameworks, all of which appear to have been accepted by USAID. It is therefore not the intent of the evaluation team to treat these deviations as unintentional or inappropriate when assessing the success of the program (unless advised otherwise). Given the Monitoring and Verification of LEE activities conducted by METAL, triangulation of output figures through collection of independent data has already been addressed.10 This is expedient, as this requirement would be very challenging in the Libyan operating environment and may be beyond the existing resource capacity of the Evaluation. Outcome-Level Results The Theory of Change/Results Frame for LEE Phase III requires the following outcome-level results: Objective 1: Increased business performance of participating Libyan women and youth Sub-objective 1.1: Increased business skills of participating women and youth entrepreneurs Objective 2: Reduced barriers to entry for enterprise development for participating Libyan women and youth 8 To be qualified and further informed by secondary targets included in the LEE Monitoring, Evaluation and Learning Plan (MELP). 9 Plus a single ‘innovation grant’ funded by MEDA itself, specifically for development of a revised on-line platform to replace the ‘Brightspace’ platform. 10 Reports also include some immediate participant feedback on events. 47  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Sub-objective 2.1: Increased access to finance by participating women and youth entrepreneurs Sub-objective 2.2: Increased leadership capacity of participating women and youth Sub-objective 2.3: Increased formal/informal business linkages established among participating women and youth and market actors These intended results are clearly designed to contribute primarily to Objective 2. of the overarching USAID/Libya Results Framework (viz. “Increased opportunity for licit economic growth and participation”). Information collected against these outcome-level results is limited, both within the MEL and PMP. In practice, actual reporting tends to be heavily skewed to reporting against output-level indicators. This is understandable, given LEE predominantly delivers training activities. Measuring outcome-level results of training is problematic, if results beyond ‘educational outcomes’ are expected. 3.0 Generic (Modal) Considerations Relevant to the Evaluation In developing an approach to any evaluation, it is wise to take time to examine some basic principles and ensure that approaches remain in line with these principles. Many of these principles, especially those related to data quality, may sound trivial, but if overlooked, will have serious implications for the validity (or ‘meaningfulness’) of evaluation approaches. Figure 1. Targeting of M&E Approaches (-can be interpreted as a Venn diagram) 48  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 As discussed, the primary modality used for delivery by LEE is participant training. While this modality presents only modest challenges in terms of reporting at output-level (e.g. number of trainings held, number of participants completing training, etc.), the challenges associated with measuring outcome-level results of training are far more problematic. There is no ‘one-size fits all’ approach to assessing the outcomes of training, as relevant results will depend on ‘why’ the training was conducted. A key issue in this regard is the need to determine if the training is carried out as part of an ‘educational’ initiative or as part of a ‘capacity-building’ (incl. institutional strengthening) initiative. 3.1 Educational Initiatives Educational initiatives simply seek to impart knowledge or qualifications to individuals. In this case, knowledge or qualifications are the intended end-result, and these things are often termed ‘educational outcomes’. For example, a school focusses on how many graduates it produces, and is not (generally) held accountable for what the graduates do with the knowledge or qualifications. However, even purely educational initiatives must meet a certain data quality threshold, if they are to validly claim success. In short, they need to test the assumption that participation in their training activities actually leads to increased knowledge. Reporting on participation alone is generally not sufficient to pass this threshold. Determining success usually means applying some form of assessment of trainees’ knowledge at the end of training, or as a prerequisite to the conferring of a qualification. Applying this level of rigor in the context of aid programs is not straightforward, as including assessment steps in training courses can actively discourage participation, especially if no recognised or ‘marketable’ final qualification is provided as an additional incentive. In such cases, proxy indications are often sought, such as participant feedback on “whether they learnt anything”. Proxy indications of this type usually fall into two forms. The first, simply asks respondents if they learnt anything on a ‘yes/no’ basis (this includes a range of apparently more complex response formulations, such as “please rate on a scale of 1 to 5 how much you learnt” or “please choose the most important things that you learnt from the following list”. Data gathered using these formulations does not strictly meet USAID DQA standards even for educational outcomes, simply because respondents do not need to have obtained any knowledge to be able to physically answer such questions.11 They are also highly likely to suffer from severe Social Expectation Bias, as humans are rarely willing to admit failings (either on their behalf or by the trainers), if there is a perception that it may prejudice acquiring further assistance. If such bias is unable to be accounted for, it may easily swamp any more meaningful data in datasets. Selecting proxies that are highly likely to be inaccurate is not a wise approach to measurement. The second form of proxy indication often sought for educational outcomes still avoids more comprehensive assessments, but simply asks respondents to repeat back something that was taught. For example, this might include participant feedback elicited through questions such as “please list the three most important things you learnt from the course”. This second form of proxy, while far from ideal, does at least meet basic DQA standards, because respondents cannot relevantly respond if they have not learnt anything (and if they try, the responses will likely show up as irrelevancies). It is worth noting that both of these forms of proxy can be gathered immediately at the completion of training. The second form of proxy does require marginally more effort to gather and analyse, but 11 In fact, it is wholly possible for someone who has not attended the training to provide answers in this form, so they retain the potential for being highly arbitrary/subjective. 49  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 the fact that it meets DQA standards, while the first form of proxy does not, would appear to render this additional effort worthwhile. 3.2 Capacity-Building (including Institutional Strengthening) Initiatives What differentiates capacity-building initiatives from purely educational initiatives is that the capacity￾building is not intended as an end of itself, rather it used as a means to a higher-level result. In other words, capacity-building initiatives implicitly intend for any new capacity to be used.12 It is therefore incumbent upon capacity-building initiatives to attempt to, at least, measure ‘use of knowledge’. This adds significant complications to Monitoring and Evaluation (M&E) efforts, because use of knowledge cannot be measured immediately on completion of training, and requires time-lagged follow-up (and re-accessing) of participants. Direct and comprehensive observational follow-up of participants would be prohibitively resource intensive for most aid settings, so common attempts at measuring use of knowledge also often involve a range of proxy measures: 1. Aspirational Proxies These are attempts to collect data immediately at completion of training and include feedback elicited by questions such as “will you apply anything you learnt from your training?” (-or effectively equivalent formulations such as “please rate on a scale of 1 to 5 how useful the things you learnt from the training will be to your work” or “please select from the following list the things from the training that you will apply in your work”). These aspirational proxies do not strictly meet a DQA standards, because, particularly in difficult operational environments, there is no guarantee that aspirations will become practice.13 They are also highly likely to suffer from severe Social Expectation Bias. 2. Follow-up, Predetermined Response Proxies These attempts to collect data on use of knowledge or skills from training do require a time￾lagged follow-up of training participants, but ask for information in the form of ‘yes/no’ responses to questions such as “have you used your knowledge from the training in your work?” (or equivalent formulations such as “please rate on a scale of 1 to 5 how useful the things you learnt from the training were to your work” or “please select from the following list the things from the training that you applied in your work”). While these attempts go to the additional effort of gathering follow-up information from participants, the type of information provided again requires no actual recall of the training and is highly likely to be subject to severe Social Expectation bias. They therefore do not strictly meet DQA standards. 3. Follow-up, Open Response Proxies These proxies are similar to the preceding type, in that they require time-lagged follow-up of training participants. However, in this case, open responses are sought using questions such as “please give three specific examples of how you have applied the knowledge or skills from your training in your work”. While it remains technically possible that respondents could simply fabricate answers to these types of open response questions, it requires much more effort to do so. Those respondents who do not actually recall the content of the course are also likely to provide responses that are detectable in their irrelevancy. Further, if sufficient information is obtained in responses (e.g. through the prompting of an interviewer), these responses may become ‘verifiable’ -in that a subset of these responses could be examined 12 Using the alternate terminology, institutions will remain unstrengthened if new skills or knowledge exist, but are never applied. 13 They may therefore be considered ‘subjective perceptions’. 50  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 (e.g. as case studies) to both collect more detailed evidence and spot check validity of claims. While far from perfect, under appropriate circumstances these proxies would meet basic DQA standards.14 This form of proxy has the added advantage of making ‘data manufacture’ more difficult and detectable not just for respondents, but also for data gathers, which is often important in remote monitoring scenarios. 3.3 Implications for LEE Evaluation It is clear from review of the LEE objectives (see Section 2.2) that although a pair of sub-objectives are pitched at the educational outcome level (viz. Sub-objectives 1.1 and 2.2), both full objectives require some measurement of knowledge use. LEE therefore appears to be a capacity-building initiative requiring some measure of use of skills to validly claim success. From the program documentation so far reviewed, it appears that the information currently gathered against MEL/PMP indicators (or for other reporting purposes), in relation to either educational or capacity-building outcomes is largely perception. It relies on the weaker of the proxies reviewed above, and does not truly test for acquisition of knowledge or gather verifiable information regarding use of that knowledge. This is not necessarily a criticism, as all aid programs need to pragmatically match M&E systems to budgets and other operational considerations. The mix of indicators agreed upon for LEE may therefore have simply been the ‘best practical option’. The issue for this evaluation is whether it needs to attempt to uphold higher data quality standards when triangulating reported results against such indicators. In practice, this would mean accessing a representative sample of participants and asking for responses that validly test knowledge in line with the second option for proxies under section 3.1 and requesting qualitative examples of knowledge/skills use, in line with option 3 under section 3.2. Maintaining this additional level of data quality rigor is the preference of the evaluation team as it will ensure greater confidence in findings. Doing so will have implications for design of all the evaluation’s data gathering instruments and overall data gathering effort. Draft data gathering instruments developed on this basis are provided in Annex 2. The success other, non-training-based, delivery modalities incorporated into LEE (see Table 1) are either directly measurable (e.g. number of new financial awards) or have direct parallels with the preceding discussion.15 4.0 Data Gathering and Analysis 4.1 Sampling Considerations Given the nature of LEE activities, there is little chance that any substantial third party data sources with sufficient attribution to these activities will be available for evaluation use. This means that the evaluation will need to depend on two key sources of data, namely MEDA documentation and individual program stakeholders. Approaches used to sample such stakeholders have direct implications for the types of findings that can be drawn from an analysis. This is particularly important when considering the sample of participants required. While gathering information from other (higher-level) stakeholders may be considered as applying an ‘Oracle’ approach that simply needs to ensure the appropriate range of 14 This use of ‘qualitative examples’ as raw data follows the methodology of the Tactical Conflict Assessment Framework (TCAF) developed by USAID for application in difficult operational environments such as Afghanistan. 15 Note the points already provided related to acquisition and use of knowledge/skills can equally be applied to networking, as networks do not exist in potentia, to be realised they must be accessed and used in some manner. 51  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 key informants is accessed, feedback from participants must be treated more equally. This means a representative sample of participants will need to be accessed by the evaluation. The first key decision that needs to be made in regard to this sample is the level of representativeness required. From initial estimates provided by the implementing partner, and taking into account the overlaps in individual participation between different types of LEE activities, it is estimated that the total number of direct participants is approximately 300 individuals.16 Starting with a simple, random selection base,17 this would require a sample of about 170 individuals to be representative at the whole of intervention level with a ‘standard’ 95% Confidence Level and 5% Margin of Error. This number can be modestly reduced by purposely stratifying across the different activity types in line with the proportions of participants in each type, but accessing participants on this scale remains likely to be beyond the practical capability of the evaluation. It is also possible to attempt to apply more complex sampling techniques, but these more complex approaches usually only work correctly when unfettered access to individual participants exists. The realities of accessing individuals in difficult environments usually adds a strong 'who can we get to’ bias that defeats their prerequisite assumptions. A pragmatic approach to such problems is to stay with basic sampling, but only claim a reduced level of representativeness (i.e. a 90% Confidence Level and 10% Margin of Error). This approach is favoured as it is also likely to be 'more honest' in terms of the realities of applying statistical approaches to difficult operating environments. It also represents a more conservative approach, which should add to overall evaluation credibility, rather than detract from it. Applying random sampling at this reduced level of representativeness brings the required 'whole of initiative' sample size down to approximately 70 individuals, which is likely more feasible within the evaluation’s resource envelope. However, this approach would have a key limitation. That is, even though the sample would be drawn from across all LEE activities, it would not allow analyses to extract reliable findings about each activity type, or provide assessments of comparative performance between activity types. This is because the sample derived above is only representative at the ‘whole of program’ level. In order to achieve a valid ability to draw conclusions about each activity type or make comparisons between them, sampling would have to be representative of each of the populations of each activity type. When the required samples for each activity type are calculated and summed18, even at the reduced level of representativeness, an overall sample size of approximately 200 is required.19 Again this appears to be beyond the existing resources of the evaluation to achieve. USAID has also requested the evaluation give some focus to the cumulative effects on participants of being involved in more than one LEE activity. A preliminary review of the list of relevant participants (who took part in more than one activity), reveals that these were almost exclusively participants in the Business Plan Competition related activities. The evaluation will therefore also undertake a small case study of 10 such individuals to attempt to identify the cumulative benefits of progressive levels of participation. This is not intended to be a statistically-based analysis, rather it is simply an opportunity to gather more detailed information regarding some of these individuals. 16 To be confirmed as part of evaluation process. 17 And assuming normal distributions. 18 Again based on MEDA preliminary estimates. 19 Given the overlaps in participation between activity types, this would probably not require accessing 200 different individuals. However, any attempt to purposely make use of the overlaps to further reduce sample size would knowingly bias the sample towards individuals who participated in multiple types of activity. 52  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 4.2 Data Gathering Modalities Whatever sample size is finally agreed upon, it will continue to be important to make data gathering as efficient as possible. While email-based or on-line surveys may appear to be the most convenient option (whether for participants or other stakeholders), this form of data gather gathering has its own set of limitations, especially when attempting to elicit higher quality data from the types of open response questions proposed for use in this evaluation (see Section 3.3). Emails are also an unreliable contact option for many Libyans, with most preferring phone-based options (including WatsApp or Viper). Ensuring the most relevant and appropriately detailed information is provided by respondents in response to questions, generally requires the questions to be administered by a pro-active interviewer, who can prompt for initial omissions and maintain response relevancy, as required. However, this does not imply a need to exclusively conduct one-on-one interviews, an interviewer can also effectively perform this function in a focus group format. However, contextual advice suggests that the logistical issues involved in convening focus groups would negate any efficiency gains made by interviewing more than one person at once. Primary data gathering for the evaluation therefore will focus on applying semi-structured, one-on￾one interview-based surveys, conducted via phone. Instruments used for this purpose will be translated as required. As discussed above, participants will be sampled separately. Higher-level stakeholders will be drawn from the following stakeholder types:20 • USAID Staff • MEDA Staff • Local [Libyan] Authorities (formal or informal) • Other USAID Implementing Partner Staff (especially those interacting with LEE) • Other Donor Staff (especially those funding similar programs to LEE) • Other AID Implementing Agencies (especially those delivering similar programs to LEE) • Clients of Participant Businesses?? Secondary data gathering will focus on document review. This includes all program documents, as well as any other relevant documentation cross-referenced in program documentation or identified by higher-level stakeholders. This may include both official reports/correspondence of any kind or any unofficial publications such as print or electronic media articles or social media posts. 4.3 Analysis Given that the raw data returned from the semi-structured interview instruments proposed is primarily qualitative examples, these will need to be translated, coded and aggregated in line with standard qualitative research approaches. Derivation of primary codes for various similar responses to each question will be data-led, rather than predetermined. Secondary codes will also be applied to each response, including codes indicating relevance to each Evaluative Question and relevance to cross-cutting considerations such as gender-related issues. Synthesis of coded responses will include both narrative summaries and relevant quantitative statistics (frequencies, rankings, etc.). 20 The evaluation will largely be reliant on USAID and MEDA identifying and providing contact details for an initial list of relevant individuals within these categories. These individuals (and potentially documentary references) will then be used to inform expansion of the list. 53  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 All instruments will additionally gather relevant header data to allow for disaggregation of data along various lines and comparison of responses from various respondent types. This will include potential for gender disaggregation and comparison. 54  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 5.0 Evaluation Design Matrix Template Questions Suggested Data Sources Suggested Data Collection Methods Suggested Data Analysis Methods Relevance EQ.1: To what extent was the design of the project approach relevant to the economic empowerment of Libyan women and youth? Documentation Participant Interviews Higher-level Stakeholder Interviews Document Review Participant (P) and Higher-level Stakeholder (HLS) Questionnaires Compilation of Documented Evidence Qualitative response coding and quantification EQ.1.A: To what extent was the LEE approach (mix of interventions, beneficiary targeting, and partnerships), properly aligned to achieve the project’s objective? Documentation Participant Interviews/Focus groups Higher-level Stakeholder Interviews Document Review Participant (P) and Higher-level Stakeholder (HLS) Questionnaires Case Study Compilation of Documented Evidence Qualitative response coding and quantification EQ1.B: If and what internal (e.g., management capacity, resources, local staffing, etc.) and external (e.g., local partnerships, host government, regulatory environment, etc.) factors affected the implementation of the project? How did these factors affect implementation? Documentation Participant Interviews Higher-level Stakeholder Interviews Document Review Higher-level Stakeholder (HLS) Questionnaire Compilation of Documented Evidence Qualitative response coding and quantification EQ.1C: To what extent was the LEE approach relevant/flexible/manageable throughout implementation in the shifting Libyan context? How effective were the steps taken to Documentation Document Review Compilation of Documented Evidence 55  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 adaptively manage the activity in light of the operating context and other constraints? Participant Interviews Higher-level Stakeholder Interviews Participant (P) and Higher-level Stakeholder (HLS) Questionnaires Qualitative response coding and quantification Effectiveness EQ.2: To what extent did LEE achieve its intended objective of improving economic empowerment of women and youth in Libya? Documentation Participant Interviews Higher-level Stakeholder Interviews Document Review Participant (P) and Higher-level Stakeholder (HLS) Questionnaires Case Study Compilation of Documented Evidence Qualitative response coding and quantification EQ.2.A: How and to what extent did the LEE interventions contribute to increased business skills and business performance of the beneficiaries? Documentation Participant Interviews Higher-level Stakeholder Interviews Document Review Participant (P) and Higher-level Stakeholder (HLS) Questionnaires Case Study Compilation of Documented Evidence Qualitative response coding and quantification EQ.2.B. How effective were the grants, rewarded a result of a business plan competition, in enabling entrepreneurs to begin businesses? Documentation Participant Interviews Higher-level Stakeholder Interviews Document Review Case Study Compilation of Documented Evidence Qualitative response coding and quantification EQ.2.C: How and to what extent did the LEE interventions increase beneficiaries’ access to financial resources for enterprise development (excluding grants awarded in the business plan competition)? Documentation Document Review Compilation of Documented Evidence 56  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 (Note this will include only one additional formal grant, but may also capture third party funding opportunities resulting from BPC events) Participant Interviews Higher-level Stakeholder Interviews Single KII of single additional grant recipients Participant (P) and Higher-level Stakeholder (HLS) Questionnaires Qualitative response coding and quantification EQ.2.D: How and to what extent how did the LEE interventions increase the enterprise development-related skills and capacity of participating beneficiaries? Documentation Participant Interviews Higher-level Stakeholder Interviews Document Review Participant (P) and Higher-level Stakeholder (HLS) Questionnaires Compilation of Documented Evidence Qualitative response coding and quantification EQ.2.E How and to what extent did the LEE interventions increase formal/informal business linkages and contribute to enterprise development? Documentation Participant Interviews Higher-level Stakeholder Interviews Participant (P) and Higher-level Stakeholder (HLS) Questionnaires Compilation of Documented Evidence Qualitative response coding and quantification Sustainability EQ.3: To what extent are the LEE results sustainable? If and what steps could have been taken to enhance the sustainability of the results? Documentation Participant Interviews Higher-level Stakeholder Interviews Document Review Participant (P) and Higher-level Stakeholder (HLS) Questionnaires Compilation of Documented Evidence Qualitative response coding and quantification 57  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Annex 1. Scope of Work STATEMENT OF WORK Summative Performance Evaluation Libya Economic Empowerment Project (LEE) I. PURPOSE OF THE EVALUATION USAID/Libya requested Monitoring and Evaluation for Tunisia and Libya (METAL) to conduct a summative (or final) performance evaluation of the Libya Economic Empowerment (LEE) activity. The purpose is to identify the extent to which LEE has achieved intended results, and assess the effectiveness and relevance of the approaches used to achieve USAID/Libya objectives. The evaluation will identify successes, best practices, lessons learned and challenges (internal and contextual factors) that affected the implementation of the activity. The final report will propose recommendations based on the findings that would help inform USAID/Libya’s programming strategy and decision-making. The primary audience for this evaluation is USAID/Libya, particularly the Economic Growth (EG) Office. II. SUMMARY INFORMATION III. BACKGROUND The post-revolution period is fraught with the challenges of reconstructing a country and developing an economy. The economic system under Gaddafi was highly centralized and focused on the oil industry. Private sector endeavors and entrepreneurship were not valued or fostered. The business climate should improve as the country stabilizes, but the private sector has historically been weak, so there is a need for targeted support such as entrepreneurship mentoring, and improving access to finance, and in particular for women and youth, who face societal barriers to credit and market systems. A. DESCRIPTION OF THE PROBLEM AND CONTEXT LEE was designed to promote the economic potential of entrepreneurship among women and youth, and thereby contribute to economic growth and stability. There is historically a lack of business support networks for entrepreneurs, and promising entrepreneurs fail to commercialize promising ideas or start-up job-creating businesses. Libyan women face cultural barriers in accessing finance and markets, and Libyan youth, particularly at-risk males, often have limited viable economic opportunities aside from militias. The October 2017 award builds on two previous awards comprising five years of experience in supporting entrepreneurship among women. This award has an added focus of male youth who are at-risk of being drawn Activity name Libya Economic Empowerment (LEE) Activity USAID office USAID/Libya Implementer Mennonite Economic Development Associates (MEDA) Cooperative agreement/Contract # No. AID-OAA-A-12-00074 Total estimated ceiling of the activity (TEC) $4,989,536 Life of activity October 2017- September 2019 Active geographic regions Libya Development objective(s) Increased opportunity for licit economic growth and participation Required evaluation? Yes External or internal evaluation? External 58  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 into extremism. B. DESCRIPTION OF THE INTERVENTION TO BE EVALUATED AND THEORY OF CHANGE LEE Theory of Change: If the Libyan non-oil sector has knowledgeable and competitive business-women and youth who operate successful and job-creating enterprises, and they have a better business support environment, adequate training, networking, and market linkages, then the private sector will be strengthened and the economic empowerment of Libyan women and youth will be improved. The program provides basic business skills training to women and youth at the start-up level, and intermediate to advanced skills training and market linkages activities to existing or aspiring entrepreneurs. Below is a high-level Results Chain or Logic Model linking project activities to their stated objectives. Overall objective: Improved economic empowerment of women and youth in Libya Objective 1: Increased business performance of I participating Libyan women p and youth Sub-objective 1.1: f b w 1 f . o ncreased business skills o articipating women and youth entrepreneurs 1.1.1: Virtual business capacity uilding platform developed for omen and youth 1.2: ICT skills training conducted r women and youth 1.1.3: Co-working location created for women and youth entrepreneurs with embedded BDS services and networking 1.1.4: ICT skills training conducted for women and youth Sub-objective 2.1: Increased access to finance by participating women and y S In of y S I b a outh 2.1.1: Youth Innovation Grants (matching) provided to youth for business creation 2.1.2: Business Plan Competition awarded to women entrepreneurs b ( Objective 2: Reduced barriers to entry for enterprise development for participating Libyan women and youth ub-objective 2.2: creased leadership capacity participating women and outh ub-objective 2.3: ncreased formal/informal usiness linkages established mong participating women and youth and market actors 2.2.1: Leadership skills training conducted for women and youth Navigator Leadership training for youth and Springboard Leadership training for women) 2.1.1: Youth Innovation Grants (matching) provided to youth for usiness creation 2.1.1: Youth Innovation Grants matching) provided to youth for business creation 59  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 C. PROJECT OR ACTIVITY MONITORING, EVALUATION, AND LEARNING (MEL) PLAN AMELP attached, currently under USAID review. LEE MEL Plan.docx I. EVALUATION QUESTIONS The Evaluation Team Leader (TL) and USAID will review the evaluation questions (EQ) below and adjust if needed prior to the finalizing of the evaluation design. Relevance: EQ.1: To what extent was the design of the project approach relevant to the economic empowerment of Libyan women and youth? EQ.1.A: To what extent was the LEE approach (mix of interventions, beneficiary targeting, and partnerships), properly aligned to achieve the project’s objective? EQ.1.B: To what extent was the LEE approach relevant/flexible/manageable throughout implementation in the shifting Libyan context? EQ.1.C: Which elements of LEE were the most effective and which were least effective in the current context? Effectiveness: EQ.2: To what extent LEE achieved its intended objectives in improving economic empowerment of women and youth in Libya? EQ.2.A: To what extent and how did the LEE interventions increase the business skills and increased the business performance of the beneficiaries? EQ.2.B: To what extent and how did the LEE interventions increase access to finance by the beneficiaries and contributed to enterprise development? EQ.2.C To what extent and how did the LEE interventions increase leadership capacity of participating beneficiaries and contributed to enterprise development? EQ.2.D To what extent and how did the LEE interventions increase formal/informal business linkages and contributed to enterprise development? Efficiency: EQ.3: To what extent LEE was able to achieve the intended results with optimal use of resources? EQ.3.A: To what extent the USAID’s investment and the leveraged local resources were commensurate with the results achieved (this section should be supported with the targets achieved under the LEE indicators)? EQ.3.B: What were the internal (including; management capacity, resources, local staffing, etc.) and external (including; local partnerships, host government, regulatory environment, etc.) factors that affected the implementation of the project and how did they affect the implementation? Perception and Value: EQ.4: How was the project perceived and valued by the stakeholders? EQ.4.A: To what extent was LEE perceived by the stakeholders to be an effective contributor to women and youth economic empowerment? EQ.4.B: Which elements of LEE have been the most valued by the project’s stakeholders, which ones were the least valued, and why? Negative Effects: EQ.5: Were there any unintended outcomes from LEE that had a negative impact? Who was affected? How and to what extent? How they could have been avoided? Sustainability: EQ.6.A: How are the LEE results sustainable and to what extent (to the extent possible this evaluation should address how the LEE contributed to the Journey to Self-Reliance)? What can be done to enhance the sustainability of the LEE results? 60  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 EVALUATION DESIGN AND METHODOLOGY This performance evaluation will utilize a mixed-methods research design, employing both quantitative and qualitative methods to strengthen the validity of the findings and provide room for data triangulation. METAL will describe and document the methodological approach that will be used and this follows USAID Evaluation best practices. The model will include an evaluation framework and assessment tools for each evaluation question, and highlight the conceptual model(s), specifying the measurement criteria to be used to respond to each question. It will discuss any risks and limitations that may undermine the reliability and validity of the evaluation results. In order to ensure the maximum value for learning and use, a description of the proposed evaluation methodology will include the following: • Review of the existing baseline relevant in data analysis. • Methods of data collection. • An evaluation design that shows how sampling will be done and appropriate sample sizes required to ensure scientific rigor. • Use of primary and secondary data in data analysis and a plan for analysis. Evaluation Design Matrix Template Questions Suggested Data Sources Suggested Data Collection Methods Suggested Data Analysis Methods 1.[Evaluation Question] 2. [Evaluation Question] 3. [Evaluation Question] DELIVERABLES AND REPORTING REQUIREMENTS 9. Evaluation Work Plan: Within two weeks of the evaluation kick-off call with USAID, METAL will present a draft work plan to include: • Draft schedule and logistical arrangements; • Members of the evaluation team, delineated by roles and responsibilities; • Evaluation milestones; • Anticipated schedule of evaluation team data collection efforts; • Locations and dates for piloting data collection efforts, if applicable; • Proposed evaluation methodology including selection criteria for comparison groups, if applicable; and • Evaluation Report outline. METAL will update the COR on any updates (interviewees, survey participants, schedule) as needed. 10. Evaluation Design: Within two weeks of approval of the work plan, METAL will submit an evaluation design, to include: • Detailed evaluation design matrix that links the Evaluation Questions from the SOW to data sources, methods, and the data analysis plan; • Draft questionnaires and other data collection instruments or their main features; 61  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 • List of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (will include sampling methodology and methods, including a justification of sample size and any applicable calculations); • Limitations to the evaluation design; and • Dissemination plan (designed in collaboration with USAID). The data analysis plan will clearly describe the TL approach for analyzing quantitative and qualitative data, including proposed sample sizes, specific data analysis tools, and any software proposed to be used, with an explanation of how/why these selections will be useful in answering the evaluation questions for this task. Qualitative data will be coded as part of the analysis approach, and the coding used should be included in the appendix of the final report. Gender, geographic, and role (beneficiary, implementer, government official, NGO, etc.) disaggregation must be included in the data analysis where applicable. All dissemination plans will be developed with USAID and include information on audiences, activities, and deliverables, including any data visualizations, multimedia products, or events to help communicate evaluation findings/conclusions/recommendations. 11. Inception Report: The inception report will describe the conceptual framework the evaluator will use to undertake the evaluation and the justification for selecting this approach. It will detail the evaluation methodology (i.e. how each question will be answered by way of data collection methods, data sources and sampling). The report will also contain a work plan, which indicates the phases in the evaluation with key deliverables and milestones. USAID/Libya will review this report and the evaluation team will submit to COR for approval before it begins implementing the evaluation plan. The inception report will clearly document and discuss how gender analysis will be integrated into the design of the evaluation. 12. Preliminary Presentation for Recommendations Development: The TL must debrief USAID on the preliminary findings of the evaluation. This meeting will provide a summary of any analytical results; discuss challenges, successes and way forward. The TL will deliver an oral presentation of the evaluation findings, conclusions and recommendations for each question to USAID, prior to finalizing the draft evaluation report. The team leader will be required to routinely update the evaluation point of contact on the progress of the evaluation. 13. Final Presentation: In coordination with USAID, and in support of Agency collaboration learning and adaptation methodologies, METAL will organize a learning event and produce a learning report. The focus of the event is to generate varied and diverse learning points including useful and actionable suggestions or proposals for addressing recurrent development challenges (based on the specific activity evaluation) in the Libyan context with the end goal being to enhance achievement of USAID objectives. The specific objectives of the learning event are: ● To disseminate findings and recommendations from the evaluation; ● To review in-depth the key lessons and their implication for future programs; and ● Most importantly, to engage stakeholders on the evaluation topic, to share lessons learned, barriers, successes, discuss recommendations and to generate a dialogue that captures stakeholder input, thoughts, and ideas on the technical approach used to achieve activity results as presented in the evaluation. 14. Final Evaluation Report: The TL will take five working days to respond to and incorporate final draft evaluation report presentation comments from USAID/Libya. METAL will then submit the final report to the COR. The TL will submit a final evaluation report that is based on analyzed facts and evidence and fully addresses all the evaluation questions. The report will be 15-20 pages in length (excluding annexes) and comply with the Checklist for Assessing USAID Evaluation Reports (see annexes). After taking into account all the new information and feedback provided on the final oral briefings and draft evaluation 62  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 report, METAL will submit 10 hard-bound copies and an electronic version of the report to the Mission. METAL will also submit an electronic version in an appropriate media including all tools and products of the evaluation, including instruments and data in data formats suitable for reanalysis. METAL ensure that Appendix I of the USAID Evaluation Policy – Criteria to Ensure the Quality of the Evaluation Report is followed. This includes: ● The evaluation report will represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not and why; ● Evaluation reports will address all evaluation questions included in the scope of work; ● The evaluation report will include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer; ● Evaluation methodology will be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report; ● Evaluation findings will assess outcomes and impact on males and females; ● Limitations to the evaluation will be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.); ● Evaluation findings will be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings must be specific, concise and supported by strong quantitative or qualitative evidence; ● Sources of information will be properly identified and listed in an annex; ● Recommendations will be supported by a specific set of findings; and will be action-oriented, practical and specific, with defined responsibility for the action. ● All quantitative data collected by the evaluation team will be provided in machine-readable, non￾proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data will be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. ● All modifications to the required elements of the SOW of the contract/agreement, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisions should be updated in the SOW that is included as an annex to the Evaluation Report. The final evaluation report must contain the following sections: ● Abstract: This section will be no more than 250 words and will describe what was evaluated, evaluation questions, methods, and key findings or conclusions. ● Executive Summary: This section will be 2-5 pages in length and will summarize the purpose, project background, evaluation design and methodologies including main evaluation questions, key findings, conclusions, and recommendations and lessons learned from the evaluation. ● Background: This section will provide a brief description of the project that highlights the scope of the project, project development hypothesis, activities undertaken in the project, key impact indicators of the project and impact areas of the project. Other activities that complemented the project activities directly or indirectly in the intervention districts must also be highlighted. ● Methodology: This section will detail the methodology and related research protocols undertaken in conducting the evaluation, data collection, analysis, selection criteria/sampling, and related constraints or limitations encountered during the project implementation and evaluation. ● Findings: Empirical facts collected during the evaluation: This section will present findings from the evaluation. The evaluation findings will be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. The evaluation findings must assess key outcomes and impacts as structured around the organizational framework of the evaluation questions. The findings will be specific, concise and supported by strong quantitative 63  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 and qualitative evidence analyzed through scientifically plausible methodologies. Sources of information used in arriving at the findings must be properly acknowledged and listed in an annex. ● Conclusions (Interpretations and judgments based on the findings): Evaluation conclusions will be presented for each key finding. The Conclusions will logically follow from the gathered data and findings. These will be explicitly justified. If and when necessary, the evaluator will state his/her assumptions, judgments and value premises so that readers can better understand and assess them. ● Recommendations (Proposed actions for management): This section will precisely and clearly present recommendations that must be drawn from specific findings. The recommendations will be stated in an action-oriented fashion, must be practical, specific, and with defined responsibility for the requisite action. The recommendations presented in this section will follow the evaluation questions as the organizational framework. ● References: This section will include all documents reviewed, including background documentation and records of technical data application and decision-making. ● Annexes: These will include, but not limited to: o Evaluation SOW (updated, not the original, if there were any modifications); o Evaluation methods; o All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; o All sources of information or data, identified and listed; o Statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team, if applicable; o Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of or describing existing conflicts of interest; and o Summary information about evaluation team members, including qualifications, experience, and role on the team. 15. Submission of Dataset(s) to the Development Data Library: Per USAID’s Open Data policy (see ADS 579, USAID Development Data) METAL will submit to the COR and the Development Data Library (DDL), at www.usaid.gov/data, in a machine-readable, non-proprietary format, a copy of any dataset created or obtained in performance of this award, if applicable. The dataset should be organized and documented for use by those not fully familiar with the intervention or evaluation. 16. Submission of Final Evaluation Report to the Development Experience Clearinghouse: Per USAID policy (ADS 201.3.5.18) METAL will submit the evaluation final report and its summary or summaries to the Development Experience Clearinghouse (DEC) within three months of final approval by USAID. EVALUATION TEAM COMPOSITION Description of team members: The team should consist of a team leader, a technical specialist, and local evaluation support staff. The team will possess the skills and experiences below: Team Leader ● Demonstrated 10 years’ experience with economic empowerment program evaluation experience in the Middle East/North Africa and in conflict contexts. ● Advanced degree (Master's or above) or equivalent in development economics related field or in a field related to an area of expertise required for evaluations. ● Solid experience, quantitative and qualitative evaluation methods. ● Solid understanding of the political environment in Libya and the Middle East. ● Demonstrated knowledge of USAID’s policies and priorities for evaluation ● Excellent communication, team management, and leadership skills. 64  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 ● Ability to work with diverse international teams and excellent interpersonal skills. ● Excellent oral and written communication skills in English, including the ability to facilitate groups and present complex material. ● Note American contracted staff are not allowed to travel into Libya Technical Specialist ● Demonstrated 8 years’ experience with economic empowerment program evaluation experience in the Middle East/North Africa and in conflict contexts. ● Advanced degree (Master's or above) or equivalent in development economics related field or in a field related to an area of expertise required for evaluations. ● knowledge of quantitative and qualitative evaluation methods including key informant interviews, focus group discussions and statistical analysis of survey findings. ● Solid understanding of the political environment in Libya and the Middle East. ● Demonstrated knowledge of USAID’s policies and priorities for evaluation ● Excellent communication skills. ● Ability to work with diverse international teams and excellent interpersonal skills. ● Excellent oral and written communication skills in English, including the ability to facilitate groups and present complex material. ● Note American contracted staff are not allowed to travel into Libya Local Evaluation Support Staff ● Non-Americans in order to travel into Libya ● Demonstrated 5 years’ experience with economic empowerment program evaluation experience in Libya. ● Demonstrated 5 years of in depth knowledge of quantitative and qualitative evaluation methods including key informant interviews, focus group discussions and statistical analysis of survey findings. ● Solid understanding of the political environment in Libya and the Middle East. ● Excellent communication skills. ● Ability to work with diverse international teams and excellent interpersonal skills. ● Excellent oral and written communication skills in English and Arabic. EVALUATION SCHEDULE The evaluation will start on May 15, 2019 Performance Evaluation Schedule LEE Activity Evaluation Schedule.xl Estimated LOE in days by activity for a team of three. Tasks Evaluation Advisor/ Team Lead Economic Growth Specialist Evaluation Support Staff 1)Preparation - In briefings, understanding of assignment and documentation, Desk Review, Tool Instruments development and Inception Report 21 19 20 2) Field Work - Data Collection in DC 29 29 15 3)Data Analysis including oral debriefing, presentation, and preparations and submission 15 15 15 65  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 to USAID for review and comments 4)Draft Report to USAID for Review and Comments 5 5 5 5)Final Draft Report to USAID 5 5 5 6)Final Report for sharing 1 1 Total LOE 76 73 61 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure quality. ● Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity; ● Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly; ● The Executive Summary should present a concise and accurate statement of the most critical elements of the report; ● Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID; ● Evaluation methodology should be explained in detail and sources of information or data properly identified; ● Limitations to the evaluation should be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.); ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions; ● Conclusions should be specific, concise, and include an assessment of quality and strength of evidence to support them supported by strong quantitative and/or qualitative evidence; ● If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females; and ● If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. See ADS 201mah, USAID Evaluation Report Requirements and the Evaluation Report Checklist and Review Template from the Evaluation Toolkit for additional guidance. Annex 2 Draft Instruments 1. Participant Questionnaire Format (DRAFT MOCK-UP ONLY) Header data Relevant Sub-Project(s): ____________________ (from participant list) Location(s): ______________________________ Date: _________________________________ 66  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Interviewer: ____________________________ Respondent sex: M/F Respondent age: up to 18 years/19 to 35 years (not always available from participant list) Survey Questions Initial Interviewer Statement: This survey only has six main questions, with a few follow-up questions for each. The questions mostly ask for three examples of things, but you do not have to provide three. One, two or even no examples are fine, if you do not have more. Question 1. What were the most useful things (if any), that did to help increase your income or business? Question 2. What were the least useful things (if any), that did for you? (Training Participants Only) Question 3a. Did you learn useful things from the training? Y/N 3b. If Yes, please provide examples of the most useful things and how you have already used them? 67  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 (BNE Participants Only) Question 4a. Did you make any useful contacts at the business networking events you attended? Y/N 4b. If Yes, Please provide examples of any new contact you have had with people you met during the meeting opportunities, and how did you use each contact (names not required). Example of Contact How used? Question 5. Of the most important benefits (if any), that you got from , which benefits continue and which benefits have now stopped? Benefits that Continue Benefits that have Stopped 68  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Question 6. What new or improved services or benefits (if any), have you been able provide to others, because of your involvement in ? New or Improved Services/Benefits 2. Case Study Format. For participants who were involved in range of progressive Business Planning Competition (training and related activities) Header data Relevant Sub-Project(s): ____________________ (from participant list) Location(s): ______________________________ Date: _________________________________ Interviewer: ____________________________ Respondent sex: M/F Respondent age: up to 18 years/19 to 35 years (not always available from participant list) Questions Initial Interviewer Statement: This survey only has five main questions, with a few follow-up questions for each. The questions mostly ask for three examples of things, but you do not have to provide three. One, two or even no examples are fine, if you do not have more. Q1. What were the key benefits (if any), that you got from each stage of your involvement in the range BPC activities? 69  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Key Benefits Q2a. By attending more than one part of the BPC program, did you get any extra benefits that you would not have received if you had only attended single activities? Y/N (prompt: combined effects) 2b. If yes, please list any new or increased benefits you received as a result of being involved in multiple BPC activities. Extra benefits Q3. Do you have any suggestions for making the BPC program as a whole more effective? Suggestions 70  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Suggestions Q4. If you received a prize or any other funding opportunity from participating in the BPC program, please provide up to three specific examples of things that this prize or other funding allowed you to do, and state the effect of doing each thing had on your business. Whats done? Effect on Business? Q5. Do you have any other comments you would like to make about the BPC program as a whole? Comments 71  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Comments 2. Higher-level Stakeholder Questionnaire Format (DRAFT MOCK-UP ONLY) Header data Name: ___________________________ Respondent sex: M/F Position: _________________________ Organization: _________________________ Location: ______________________________ Date: _________________________________ Interviewer: ____________________________ Respondent type: (can be more than one) o USAID Staff o MEDA Staff o MEDA Partner Organizations o Local [Libyan] Authorities (formal or informal) o Other USAID Implementing Partner Staff (especially those interacting with LEE) o Other Donor Staff (especially those funding similar programs to LEE) o Other Aid Implementing Agencies (especially those delivering similar programs to LEE) o Clients of Participant Businesses o Other Survey Questions Initial Interviewer Statement: This survey only has less than ten main questions, with a few follow-up questions for each. The questions mostly ask for three examples of things, but you do not have to provide three. One, two or even no examples are fine, if you do not have more. Question 1. What were the most positive aspects of the MEDA/LEE/USAID program (if any)? And what were the effects of these things? Positive Things Effects 72  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Question 2. What were the most negative aspects of the MEDA/LEE/USAID program (if any)? And what were the effects of these things? Negative things Effects Question 3. What suggestions for improvement do you have (if any)? And what would be the effects of these changes? Suggested Changes Intended Effects 73  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Question 4a. In your opinion, did the MEDA/LEE/USAID program increase the business skills of the beneficiaries? Y/N/Don’t Know 4b. If Y or N, why/why not? Question 5a. In your opinion, did the MEDA/LEE/USAID program increase leadership capacity of participating beneficiaries? Y/N/Don’t Know 5b. If Y or N, why/why not? Question 6a. In your opinion, did the MEDA/LEE/USAID program increase formal/informal business linkages? Y/N/Don’t Know 6b. If Y or N, why/why not? Question 7a. In your opinion, was the MEDA/LEE/USAID program an effective contributor to women and youth economic empowerment? Y/N/Don’t Know 7b. If Y or N, Why/why not? Question 8a. In your opinion, was the MEDA/LEE/USAID program an effective contributor to broader enterprise development? Y/N/Don’t Know 8b. If Y or N, Why/why not? Question 9. What were the key factors that posed challenges to the MEDA/LEE/USAID program, and what were their effects? Challenges Effects 74  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 Annex 3. Draft Work Plan Draft Workplan September October November December January February wk1 wk2 wk3 wk4 wk1 wk2 wk3 wk4 wk1 wk2 wk3 wk4 wk1 wk2 wk3 wk4 wk1 wk2 wk3 wk4 wk1 wk2 Document Review Produce Draft Inception Report for Distribution Take Feedback on Inception Report Test Intended Methodology Finalize Inception Report Data Gathering Participant Survey Identify Contactable Participants Conduct Survey Case Study (of Participants of Multiple Business Plan Competition Activities) Conduct Interviews Higher level Stakeholder (HLS) KIIs Obtain HLS lists Conduct Interviews Analysis Translation of Data Gathered Coding of Qualitative Responses Quantitaive Analysis of Coded Data Preliminary Presentation for Recommendations Development Develop Draft of Final Report Final Presentation Submit Draft of Final Report for Feedback Obtain Feedback and Incorporate Feedback Into Report Submit Final Report (USAID feedback plus 7 working days) 2108 2019 2019 75  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 ANNEX 3: FINAL TIMELINE 76  Libya Economic Empowerment Program—Final Performance Evaluation – METAL AID-280-TO-17-00001 ANNEX 4: COMPILED ACTUAL TEXT RESPONSES FOR MORE INFORMATION, PLEASE SEE THE UPLOADED DATA ON USAID’S DEVELOPMENT DATA LIBRARY (DDL).