i GROWTH, ENTERPRISE, EMPLOYMENT, AND LIVELIHOODS (GEEL) END LINE PROCESS EVALUATION EVALUATION REPORT April 2022 DISCLAIMER This report is made possible by the support of the American people through USAID. The authors’ views expressed in this publication do not necessarily reflect the views of USAID or the United States Government. GROWTH, ENTERPRISE, EMPLOYMENT, AND LIVELIHOODS (GEEL) END LINE PROCESS EVALUATION EVALUATION REPORT Contract Title: Evaluations, Assessments, and Analyses (EAA) Indefinite Delivery, Indefinite Quantity (IDIQ) Contract Contract Number: 72062320D00003 / 72062321F00004 Task Order Title: Growth, Enterprise, Employment, and Livelihoods (GEEL) Evaluation Version: Version 1.0 Submitted: April 6, 2022 Contractor: Integra Government Services International LLC 1156 15th St NW, Suite 800 Washington, DC 20005 USAID Office: USAID/Somalia COR: Diba Wako Photo Credit: RTI International. “USAID/Somalia Growth, Enterprise, Employment, and Livelihoods Program.” (2016) DISCLAIMER This report is made possible by the support of the American people through USAID. The authors’ views expressed in this publication do not necessarily reflect the views of USAID or the United States Government. ii TABLE OF CONTENTS LIST OF TABLES IV LIST OF FIGURES IV ACRONYMS V EXECUTIVE SUMMARY 1 1. INTRODUCTION 10 2. EVALUATION DESIGN AND METHODOLOGY 12 3. KEY FINDINGS 18 4. EVALUATION DELIVERABLES TIMELINE 41 ANNEX I: DRAFT DATA COLLECTION TOOLS 42 ANNEX II. ILLUSTRATIVE EVALUATION MATRIX 46 ANNEX III. REFERENCES 48 ANNEX IV. SAMPLING APPROACH 49 ANNEX V. SAMPLE DATA CODING MATRIX 52 ANNEX VI. COVID PROTOCOLS FOR IN-PERSON DATA COLLECTION 54 iii LIST OF TABLES TABLE 1: SUPPORTING AND HINDERING FACTORS FOR TASK ORDER 2 SUCCESS 4 TABLE 2: SUPPORTING AND HINDERING FACTORS FOR TASK ORDER 4 SUCCESS 6 TABLE 3: RECOMMENDATIONS 7 TABLE 4: CHANGE MANAGEMENT 20 TABLE 5: MANAGEMENT AND IMPLEMENTATION FACTORS IN RESPONSE TO SHOCKS 24 TABLE 6: TASK ORDER 4 PERFORMANCE INDICATORS 33 TABLE 7: SUPPORTING AND HINDERING FACTORS FOR TASK ORDER 4 SUCCESS 36 TABLE 9: EVALUATION MATRIX 46 TABLE 10: KEY INFORMANTS (NON-BENEFICIARY) TABLE 11: KEY INFORMANTS (BENEFICIARIES) TABLE 12: ANALYSIS OF CLANS BY LOCATION FOR FOCUS GROUP DISCUSSION 49 49 50 TABLE 13: DATA CODING 52 LIST OF FIGURES FIGURE 1: KEY INFORMANTS BY LOCATION 13 FIGURE 2: KEY INFORMANTS BY SEX AND TYPE 13 iv ACRONYMS ADS Automated Directive Service AMELP Activity Monitoring, Evaluation, and Learning Plan CDCS Country Development Cooperation Strategy COP Chief of Party COR Contracting Officer’s Representative DO Development Objective DQA Data Quality Assessment EAA Evaluations, Assessments, and Analyses FGD Focus Group Discussion FI Financial Institutions GEEL Growth, Enterprise, Employment, and Livelihoods Ha hectares HACCP Hazard Analysis and Critical Control Point IDP Internally Displaced Person IP Implementing Partner IRB Institutional Review Board IRG International Resources Group ISO International Organization for Standardization KII Key Informant Interview IRG International Resources Group MEL monitoring, evaluation, and learning MoAI Ministry of Agriculture and Irrigation MoCI Ministry of Commerce and Industries MoPIED Ministry of Planning, Investment, and Economic Development MOU Memorandum of Understanding PIMS Promoting Inclusive Markets in Somalia SACCO Savings and Credit Cooperative Organization SARIS Somali Agricultural Regulatory Inspection Services SBCF Somali Business Catalytic Fund v SOBS Somali Bureau of Standards SOMINVEST Somali Investment Promotion Office SOW Scope of Work SSI Small-scale Infrastructure SSF Somali Stabilization Fund TA Technical Assistance TO Task Order TOC Theory of Change TPM Third-Party Monitoring VC Value Chain VSLA Village Savings and Loan Association USAID United States Agency for International Development USG United States Government vi EXECUTIVE SUMMARY Integra Government Services International LLC (Integra) and its partner Consilient were contracted under the USAID Evaluations, Assessments, and Analysis (EAA) Indefinite Delivery, Indefinite Quantity (IDIQ) contract to conduct the Growth, Enterprise, Employment, and Livelihoods (GEEL) End Line Process Evaluation, implemented by International Resources Group (IRG), a subsidiary of RTI International. This Evaluation Report describes the background to this task order, the methods used to collect and analyze data to inform the evaluation, and explains the findings, recommendations, and conclusions drawn from the data. BACKGROUND AND PURPOSE USAID/Somalia’s Economic Growth office implemented the five-year, $74,000,000 GEEL Activity (September 2015 - September 2021) to promote inclusive economic growth in Somalia by promoting systemic and sectoral changes in agriculture, fisheries, and livestock by stimulating production, processing, employment, and enterprise development along each of the value chains. The activity comprised four Task Orders: TO1: overall management, performance, and compliance of the activity (e.g., cross-cutting issues, monitoring, evaluation, and learning (MEL) and gender mainstreaming); TO2: systemic and sector￾specific activities for high-potential value chains to promote growth (dairy, livestock, fisheries, sesame, among others); TO3: sustainable energy supplies; and TO4: strengthening youth livelihoods through training and IGAs. This process evaluation examines TO1, TO2, and TO4.1 1 TO 3, which focuses on energy, was not included in USAID/Somalia’s scope for this evaluation. EVALUATION PURPOSE AND QUESTIONS The purpose of this evaluation is to: (i) assess internal factors (e.g., management capacity, resources, MEL systems, local staffing, etc.) and external factors (e.g., local partnerships, host government, regulator environment, etc.) that affected GEEL’s ability to adapt to changing political and economic conditions, as well as other shocks like COVID-19; and (ii) to provide information and analyses that can be used to improve future economic growth, youth and gender programming, and inform implementation of USAID/Somalia’s new flagship economic growth activity. The SOW for this evaluation posed the following four evaluation questions: 1. How did the management of GEEL evolve over time due to internal and external changes? 2. What lessons can be drawn from GEEL’s experience implementing TO2? 3. What lessons can be drawn from GEEL’s experience implementing TO4? 4. How can GEEL’s implementation experience be leveraged by USAID in its new strategic pivot under its 2020 - 2025 Country Development Cooperation Strategy (CDCS)? METHODOLOGY The evaluation team used a mixed-methods approach, combining qualitative data collected through Key Informant Interviews (KIIs) and Focus Group Discussions (FGDs), quantitative data collected from implementing partner monitoring reports and surveys, and quantitative and qualitative information 1 collected from implementing partner studies and assessments as well as evaluations and documentation of similar programs. A two-tiered sampling approach was used to identify and select individuals to target for inclusion. The initial sampling approach is purposive followed by snowballing. It is important to note that more than 90 percent of respondents were identified by the evaluation team ahead of time through purposive sampling, and only the remainder were identified via snowball sampling. In all, 60 KIIs and five FGDs were carried out with 18 of the KIIs remote and the remainder conducted in person. Remote KIIs with USAID/Somalia Mission staff and GEEL project staff were conducted using Google Meet and Zoom. The remaining respondents were sampled from four primary locations: Mogadishu, Jowhar, Baidoa, and Afgoye. A detailed approach to this sampling, broken down by respondent category and location, is included in Annex IV. Annex IV also includes an analysis of clans and their relationships, which was used to ensure that KIIs and FGDs were inclusive of marginalized groups. CONCLUSIONS This section presents findings by evaluation question. The fourth evaluation question has been combined into an overall recommendations section, because the recommendations for the new CDCS derive directly from the recommendations produced through the analysis for the first three EQs. EVALUATION QUESTION 1: HOW DID THE MANAGEMENT OF GEEL EVOLVE OVER TIME DUE TO INTERNAL AND EXTERNAL CHANGES? Internal factors will refer to both management (management capacity and structure, resource levels, MEL systems, staffing, and USAID/Washington-driven changes, such as the pivot to resilience) and implementation (local partnerships, relationships with host government, engagement processes), whereas external factors refer to exogenous (to the program) changes, such as political and economic fluctuations, weather shocks, and the emergence of COVID-19. Advantages of local leadership and technical expertise: The hiring of a local chief of party (COP) and additional local technical staff was beneficial to the credibility and success of the program. Though the engagement of an acting COP allowed GEEL to begin operating in the early months, GEEL and USAID informants reported that the acting COP did not have a good relationship with local stakeholders and a good portion of the initial technical team, which was manned largely by expatriates. Sufficient lead time is needed to recruit specialists who are critical to project success: The Gender and Youth Advisor (GYA) was supposed to come on board in Year 2 to implement the gender strategy, but the position was not filled. While a consultant was hired in Year 3 to fill the gap, the gender strategy, and therefore its benefits, were delayed by not being able to staff the position. Theories of change and performance measures should be agreed upon at the start of the program: The theory of change (TOC) for the activity was not developed until Year 2, and the activity monitoring, evaluation, and learning plan (AMELP) was not approved until Year 2. The TOC should inform the design and implementation of each component of an activity while the AMELP provides a framework for measuring progress and identifying needs for adaptation. Without these tools in place, the project could not systematically collect data that could have informed adaptive management and implementation, thus missing out on activities or opportunities that could have improved implementation and encouraged more participation of women and youth. 2 Changes in program structure should be accompanied by updated management structures, performance measures, and theories of change: TO4 (Somali Youth Integrated Livelihoods) was introduced during the final year of the program. One management challenge presented by this change, identified by two USAID informants, was that TO4 was managed jointly with the Social Services (now Education) Office. According to the informants, this left TO4 somewhat outside the structure of GEEL and there was some stovepiping of information and decision-making which was apparent primarily as sub￾optimal coordination on data collection, information sharing, and the learning agenda. Consider the impact of design on the ability to reach target populations: Based on feedback from GEEL staff and participants, changes were made to the grantmaking process in Year 2, including an increase in in-kind grants and a focus on technical assistance. This was a response to the activity’s difficulty in engaging youth and women-owned businesses under its original design. Flexibility provided by USAID is critical: Probably the most oft-cited lesson learned was that the key to effective adaptive management under GEEL was the flexibility that USAID allowed GEEL staff to explore different ideas in the face of challenges, including making suggestions based on GEEL activities’ ongoing learning efforts. Several GEEL staff interviewed noted that they had strong relationships with the Contracting Officer’s Representative (COR) and were able to discuss ideas that arose through ongoing efforts at learning under GEEL’s CLA Strategy. They believed that these ideas received consideration, and they often got permission to implement them. Several GEEL and USAID respondents cited delays early in the program in fixing management and implementation challenges which were relieved through more regular communication and a stronger relationship with the COR. EVALUATION QUESTION 2: WHAT LESSONS CAN BE DRAWN FROM GEEL’S EXPERIENCE IMPLEMENTING TO2? A. DID GEEL ACHIEVE ITS PRIVATE SECTOR DEVELOPMENT OBJECTIVE? TO2 is focused on implementing sector-specific activities to promote the development of five high￾potential value chains (VCs): dairy and livestock, sesame, fisheries, banana, and fruits and vegetables. It is clear that GEEL achieved its objective under TO2. The project was engaged in systemic and sector-specific activities in dairy/livestock, sesame, fisheries, bananas, and fruits and vegetables. KIIs and the quarterly and annual reports are filled with testimonials and evidence. GEEL also succeeded in partnering with sector businesses to match investment and to encourage local financiers from banks and micro-lending institutions either to directly finance or co-finance through blended approaches with GEEL and other partners. Table 1 provides a summary of the supporting and hindering factors in the success of TO2. 3 TABLE 1: SUPPORTING AND HINDERING FACTORS FOR TASK ORDER 2 SUCCESS SUPPORTING FACTORS Leadership, local experience, and consistency GEEL got off to a slow start. On top of the implementation challenges, project staff had not established relationships with the stakeholders therefore the feedback loop to make critical changes was not established early in the project. The early slow start to the program was overcome once the project was staffed with a local COP and a diverse team of local and international staff with relevant experience. Relationships were key to navigating a complex environment and managing internally and externally driven changes. Partnership with local businesses requires familiarity and trust, which takes time and effort. Focused on “lead firms,” GEEL established relationships by learning about their operations, knowing their constraints, and working with the firms to address capital equipment needs, improve processes, and learn compliance to international standards. When the focus shifted to resilience in the face of drought and pestilence, GEEL drew on its network of relationships, knowledge of market actors for guidance, including on how to adapt strategic alliances to most help the displaced, vulnerable, and marginalized. Adaptive Management￾Flexibility and responsiveness from the GEEL and USAID teams were critical to the success of all components. The initial approach of GEEL was to match investment with rather large grants ($500k). It became clear early on that there were limited existing firms who could absorb such an amount easily, given the requirement to match at least that amount. When the requirements were relaxed to allow smaller matches and less restrictive terms, GEEL was able to distribute many more grants to medium or even small businesses. These adjustments also helped reach more businesses owned or operated by women. Learning from earlier programs Immediately preceding and/or parallel to the implementation of GEEL, the DFID￾funded Promoting Inclusive Markets in Somalia (PIMS) project, and the World Bank-sponsored Somali Business Catalytic Fund (SBCF) project also engaged private sector/market systems and investment programs through matching grant and technical assistance modalities. GEEL also participated early on in key convenings such as the Somali Renewable Energy Forum, developed and implemented by NGO Shuraako. 4 Targeted technical assistance, (vocational training, capacity building, advisory services), rather than just capital was critical to business success. As GEEL solicited the participation of beneficiaries, increasing requests were made for targeted advisory services or technical assistance that was specialized, business and/or sector-specific. KIIs from this process evaluation indicate that the types of support services being sought were not readily available in Somalia. GEEL therefore was tasked to identify and mobilize providers that had specific knowledge to meet the needs of the businesses. One example of this dilemma was in the fisheries sector. GEEL saw that hygienic handling and preservation of catch, as well as proper processing and packaging to any sort of standard, largely did not exist in the market. In an effort to link fisheries beneficiaries to export markets, GEEL identified prominent food fairs, including the Dubai Sea Food Fair, then mobilized participation for fishing firms to attend trade and food fairs. In the case of their resilience work, where it required addressing yields of crops to mitigate food insecurity, the use of certified seeds and quality inputs was critical. GEEL was able to support companies who developed certified seeds and, in the process, worked with the policymakers on specific delivery of key services to support national frameworks that could provide oversight, like Somali Agricultural Regulatory Inspection Services (SARIS), among others. HINDERING FACTORS Complex Operating Environment Somalia is a highly complex operating environment that is vulnerable to various shocks (i.e., violent extremism, inter-clan, or other tribal line conflicts, climate/draught, political volatility, economic uncertainty, public health/pandemic outbreaks, inadequate education and skills training) Low Management Capacity to engage donor Even formal businesses in the Somali context are often unfamiliar with standardized management, accounting, or quality control processes and are heavily burdened with the processes required to achieve a matching grant or secure technical support. Impacts of Humanitarian Assistance Humanitarian organizations inadvertently undermine certain development and/or market systems objectives, for example by distributing food supplies, which disrupts local markets. This can have unintended consequences for local communities. EVALUATION QUESTION 3: WHAT LESSONS CAN BE DRAWN FROM GEEL’S EXPERIENCE IMPLEMENTING TO4? Task Order 4 (Somali Youth Integrated Livelihoods) was a component of GEEL that was implemented in the final, extension year of the activity. It was intended as a complement to TO2, increasing participation of women and youth in training and economic activity. Despite the challenges presented by COVID-19’s emergence shortly afterthe launch of TO4, the activities’ objectives above were largely achieved. Data from the soon-to-be-completed Mission Portfolio Performance Review shows that targets were largely met. Given the short duration of this component however, it is important to note that conclusions based on data gathered for TO4 are based more on snapshots than longer term observations. They can nonetheless provide important information for the design and piloting of such activities. 5 TABLE 2: SUPPORTING AND HINDERING FACTORS FOR TASK ORDER 4 SUCCESS SUPPORTING FACTORS 1. Collaboration with stakeholders in COVID-19 response. 2. Market-driven training in addition to meeting the need for skills in several sectors, by connecting youth with job opportunities. These efforts helped overcome the influence of social networks in favor of more merit-based hiring. 3. Better understanding of skills mismatches gained through TO4/TO2 collaboration—GEEL leveraged VC partners from TO2, including BECO, Somalia’s largest utility. 4. Connections between training, internships, and employment opportunities through private sector collaboration. This included career counseling and other services for job placement, all in collaboration with TO2 partners and others. 5. Extensive partnerships with government and other stakeholders, including Agriculture Youth Partners and Banadir Regional Administration. This included capacity building of partner organizations to carry out training. a. Literacy and Numeracy Assessment b. Under partnership with the City University of Mogadishu, two greenhouses and a poultry farm were provided to meet the need for more practical agricultural training. 6. Engagement on platforms likely to reach youth, including more digital outreach and social media—these efforts helped increase access to information about job opportunities, making it accessible to a broader range of potential applicants. 7. Youth involved in the program proactively engaged others through the Somalia Agricultural Girls Association, which increased awareness and participation young women (Final Report, TO4). 8. Building on existing skill sets and/or sectoral involvement. Although it is important to expand opportunities in new sectors, it is also important to build on existing viable foundations. With 98 percent of milk distributors being youth and/or women, GEEL focused on training in handling and business management, including the provision of iceboxes to increase shelf-life during transit (TO4 Work Plan, 2020). HINDERING FACTORS 1. Lack of opportunity, even for the educated. This was identified as a challenge by GEEL studies, several youth focus group participants (male and female), and by respondents from both GEEL and USAID in KIIs. 2. COVID-19 restrictions on movement and other impacts such as export restrictions and the like, which affected job opportunities and overall economic activity, with young people who are new to their positions or to business most likely to feel the immediate effects of shocks. 3. Short duration of activity and lack of focused effort to capture lessons learned through follow-up data collection. 6 RECOMMENDATIONS: EVALUATION QUESTION 4 HOW CAN GEEL’S IMPLEMENTATION EXPERIENCE BE LEVERAGED BY USAID IN ITS NEW STRATEGIC PIVOT UNDER ITS 2020 - 2025 COUNTRY DEVELOPMENT COOPERATION STRATEGY (CDCS)? The new CDCS contributes to the overall U.S. Government policy priority of preventing and countering violent extremism. Lessons learned from GEEL will be applied to the upcoming economic growth activity which will support both Development Objectives (DOs) in the new CDCS: DO1 – Targeted local institutions govern in a more legitimate manner, diminishing influence of violent extremist organizations and DO2 – Enable marginalized Somalis to more effectively withstand shocks and stresses. Table 3 details the recommendations for applying GEEL lessons learned to programming under the new CDCS. TABLE 3: RECOMMENDATIONS 1. Do the groundwork to determine which skill sets and commodities are the best investment of program resources. Optimal investments depend on a variety of factors, including target populations and geographic priorities. With the focus on IDPs and other vulnerable populations, donor investment in the most promising value chains from a private sector growth perspective, though an important consideration, should not crowd out ones that create the most opportunity for target populations. As an example, under GEEL, fish processing and milk distribution were determined to be areas where women and youth were engaged and thus became priorities for investment. Also consider skill sets outside more traditional economic growth sectors. In the focus group discussion among young women in Mogadishu, all participants agreed that training on skills such as tailoring or beauty services would be beneficial to their longer-term economic prospects. Tailoring and beauty services are rather common and are not necessarily scarce or high growth, though beneficiaries see within each of these vocations’ potential for a marketable, potentially portable business that they can feasibly grow into a newer more profitable market or in times of displacement, reestablish elsewhere. 2. Engage host government and other stakeholders on land and other input availability in target areas. Greenhouses were a successful intervention under TO4, and respondents in both male and female youth focus groups indicated that they valued the training and equipment they received through these interventions. Greenhouses and other in-kind support to productive capacity require land, however, which is especially challenging when dealing with IDP populations. Respondents from GEEL indicated that the federal government is exploring options for providing land to IDPs in the Mogadishu area. Any follow-on activity should carefully explore the status of those efforts and any alternatives when considering how best to support those populations. This is also likely to be true for urban poor youth. 3. Timing of interventions is critical to success. As a high-level government official in Baidoa emphasized in an interview, technical assistance and in-kind support that do not sync up with growing seasons can reduce the benefits to participants. With proper timing, participants can apply learning and inputs in real time, maximizing the benefits of training. In the worst cases, missed opportunities due to poor timing or delays can affect the ability of smallholders and small business owners to repay loans. 4. Maintenance and operation of capital investments should be considered during design. 7 As noted in EQ 3, although FGD participants were generally happy with the in-kind support they received, one participant noted that there were problems with some of the drip irrigation systems GEEL installed and that the lack of follow up by the program led to delays in fixing these problems. Although they did not report similar problems with greenhouses, greenhouses also require maintenance, and any program that supplies these types of infrastructure investments must consider longer term upkeep and operation, if the program’s benefits are to be sustainable. 5. Look for lessons in other programs, including USAID's and those of other donors. As noted in EQ 2, GEEL benefited from coordination with PIMS and other existing programs and from their respective lessons learned. With an early shift in GEEL’s approach, stepping away from its initial go-it-alone approach to instead collaborate with PIMS rather than competing with each other in support of mutually identified key value chains, each was able to contribute according to their comparative program strengths, thereby achieving more impactful outcomes. Likewise, GEEL not only improved its ability and willingness to learn from other development partner programs, but also became a main source of partnering and collaboration. GEEL was known for sharing and supporting for example the SSF II efforts, BRICS and SomREP efforts. For future programming, it is important to evaluate the donor landscape to identify both potential partners and learning opportunities with existing programs to increase coordination, collaboration and partnerships. 6. Know whom you are talking to. Elite capture is always a risk with donor programs and is especially difficult to avoid in complex, dynamic environments, where donors are relatively inexperienced. It is critical to do the groundwork to make sure that engagement processes are truly reaching representative samples of target populations. Annex IV provides a preliminary clan analysis that may be useful for future programming efforts. 7. Map out a focused learning agenda based on a Theory of Change and AMELP. The learning agenda can evolve, as it did with GEEL during the pivot to resilience, but it is important to have a clear learning agenda that has been shared with both the implementer and any third-party monitoring implementers. This can help guide reporting and ensure that activities are collecting data on the outcomes that are most important to the learning agenda. As adaptations are made to the activity, the learning agenda, AMELP, and TOC should be updated as needed, ideally through discussions with relevant stakeholders. 8 8. Make technical training more accessible to urban poor youth and IDPs. This is especially challenging under COVID restrictions, as these populations are least likely to have access to virtual learning options. As was the case with greenhouse training, activities should identify which, if any, training is most critical to conduct in-person, and ensure that proper facilities and PPE are available for participants. Make use of tools, such as the literacy and numeracy assessment tool developed by GEEL, to identify the most urgent training needs of target populations. 9. Keep it simple. For resilience-focused activities, it is not necessarily innovation that counts. Focusing on existing market activities and how they can be adapted to vulnerable or marginalized populations is the best place to start. For example, we know that access to finance is a major barrier to entry for marginalized groups. Group lending activities are a tried￾and-true method for bolstering resilience and saw success under GEEL. They are especially helpful for groups that do not have access to finance through more mainstream financial institutions. 10. Fortify group lending and mutual support group effectiveness through creative risk sharing schemes with local FIs and organizational strengthening efforts for self-help groups. 9 Beneficiaries requiring resilience support are often informal earners pursuing basic livelihoods or informal businesses with few options for formal finance. GEEL found that its work with local financial institutions during their COVID response and resilience work phase made rapid impacts and lasting success. GEEL supported lending to ‘groups’ such as self-help Ayuda/Hagbad, SACCO'S and VSLA’s and worked with local banks to provide a risk sharing grant that banks were to match X a multiple in lending capital and target those most vulnerable communities. This fortified access to operating funds for businesses of marginalized communities and reinforced the mutual support ‘ecosystem’ of group members taking care of other group members in a shock when these groups could continue to cycle critical funds in a time of most needs. When supporting groups, it also allows the less literate or less formal benefit from others in the group who are, and by virtue of group association, on a basic level begin to bridge a gap from informality into at least basic formality (linkage to a bank or lender) and thereby increasing the likelihood of formal savings hence a reserve in future shocks. 1. INTRODUCTION Integra Government Services International LLC (Integra) and its partner Consilient were contracted under the USAID Evaluations, Assessments, and Analysis (EAA) Indefinite Delivery, Indefinite Quantity (IDIQ) to conduct the Growth, Enterprise, Employment, and Livelihoods (GEEL) End Line Process Evaluation, implemented by International Resources Group (IRG), a subsidiary of RTI International. This Evaluation Report describes the background to this task order, the methods used to collect and analyze data to inform the evaluation, including challenges and limitations, and explains the findings, recommendations, and conclusions drawn from the data. 1.1 PROJECT BACKGROUND The five-year, $74,000,000 USAID/Somalia GEEL Activity was intended to promote inclusive economic growth in Somalia by promoting systemic and sectoral changes in agriculture, fisheries, and livestock by stimulating production and processing, employment, and enterprise development along each of the value chains. It comprised four task orders (TOs): ● TO1 focused largely on overall management, performance, and compliance of the activity, as well as cross-cutting issues, including monitoring, evaluation, and learning (MEL) and gender mainstreaming. ● TO2 focused on systemic and sector-specific activities along high-potential value chains to promote growth: dairy and livestock, sesame, fisheries, banana, and fruits and vegetables. This included efforts to improve access to finance and work through the private sector to extend technologies and practices for smallholder farmers. ● TO3 focused on supporting sustainable energy supplies.2 ● TO4 focused on strengthening the ability of Somali youth to engage in the economy as entrepreneurs, employers, and employees. 2 At the request of the Mission, this TO was not included in the evaluation. This evaluation focuses on assessing processes applied that relate specifically to TO1, TO2, and TO4. The following section presents how Integra assessed each of these TOs in terms of which processes worked well (or not) and why, identifying any bottlenecks and inefficiencies that could be improved to inform future economic growth programming in Somalia. 1.2 EVALUATION PURPOSE AND EVALUATION QUESTIONS The purpose of this evaluation is to assess internal factors (e.g., management capacity, resources, MEL systems, local staffing, etc.) and external factors (e.g., local partnerships, host government, regulator environment, etc.) that affected GEEL’s ability to adapt to changing political and economic conditions, as well as other shocks like COVID-19. It is also intended to provide information and analysis to improve future economic growth, youth, and gender programming and to inform the implementation of USAID/Somalia’s new flagship economic growth activity. This evaluation is not a performance evaluation, but rather a process evaluation, focused on assessing processes GEEL used to adapt to Somalia’s dynamic context, implement systemic and sector-specific 10 activities to promote the development of high-potential value chains, and improve the livelihoods of youth by increasing their participation in the economy as entrepreneurs, employers, and employees. USAID posed the following evaluation questions to learn more about how three out of four of GEEL’s TOs (TO 1, 2, and 4) were implemented, what worked well and why, what were the time lags, bottlenecks, and inefficiencies, and how it could have been improved. EQ 1. How did the management of GEEL evolve over time due to internal and external changes? 1.1 What were the major lessons learned from these changes and the change management processes? 1.2 What lessons can be drawn for USAID in managing complexity-aware programming and allowing for adaptive management? 1.3 How did internal factors (e.g., management capacity, resources, MEL systems, local staffing) and external factors (e.g., local partnerships, host government, regulatory environment) affect GEEL’s ability to adapt to changing political and economic conditions? EQ 2. What lessons can be drawn from GEEL’s experience implementing TO2? 2.1. Did GEEL achieve its private sector development objective? 2.2. What were the main supporting and hindering factors for achieving this objective? EQ 3. What lessons can be drawn from GEEL’s experience implementing TO4? 3.1 Did GEEL achieve its youth livelihood objective? 3.2 What were the main supporting and hindering factors for achieving this objective? EQ 4. How can GEEL’s implementation experience be leveraged by USAID in its new strategic pivot under its 2020 - 2025 Country Development Cooperation Strategy (CDCS)? 11 2. EVALUATION DESIGN AND METHODOLOGY A process evaluation focuses on a project’s procedures, policies, management approaches, and other elements related to how a program was implemented. Although it does not directly address performance, the results can be used to improve implementation and therefore, the performance of future projects. This process evaluation for GEEL is intended to inform the design and implementation of future activities focused on resilience and inclusive economic growth under USAID/Somalia’s new CDCS. In particular, this evaluation will focus on how the management, processes, and procedures of the program affected its ability to adapt to internal and external changes. The processes and approaches to engagement with target populations are also examined to understand how better to engage youth, women, and other marginalized groups in the follow-on activity. The evaluation team used a mixed-methods approach, by combining qualitative methods to gather primary data with quantitative methods to be used in review of secondary data sources (i.e., implementing partner monitoring reports, studies, and assessments). Although the SOW for this activity cited the possibility of consulting an Institutional Review Board (IRB), the team determined, in consultation with USAID, that the approach included sufficient privacy safeguards for respondents, so an IRB would not be necessary. 2.1 DESK REVIEW PROCESS The ET began by reviewing project documentation and assessments provided by USAID/Somalia. Additional materials were added throughout the course of the evaluation, including studies recommended internally by team members and others the team became aware of during KIIs. 2.2 KEY INFORMANT INTERVIEWS The evaluation team split KIIs into several categories: • Non-beneficiary private sector: Private sector and CSOs that were either involved with the program as partners or were considered to have potentially valuable insights into the program; • Beneficiaries: stratified by demographic characteristics (age, gender) and value chain participation; • USAID/Somalia Mission staff; • GEEL project staff; and • Government representatives. A two-tiered sampling approach was used to identify and select individuals to target for inclusion from the beneficiary lists provided by the IP. The initial sampling approach used in the process evaluation was purposive sampling followed by snowballing. It is important to note that more than 90 percent of respondents were identified by the evaluation team ahead of time through purposive sampling, with identified via snowball sampling. 12 Using guides developed during the inception phase of the evaluation, the team conducted 60 semi￾structured interviews, both in-person and remote. It should say Figures 1 and 2 show breakdowns of informants by geography, sex, and type. FIGURE 1: KEY INFORMANTS BY LOCATION Afgoye Baidoa Jowhar Mogadishu 13% 13% 12% 62% FIGURE 2: KEY INFORMANTS BY SEX AND TYPE 35 Benficiaries Non-Beneficiary Government USAID GEEL Private Sector M F 0 5 10 15 20 25 30 The data collection tools are provided in Annex I, and are broken up by stakeholder group (e.g., program participant, implementing partner, private sector, etc.). Eighteen of the KIIs were conducted remotely, and 42 were conducted in-person in Mogadishu, Afgoye, Baidoa, and Jowhar. A list of key informants by type and affiliation may be found in Annex IV. All identifying information has been removed and replaced with unique identifiers to mitigate privacy concerns. Interviews were recorded with the permission of the 13 respondent and both the recordings and written summaries will be shared with USAID separately from this report, as there is identifying information in the summaries. Of these, two discussion groups took place in Mogadishu and sought to capture perspectives of youth (ages 15-24)3 and to encourage free dialogue groups that were homogenous with respect to participant gender — one with males and the other with females. Another FGD was carried out in Baidoa, specifically looking to capture perspectives of female beneficiaries of all ages. In Afgoye, private sector beneficiaries representing male and female participants were invited to discuss their experiences with GEEL and their perceptions of impediments and supporting factors that have impacted their success. The fifth FGD was held in Jowhar, with three male and three female sesame value chain actors. 3 USAID’S 2012 Youth in Development Policy defines youth as ages 10-29 but GEEL focused on ages 15-24 to best capture youth who were completing their education and entering the job market. https://www.usaid.gov/policy/youth 2.2.1 CONSIDERATIONS FOR IN-PERSON DATA COLLECTION Interviews were conducted by locally based researchers, to ensure language and dialect compatibility with respondents. Introductions with program beneficiaries and other stakeholders were facilitated by the GEEL team and USAID for government stakeholders, but all in-person interviews were arranged and conducted by the evaluation team to ensure respondents felt they could speak openly. Informants were also informed of their right to refuse to participate. The team tracked interview progress with a spreadsheet that was shared across the evaluation team to ensure timely completion and to be able to adjust schedules quickly, as needed. Further, each researcher followed a fieldwork plan that included logistical information and the target number of KIIs/FGDs for their respective location(s). 2.2.2 CONSIDERATIONS FOR REMOTE DATA COLLECTION In addition to the in-person interviews, remote interviews were conducted by other members of the evaluation team, including the Team Lead, Senior Technical Advisor, Local Gender Expert, and Researcher with stakeholders willing and able to participate. Google Meet and Zoom (for non-USAID staff only), were used for remote interviews, depending on the preference of the respondents. Team members obtained oral consent from respondents to record the interview by reading a short paragraph that introduced the interviewer, explained the purpose of the discussion, and established the time frame for completing the questions. Respondents were selected based on recommendations from USAID, GEEL staff, identification from the literature review, and through discussions with other respondents. Below is a list of respondents by category. USAID/Somalia Mission staff including Office Directors and Deputies, Contracting Officer’s Representative (CORs), Gender and Youth advisors, and other relevant sectoral and technical specialists; • Project staff: Chief of Party, technical staff, and M&E leads; • Private sector beneficiary companies selected from each of the value chains; • Private sector beneficiary companies applied, but not selected, from each of the value chains; • Civil society, private sector organizations and other relevant organizations/partners (i.e., Chamber of Commerce, business or trade associations, and investment promotion companies); 14 • Local government agencies: Ministry of Planning, Investment, and Economic Development (MoPIED), Ministry of Commerce and Industries (MoCI), Ministry of Agriculture and Irrigation (MoAI), Somali Bureau of Standards (SOBS), Ministry of Fisheries, and Somali Investment Promotion Office (SOMINVEST); • Local financial institutions: Dahabshiil, IBS Bank, Premier; and • Gargaara Fund. 2.3 FOCUS GROUP DISCUSSIONS The evaluation team conducted a total of five FGDs across all site locations. As stated earlier, focus group participant categories included: 1) women, 2) youth (one female and one male group), 3) private sector representatives, and 4) sesame value chain actors in Jowhar. The team used the clan analysis provided in Annex IV to ensure inclusion of minority clans in focus groups. 2.4 DATA ANALYSIS This end line process evaluation includes an Evaluation Matrix (Annex II) that maps the EQs to data sources and data analysis approaches. The ET collated data from the document review and interviews, then analyzed it to develop the findings to answer the EQs. Interview and FGD data were organized in matrices where interviewers recorded the information derived from the discussions. These matrices include characteristics of respondents such as sector (e.g., private, government, civil society), location, name of interviewer, and whether the interview was remote or in-person. Analysis included tables and figures showing the demographics and other characteristics of respondents by location and the activities they participated in (for beneficiaries). Analysis also includes performance reporting for relevant TOs and comparisons to targets for key indicators for TO2 and TO4. Quantitative analysis compared outcomes to targets, cross-tabulating where possible (since reporting does not include raw data, disaggregation by sex and other characteristics is only possible when it is specifically reported.) Both qualitative and quantitative analysis was carried out using Microsoft Excel. Verification was carried out through triangulation among multiple data sources, both qualitative and quantitative (when there is relevant quantitative data), and included follow-up calls to select respondents. The Team Lead assigned responsibility for recording data to team members during the course of fieldwork. A matrix like the one in Annex V was used for uniform entry of interview data. A master copy of all interview data was maintained by the Team Lead, and all team Members were able to access the data for analysis. This data will be provided to USAID in proper format (per USAID policy) at the conclusion of the evaluation. Primary Data collection: Qualitative methods used in this evaluation included key informant interviews (KIIs) and Focus Group Discussions (FGDs). The ET carried out 60 interviews with beneficiaries, the private sector, USAID/Somalia Mission staff and GEEL project staff. Eighteen of the interviews were conducted remotely using Google Meet and Zoom. The remaining respondents were interviewed in-person and drawn from four primary locations: Mogadishu, Jowhar, Baidoa, and Afgoye. In addition to the KIIs, the study team carried out five FGDs across all of the aforementioned primary locations. Beneficiaries were targeted for inclusion, specifically focusing on capturing perspectives from youth, women, and private sector actors involved in a high-potential value chain. To ensure dialogue for rich data, some focus groups were 15 homogenous for gender and age. A detailed approach to this sampling, broken down by respondent category and location, is included in Annex IV. As noted at the inception stages of this evaluation, these locations represent relatively high-intensity geographies for GEEL programming (most beneficiaries appear to operate within these cities and districts). At the same time, their size – both in terms of population and economic activity – is such that government offices, civil society organizations, and private sector representatives (particularly financial institutions) are all present and operating. Location samples largely overlap between the beneficiary and non-beneficiary groups. 2.5 DATA LIMITATIONS, CHALLENGES AND RISK MITIGATION Some of the discussions involve sensitive data. Investment matching or grants to larger companies were complex and tied to specific contributions. Deeper discussions, which can be sensitive or that relate to proprietary information, limited the amount or depth of data collection possible at the time of the evaluation. To mitigate any reluctance among participants to take part in interviews or speak openly, Integra offered multiple options for platforms to connect for interviews (e.g., telephone, Google Meet, Zoom, etc.), allowing interviewees to choose which they were most comfortable with. Instruction was provided to informants for accessing communication platforms as needed. Further, to mitigate potential bias introduced by researchers, they alternated roles in teams of two during data collection, with one serving as interviewer and the other as notetaker. Each team of two documented, reviewed, and agreed on the notes together. The notes are not transcriptions, but rather summaries of the main points discussed and the responses. The notetaker and interviewer also recorded their own perceptions and observations. Audio recordings were another safeguard against interviewer bias, because any disagreements between the notetaker and interviewer could be reviewed from the recording and, if need be, translated to enable other team members to provide their input. An additional potential source of bias is found in participant selection, particularly since this evaluation took place well after the end of the program. It is possible that participants who experienced success would be more likely to agree to an interview regarding GEEL and/or that they would be easier to find due to increased stability through employment or entrepreneurship. Although it is not possible to completely eliminate the effects of or risk of such bias, the research team took steps to minimize the impact. The first step was choosing a diversity of locations to conduct KIIs and FGDs, because respondents of different backgrounds and different kinds of involvement with GEEL are less likely to have a consistent bias in their responses. In addition, snowballing allowed the team to track down participants who may have moved or changed contact information since their GEEL involvement, thus mitigating the risk that the team would over￾sample successful respondents. Also related to the selection of geographic focus locations, it is clear that female respondents were not as well represented in the sample as they were in the program. Some reasons for this were the difficulty in reaching female respondents to request their participation and that the choice of geographic location sometimes maximized beneficiary populations overall but at the expense of representation. Additional time for field work could have helped to remedy this but at a substantial cost to the budget. The ET attempted to mitigate this to the extent possible through female participation in focus groups, including a female only focus group. 16 For more general risk mitigation, Integra uses an International SOS service to provide our field teams and staff with the most up-to-date information regarding ongoing and developing situations related to their health, safety, security, unfolding political events, and natural disasters. Integra was also prepared, if the security situation required it, to take other measures such as the provision of satellite-based communication devices and additional vehicles, although fortunately, these measures were not necessary during this activity. Team members traveling in Somalia to conduct KIIs and/or FDGs performed daily check-ins with the Team Lead during fieldwork, primarily through email or through the team’s regular check-in calls. All COVID-19 protective measures were implemented for in-person KIIs or FGD. A copy of these protocols has been included in Annex VI, including social distancing of six feet. Protective equipment, such as masks and hand sanitizer, was provided to both the interviewers and the respondents. In addition to these protocols, all government guidance and regulation was followed. 17 3. KEY FINDINGS 3.1 EQ 1: HOW DID THE MANAGEMENT OF GEEL EVOLVE OVER TIME DUE TO INTERNAL AND EXTERNAL CHANGES? For this question, the evaluation team examined several changes that occurred during GEEL, both internal to the program and external. Internal factors will refer to both management factors (management capacity and structure, resource levels, MEL systems, staffing) and implementation factors (local partnerships, relationships with host government, engagement processes, the pivot to resilience), and external factors refer to exogenous (to the program) changes such as political and economic fluctuations, weather shocks and the emergence of COVID-19.4 External factors are addressed in part c of the response. 4 The Ministry of Health announced the first confirmed case of COVID-19 in Somalia on March 16, 2020, which was followed by the expected closures and restrictions on movement and activities (Final Report, Task Order 4, 2021). Although this question is not explicitly focused on TO1, TO1 is the management and oversight TO for all the other TOs, so requirements and reporting on TO1 inform the response to this EQ. Evidence for this question is drawn primarily from KIIs with USAID and GEEL staff, as well as project documentation detailing external changes, GEEL’s response, and related management and implementation approaches. 3.1.1 WHAT WERE THE MAJOR LESSONS LEARNED FROM THESE CHANGES AND THE CHANGE MANAGEMENT PROCESSES? Advantages of local leadership and technical expertise: Some of the clearest lessons arose from changes that took place early on in the program, such as staff turnover, the hiring of a local COP, and the hiring of additional local technical staff. Though the engagement of an acting COP allowed GEEL to begin operating in the early months, GEEL and USAID informants reported that the acting COP did not have a good relationship with local stakeholders and a good portion of the initial technical team, which was manned largely by expatriates. Although the technical team was strong in subject-matter expertise, they did not have local Somali experience and lacked the fundamental understanding of the culture, did not have existing networks of linkages to local partners, and overall were not versed in how to engage beneficiaries and Somali stakeholders effectively. According to project reporting and KIIs with USAID and GEEL staff, with the local COP on board, the program was better able to recruit locally knowledgeable technical experts. This highlights the importance of clearly defining and prioritizing the necessary skill sets and experience required for successful startup and implementation, including the appropriate mix of local and international leadership and technical expertise. In GEEL’s case, bringing on a knowledgeable Somali COP improved local staff recruitment and retention, ensuring that GEEL staff would be able to have more in-person engagement with beneficiaries and stakeholders and provide a Somali face to GEEL activities. Sufficient lead time is needed to recruit specialists who are critical to project success: GEEL planned to implement its Gender Strategy in Year 2 of the program, in keeping with its original work plan, although implementation was ultimately delayed by difficulties in recruiting a Gender and Youth Advisor (GYA) which was planned for Year 2 but not filled. A short-term consultant was brought in in Year 3 (Annual Report, Years 2 and 3). The GYA was to provide internal training to GEEL staff on gender and engaging women and youth and to review reviewing current activities to see how they could be adapted 18 to be more appealing and accessible to those demographics (Annual Reports Year 2 and 3; Work Plan, Year 2). The gender strategy and its benefits were delayed by not being able to staff the position (USAID and GEEL staff interviews; Women-only FGD). Theories of change and performance measures should be agreed upon at the start of the program: The TOC for the activity was not developed until Year 2, and the AMELP was not approved until Year 2. TOCs are critical to informing both implementation and measurements of progress, and AMELPs are needed to track progress and identify needs and opportunities for adaptive management. A rigorous TOC would have helped avoid some of the pitfalls encountered during early implementation. In particular, it could have identified the disadvantages of the original, more hands-off approach, which relied on fairly large-scale grants, intended to leverage additional funding with minimal management burden. The process presented reporting and application burdens that could not be met by smaller or informal businesses. This led to a missed opportunity to engage women and youth in critical activities. In addition to the lack of a TOC, the lack of an AMELP and a baseline also undermined the ability to measure progress early. Without an approved AMELP in place, the project was unable to systematically collect data that would have been able to inform adaptive management and implementation, thus missing out on activities or opportunities that could have improved implementation and encouraged more participation of women. Changes in program structure should be accompanied by updated management structures, performance measures, and theories of change: TO4 (Somali Youth Integrated Livelihoods) was introduced during the final year of the program. One management challenge presented by this change, identified by two USAID informants, was that TO4 was managed jointly with the Social Services (now Education) Office. According to the informants, this left TO4 somewhat outside the structure of GEEL, and there was some stovepiping of information and decision-making. This manifested primarily as sub￾optimal coordination on data collection, information sharing, and the learning agenda. The lesson to be learned from this is that in addition to the drafting of an updated AMELP and TOC, clear roles and responsibilities under the new structure and the willingness to review and adapt that structure as needed would have contributed to a more unified approach in the final year of GEEL activities. This highlights the importance of providing clear, written management structures at all levels of an activity for both USAID and the implementer. These structures should be amended to reflect changing roles and responsibilities and the implications of these changes for program management and implementation. The success of the technical activities, as well as verification of that success and the collating of lessons learned, is built on robust management structures, processes, and systems. It is also worth considering the management challenges created by the addition of a whole new component, such as TO4 with only one year remaining in the activity. Although GEEL staff and USAID overall did an excellent job of implementing this change, and it has provided lessons learned for future programming (see EQ 4), it did create an additional management burden (because it required additional planning, outreach, and activities), which may have affected GEEL staff’s ability to focus on handover and closeout during the final years, according to USAID and GEEL informants. One government official commented that GEEL would have benefited from closer coordination with the government during closeout, to ensure sustainability. It is difficult to know how that would have changed outcomes for GEEL, particularly because COVID-19 response and restrictions hampered coordination and communication. Consider the impact of design on the ability to reach target populations: Based on feedback from GEEL staff and participants, in Year 2, changes were made to the grantmaking process, including an 19 increase in in-kind grants and a focus on technical assistance. This was a response to the activity’s difficulty in engaging youth and women-owned businesses under its original design. From a process perspective, this shows the importance of considering both the costs and benefits of streamlining design and staff. Although it is true that the initial deal-based structure with performance incentives reduced overhead and management costs of the activity, it was not possible to engage the right populations in the activity, because women- and youth-owned businesses were largely ineligible or unable to apply for grants under the original design. Table 4 shows the most important changes, as determined by document review and interviews, along with the program impacts of those changes and any outcomes and lessons that have been identified. TABLE 4: CHANGE MANAGEMENT FACTORS CHANGE PROGRAM RESPONSE TO CHANGE OUTCOMES AND LESSONS Internal MANAGEMENT Gender Strategy Implementation Gender and Youth Advisor (planned for Yr 2; not filled and brought temp consultant in Yr 3); staff training on gender; reviewed activities to increase engagement of women and youth. Gender strategy and its benefits were delayed. Lesson is that it is important to allow sufficient time to recruit specialists critical to implementing program objectives Staffing Staffed up in Year 2, including MEL; Local national COP Local knowledge and context-specific expertise is critical Approved AMELP and TOC Improved data collection and reporting Need TOC and AMELP in place for planning work and for proper adaptive management IMPLEMENTATION Pivot from deal￾making GUC tool launched Yr 2; GUC manual; tools provided to grantees; focus on target populations From a process perspective, this shows the importance of considering both the costs and benefits of streamlining design and staff; although it is true that the initial deal-based structure with performance incentives reduced overhead and management costs of the activity, it was not possible to engage the right populations in the activity, because women- and youth-owned businesses were largely ineligible or unable to apply for grants under the original design Pivot to resilience and addition of TO4 Theory of Change and AMELP adapted; worked with resilience advisor to integrate resilience goals in Shared management responsibilities with Education Office for TO4; needed better coordination on information sharing. 20 programming and performance measures. 3.1.2 WHAT LESSONS CAN BE DRAWN FOR USAID IN MANAGING COMPLEXITY AWARE PROGRAMMING AND ALLOWING FOR ADAPTIVE MANAGEMENT? According to USAID’s 2021 Discussion Note on Complexity-Aware Monitoring, the following are characteristics of a complex environment: • Cause-and-effect relationships are uncertain; • Stakeholders bring diverse perspectives to the situation, making consensus impractical; • Contextual factors are likely to influence programming; • New opportunities or new needs continue to arise; and • The pace of change is unpredictable. The Somali operating environment is highly complex in terms of several factors: security, political economy, governance, clan dynamics, and elite capture. According to the above-referenced Discussion Note, Somalia meets the definition of a complex environment, in that there is low agreement about what needs to be done and low certainty about cause and effect when it comes to interventions and external shocks. The approach recommended under these conditions is probe-sense-respond, which is consistent with the approach taken by GEEL. GEEL staff continually collected feedback during implementation (probe and sense) which was then interpreted in the form of recommendations for USAID in adaptive management (respond.) Probably the most oft-cited lesson learned was that the key to effective adaptive management under GEEL was the flexibility that USAID provided to GEEL staff to explore different ideas in the face of challenges, including making suggestions based on GEEL activities’ ongoing learning efforts. This flexibility derived primarily from the relationship between the COR and project staff, but it would be possible to encode a certain amount of flexibility into a contract for future work. Several GEEL staff interviewed noted that they had strong relationships with the COR and were able to discuss ideas that arose through ongoing efforts at learning under the program (GEEL CLA Strategy). They believed that these ideas received consideration and they often got permission to implement them, although there were sometimes delays, particularly in the early phases of the activity. Several GEEL and USAID respondents cited delays early in the program in fixing management and implementation challenges, which were relieved through more regular communication and a stronger relationship with the COR. Through this relationship, GEEL staff were able to receive rapid approval to make changes to activities to adapt to shocks or to capitalize on opportunities identified during implementation. Several examples of this are provided in section 3.1.1 below. The COVID-19 pandemic has served in many ways as a reminder of the importance of prioritizing local technical expertise during recruiting. In complex environments, local expertise is critical to the effective design, rollout, and implementation of development programs. Without this knowledge, it is impossible to account for context-specific factors that can significantly affect a program’s success. In addition to context-specific technical expertise, in many insecure environments, local staff have the ability to move more freely and engage directly with beneficiaries. In insecure environments, USAID and some project 21 staff may not have the ability to move around as freely as local staff, which creates management challenges, particularly regarding monitoring. Third-party monitoring (TPM) is frequently used to address this and other challenges in complex environments, and whereas it can mitigate these concerns to some extent, effective use of third-party monitoring does create an additional management burden. In the GEEL example, third- party monitoring was used to verify reported data rather than to replace direct project monitoring; nonetheless, a generalizable lesson derived from this example is that optimal use of TPM data requires coordination between the implementers of and activity and the TPM provider to ensure that monitoring visits are focused on learning priorities and to articulate lessons learned. 3.1.3 HOW DID INTERNAL (MANAGEMENT) FACTORS (E.G., MANAGEMENT CAPACITY, RESOURCES, MEL SYSTEMS, LOCAL STAFFING) AND EXTERNAL (IMPLEMENTATION) FACTORS (E.G., LOCAL PARTNERSHIPS, HOST GOVERNMENT, REGULATORY ENVIRONMENT) AFFECT GEEL’S ABILITY TO ADAPT TO CHANGING POLITICAL AND ECONOMIC CONDITIONS? Section 3.1.1 discusses internal management and implementation factors, so this section therefore focuses on how management and implementation factors affected GEEL’s ability to respond to the external shocks of drought, security fluctuations, and the emergence of COVID-19. Management To preserve gains made by the program and, more important, productive assets and access to critical resources for beneficiaries, GEEL had to respond quickly to the drought in 2017 and COVID-19 in 2020. Due to the flexibility provided by USAID and the strength of the project’s relationships with financial institutions (FIs) and the private sector, GEEL activities were able to make some quick adjustments (see Implementation section below for additional detail.) (Sources: Annual Report Year 3; TO4 Final Report; KIIs with USAID and GEEL staff, local government officials in Mogadishu and Baidoa). Despite ongoing serious and fluctuating security threats, the GEEL staff began to make progress toward goals in Year 2, continuing to build relationships with the private sector, youth, women, the diaspora, and other stakeholders and increasing reliance on local staff during recruitment challenges. One change that helped spur progress was the acquisition of a more secure location for project offices, which mitigated staff turnover due to security threats. Further, understanding of the security situation was facilitated by relationships with the host government and with local private sector partners and other organizations. Additional staff on the ground following the opening of a GEEL office in Hargeisa in Year 2 also allowed for more eyes and ears in the field and more ongoing contact with communities and beneficiaries. (Annual Report Year 3). One concern raised with the ET was the project’s MEL capacity, particularly during the early years of the activity. Reporting was not always clear in its presentation of changes over the course of the project or comparisons to targets. Outcome Surveys did not always explain sampling techniques or other methodological details or discuss changes in outcomes and their implications. USAID could also have provided more input to shape reporting, such as a reporting template. According to interviews with several USAID staffers, the lack of clarity and reporting may have impacted the tracking of adaptations in the first few years of the program and during drought response. Implementation Relationships with partners were key to GEEL activities being able to make quick adjustments in the face of drought in 2017 and COVID-19 in 2020. These relationships allowed GEEL staff to quickly identify 22 needs and opportunities in the face of shocks and to coordinate with other actors, such as FIs, that held GEEL beneficiary loans. In the face of both shocks, GEEL supported beneficiaries in seeking longer loan repayment terms and in increasing productive capacity and market access to support incomes and keep markets stocked with food commodities (See EQs 2 and 3 for more discussion of the technical aspects of these efforts). GEEL built relationships with private sector partners, FIs, government officials at all levels, and other stakeholders that often continued for the duration of the program, which was key to maintaining continuity and success of activities in the face of shocks. One note of caution is that although rapid responses are important in responding to shocks or other rapid changes, it is important to follow proper channels to ensure compliance with policy and other directives. An example provided to the evaluation team was that during the recent locust invasion, inputs such as pesticides provided to farmers were not fully vetted using the list of USG-approved pesticides. There is no evidence that any unapproved substances were distributed, but such an oversight could affect anything from an environmental assessment to human or animal health. One clear lesson is that relationships were key to navigating a complex environment and managing internally and externally driven changes. From the outset, GEEL was designed to leverage partnerships and to build on learning and collaboration with other stakeholders. (The response to EQ 2 below contains a more in-depth discussion of the importance of private-sector partnerships under GEEL.) According to project documentation and several USAID and GEEL informants, these relationships were a key factor in GEEL’s ability to respond quickly and effectively to change. During drought and COVID-19 restrictions, GEEL drew on its network of relationships and knowledge of markets provided through VC studies and implementation experience for guidance, including on how to adapt strategic alliances to most help the displaced, vulnerable, and marginalized. For example, during the drought that severely affected the livestock and dairy sectors, GEEL worked with partners within the sector to rapidly reprogram activities, pivoting to fodder production/commercial off-takers. When COVID-19 led to sudden and unpredictable restrictions on movement and economic activity, GEEL worked with partners and other stakeholders to quickly devise a response strategy, at the direction of USAID. This led to a multi-pronged approach that capitalized on GEEL’s existing knowledge and activities. For example, when the Hajj (a significant driver of demand for live animal exports from Somalia) was cancelled, live animal exports were affected, but GEEL was able to work with local meat markets to increase activity. This supported the income of exporters, while ensuring that local populations continued to have access to food. More generally, the approach included improvements to market access and facilities as a means of maintaining supplies of affordable foods in the face of increased input prices. As with the drought, GEEL also worked with financial sector partners and beneficiaries to adjust terms of repayment to protect small business assets. 23 TABLE 5: MANAGEMENT AND IMPLEMENTATION FACTORS IN RESPONSE TO SHOCKS FACTOR DROUGHT SECURITY COVID-19 Positive Negative Positive Negative Positive Negative MANAGEMENT Management Capacity and Structure Was able to adapt ongoing activities to preserve assets and critical resources for beneficiaries; Strong relationship with USAID allowed for flexibility in response. Leadership under local COP means a better understanding of the risks of field visits Was able to adapt ongoing activities to preserve assets and critical resources for beneficiaries; Strong relationship with USAID allowed for flexibility in response. MEL Systems Lack of clear reporting and monitoring capacity made it more challenging to track the outcomes of adaptations Switch to remote monitoring; clear understanding of theoretical linkages between interventions and adaptations and the goals and populations they are meant to serve According to GEEL staff, up to 70 percent of data was self￾reported in this period Staffing Additional staff based in the field were eyes and ears for opportunities to adapt the activities to respond to the drought Lack of Gender and Youth Advisor meant that efforts were not as tailored to these populations as they could have been Increase in local staff including COP in order to allow movement in more difficult to access areas 24 IMPLEMENTATION Local Relationships Worked with partners to quickly identify needs and strategize response. Cooperation with local organizations and communities can help ensure safety of staff through a better and more timely awareness of threats Worked with partners to quickly identify needs and strategize response. Relationships with Host Government Worked with government to ensure continuity of activities in affected areas Cooperation with local officials can help ensure safety of staff through a better and more timely awareness of threats Worked with government to ensure continuity of activities Engagement Processes Lack of Gender and Youth Advisor meant that efforts were not as tailored to these populations as they could have been Engaging with youth who lack economic opportunity may reduce risk of some forms of extremist violence Existing networks built through engaging women and youth, especially under TO4, allowed for a more tailored response that accounted for the needs of those groups. Source: Authors 1 3.2 EQ 2: WHAT LESSONS CAN BE DRAWN FROM GEEL’S EXPERIENCE IMPLEMENTING TO2? 3.2.1 DID GEEL ACHIEVE ITS PRIVATE SECTOR DEVELOPMENT OBJECTIVE? TO2 is focused on implementing systemic and sector-specific activities to promote the development of the following high-potential value chains (VCs): dairy and livestock, sesame, fisheries, banana, and fruits and vegetables. GEEL supported investment in these sectors through interventions with singular 25 beneficiaries as well as sector-wide interventions with aims to improve access to finance, promote best practices and business development advice). Investments took the form of grants, small-scale infrastructure (SSI), technical assistance (TA), and Investment facilitation. GEEL achieved its objective under TO2. KIIs and the quarterly and yearly reports are filled with testimonials and evidence to support this finding. Dairy and Livestock For example, in dairy and livestock, quality feed is crucial to increase weight before slaughter and increase routine production of milk. Primarily nomadic, Somali livestock herders historically relied on grazing their camels, sheep, and goats, though as droughts persist, access to public grazing land shrinks, and urbanization increases, less reliable nomadic practices have given way to more certain sources of support for their herds. The issue of poor access to year-round or well-balanced animal feed has been a challenge for Somali livestock and dairy farmers for a long time. GEEL responded to the market obstacle of inconsistent access to quality feed through widespread training of dairy and livestock producers to engage in diverse fodder production for their own herds to consume and for commercial sales. The KII with one camel dairy farmer revealed that his particular operation increased their production of fodder greater than fivefold. In addition, he was able to build a storage structure and expand his land to grow more fodder for his own dairy and supply area dairies or livestock traders with fodder in the off season. Where past fodder growing practices involved farms growing local grasses, the GEEL 2021 Annual report on outcomes in the VCs says that 50 percent of all livestock respondents grew fodder on their farms and used approximately 18 hectares (ha) of land for fodder production (which was an increase from 33 percent in the 2019 survey). Of these respondents, 67 percent had introduced new fodder varieties on their farms, including alfalfa, lucerne grass, Sudan grass, and Napier grass. The report further explains how the volume of each of the varieties grown by the individual farms had been shifted by the farmer year over year, as real-time learning confirmed which varieties were best suited to their particular soil type, livestock preferences, and were most resilient. As found in the 2018 Annual Progress Report, to further transform the dairy and livestock sector, matching grants and TA to upgrade irrigation technologies and introduce improved varieties of grasses for field grown fodder. SSI consisted of building storage for fodder harvests and TA to raise awareness of hygienic and sanitary practices in milk handling. GEEL engaged beneficiaries across the livestock VC, including milk producers, milk retailers/vendors, fodder producers, butchery operators, commercial dairy farmers and livestock traders, though individual business owners often engaged in more than one of these categories in their individual ventures. Survey findings the 2021 annual report further indicate that in livestock and dairy, 54 percent improved milk-handling practices, 46 percent improved milk processing, 33 percent engaged in an outgrower scheme, 33 percent accessed milk-cooling facilities, 29 percent had access to artificial insemination, and 25 percent improved their business management practices. In terms of equipment, 21 percent moved to use of improved milk cans, and 17 percent indicated training on animal disease management. Fisheries In terms of the fisheries sector, GEEL engaged firms of all sizes ranging from Micro (1-10), to Small (11- 50), and Medium (51-100). Firms were located in the three regions of Somaliland, Puntland, and Banadir. Broad evidence of increases in the VC includes recorded seafood sales volumes improving significantly over the term when GEEL was involved. For example, the 2021 GEEL Annual Report indicates measures 26 from the 2018 - 2020 season showed a marked improvement. In 2018, the combined areas of Puntland, Somaliland, and Banadir sold a total volume of seafood to both export and local markets of 469,830 kg. The same combined areas in 2020 sold a much improved 817,940 kg, which is a 74.1 percent increase of 348,110 kg. The survey respondents indicated that increases were attributed to greater availability of new and used fishing boats and the use of improved fishing technologies (45 percent), and investment in cooling facilities (70 percent). Other factors were reductions in fish losses (15 percent), engagements with new suppliers/buyers (15 percent), access to new markets with better prices (5 percent), the overall fishing conditions/climate were favorable (10 percent), and overall increased market demand (20 percent). To correlate these outcomes, the 2019 GEEL Annual Report indicates that investment in the prior two years (2017 - 2019) included a total of 195 fishing boats at a value of $1.5 million; 232 cooling boxes were at a value of $146,085.00. An additional 38 fish vendor shops were invested in, at the cost of $10,255.00. A total of 11,373 units of fishing gear were purchased, at a cost of $298,975.00. Finally, eight solar-powered systems were purchased at a cost of $2,150.00. The aforementioned investments are a combination of firms investing with their own capital and matching grants for equipment from GEEL. Where a firm did not have the working capital on hand, GEEL also assisted facilitation of loan applications to local financial institutions. When asked about support to fisheries, and who had provided commendable support that led to these increases, overall, 67 percent of all respondents credited USAID-GEEL. Regionally, this breakdown includes 78 percent Puntland, 38 percent Somaliland, and 69 percent Banadir. By comparison, the FAO was credited with 3 percent, the Government 3 percent, and other fishing/export companies with 5 percent. GEEL facilitated international exposure to markets and best practices for the fishing companies, something that opened the eyes of the operators to the expectations of export markets, and just how much Somali fisheries practices needed to improve. According to the 2021 final Annual Report, since 2018, 21 percent of the fish business owners had attended training on fish catch and handling practices, 5 percent had gone to the Dubai Sea Foods Fair, and another 3 percent had attended a competitiveness event. Other trade fairs that the respondents attended were in Kenya, Ethiopia, Turkey, and Somalia (Garowe, Mogadishu, and Baidoa). Most of those who attended training indicated that they were trained on fish hygiene, handling and marketing, quality controls and management systems, packaging, and new market linkages. Of the above-referenced events and training, USAID-GEEL facilitated the majority, at 82 percent. Achievements include that 18 percent of respondents improved business management practices, 8 percent met new buyers, 8 percent had good exposure to end-market requirements, and 5 percent engaged in networking. Respondents adopted several new management practices: 44 percent improved quality management, 13 percent improved operations management, 8 percent adopted new sustainable fishing practices and improved fish-handling techniques, and 5 percent enhanced procurement management practices. For fishing companies that attended the Dubai Sea Foods Fair, interactions with buyers and other fish purveyors showed that to engage export, a company had to certify its products and employ best practices. Through such events, GEEL was successful in raising awareness among the fisheries beneficiaries on how critical quality standards were to their success. This led to many of the fishing companies adopting better processes and establishing Hazard Analysis and Critical Control Points (HACCP) plans, and pursuing 27 standards benchmarked at international levels such as the International Organization for Standardization (ISO), and even connecting with independent bodies such Société Générale de Surveillance (SGS) to assist in achieving certification. Although it remains critical for the Government to likewise develop institutions that administer such critical standards and certifications, GEEL was at the forefront of this development, and both benefited businesses and government actors with examples of compliance. 3.2.2 WHAT WERE THE MAIN SUPPORTING AND HINDERING FACTORS FOR ACHIEVING THIS OBJECTIVE? SUPPORTING FACTORS Leadership, local experience, and consistency At the beginning of the GEEL program, the principal contract deliverable was to achieve a series of large investments through engaging investors of all types such as diaspora, impact, domestic, and foreign. As noted in an interview with the COP, the target was to achieve at least 20 matching grants to businesses in the amount of $500,000 or more, and the initial approach was to forge ahead alone as GEEL without developing local partnerships. Promotional events brought awareness of GEEL objectives to businesses who responded positively to the premise of matching grants to investment. It is important to note that there were initial delays engaging interested investors. These were largely caused by unclear requirements for participation, and complicated technical processes stalled enthusiasm. On top of the implementation challenges, project staff had not established relationships with the various VC lead-firm beneficiaries, so the feedback loop to make critical changes was not established early on in the project. The early slow start to the program was overcome, once the project was staffed with a local COP and a diverse team of local and international staff with relevant experience. Understanding the marginalization from clan dynamics and the local political economy allowed GEEL to adjust when needed and respond to shocks. The understanding of these local dynamics led to the realization that they had to adjust to smaller loans and other means, such as in-kind grants, to reach vulnerable populations. In terms of what allowed GEEL to develop some consistency, the GEEL design was initially a five-year program and extended two more years for a total of seven, as opposed to other project implementation terms such as PIMS and SBCF, which were both four-year programs. When adopting a market systems approach, there can never be a ‘one-size fits all’ solution; instead, each is shaped and refined through the careful analysis and understanding of specific value chains. Market systems development does not necessarily happen quickly, and often takes time and consistent observation and data collection. PIMS in particular, by employing market systems approaches, was able to develop great familiarity with their focus VCs in the four years of the program. With the advantage of an additional three years like GEEL had, the program would likely have had even greater success with market systems. The period of performance for GEEL was long enough to ensure that staff had the opportunity to familiarize themselves with sector actors, engage with the market changes, and build important relationships. After the initial change in COP, there were no further changes at the COP level other than an added one-year extension for the addition of the resilience component. The consistent leadership preserved institutional memory and gave a reliable and responsive contact for stakeholder engagement. 28 Relationships were key to navigating a complex environment and managing internally and externally driven changes. As reported through KIIs with business beneficiaries as well as implementing partner staff, partnership with local businesses requires familiarity and trust, which take time and effort. When GEEL refined its focus to specific sectors, efforts to establish networks and relationships within these sectors became a priority. Focused on “lead firms,” GEEL established relationships by learning about their operations, knowing their constraints, and working with the firms to address capital equipment needs, improve processes, as well as institute and learn compliance to international standards. The investment of time and interest with the lead firms brought familiarity and trust, which extended beyond just the primary firm. For example, KIIs with implementing partners agricultural experts informed that for producers and smallholders, who were accustomed to conducting business using traditional methods that were less effective, once GEEL developed relationships between technical experts and lead firms closely aligned with smallholders as processors or buyers, the extended relationships fostered trust and interest in attending demonstration farm exhibitions, applying improved farming techniques, and consequently, greater yields. As was confirmed through KII with a lead firm, the lead-firm relationships enabled stronger collaboration and quicker uptake of improved techniques for producers who had linkages to the lead firms. In the same way, when the focus shifted to resilience in the face of drought and pestilence, GEEL drew on its network of relationships, knowledge of market actors for guidance, including on how to adapt strategic alliances to most help the displaced, vulnerable and marginalized. For example, by the middle of Year 2, drought had severely impacted livestock and dairy sectors, but relationships within the sector allowed rapid “reprogramming” to pivot to fodder production/commercial off-takers, improve fodder growing practices, and fodder processing (pellets and dry storage). In addition, in areas that had been badly affected by drought and COVID-19, GEEL recognized the need to address food security. The focus was to ensure increased farming yields, and this was brought about by establishing partnerships with firms known to GEEL. These firms offered quality inputs, such as certified seeds, quality fertilizers, and access to advisory services and demonstration of better farming techniques. Not only did the smallholders and vulnerable producers immediately benefit from the inputs, these providers of quality inputs also established footholds in the market, where they are more likely to stay and serve the known producer community. Flexibility and responsiveness from the GEEL and USAID teams were critical in adaptive management and resultant success of all components. The initial approach of GEEL was to match investment with large grants ($500,000) issued to medium to large firms. It became clear early on that absorptive capacity among existing firms was fairly low, with only a few who could absorb such an amount easily, given the requirement to match at least that amount. USAID’s flexibility in reviewing and amending grant requirements to be less restrictive was critical in allowing micro-, small and medium-size businesses to access support. Key to this was the ability for there to be smaller matches to grants received. These adjustments also helped reach a critical segment of the more marginalized businesses, those owned or operated by women, as the average ticket size of a “loan” or need for capital infusion is much lower for these female-owned and operated businesses. These were relaxed terms allowed GEEL to reach a larger pool of businesses that qualified to be grantees, and, those that were female-owned, unlike those captured through the original approach. In terms of how the GEEL grants had an impact on access to finance, following prolonged drought, pestilence, and with the emergence of COVID 19, the orientation of the “matching grant” approach gave 29 way to a re-oriented investment strategy that called for partnerships with local banks and funds to offer a form of “blended” finance. GEEL was able to offer an incentive grant to the banks that could be used within the bank to support expanded operational ability serving medium and small borrowers, and in return, the bank was to extend credit at five times the grant amount, thereby leveraging the grant funds to reach many more borrowers. This proved critical, because when shocks were experienced, borrowers initially found fewer options from conventional lenders, because they had pulled back in response to the shocks. The GEEL leverage allowed the banks to essentially realize a “risk sharing” aspect of the scheme, having already been granted one-fifth of the amount they were to extend in credit, institutionally, the banks could then be more confident in lending at a time of greater risk. GEEL technical team members who worked with commercial banks were able to assist applicants struggling to meet the terms of their loans. Reasons for this were largely attributed to impacts from drought and COVID, inability to renegotiate terms of repayment, or in other cases, challenges accessing lines of credit. When banks were pulling back due to COVID impact on businesses performances, this evaluation found that the risk-sharing mechanism was able to mitigate against the aforementioned challenges, giving commercial banks the ability to be more flexible. In an interview with a financial institution representative, one example given was that Agribank, an agriculture lending-focused bank, became a key partner in particular for producers, as their strategic focus was already many of the target beneficiaries of GEEL, and the risk-sharing scheme between GEEL and Agribank brought capital to key agricultural businesses. When GEEL engaged efforts to address resilience in the last two years of the program, one activity specifically linked to access to finance. Many of the beneficiaries of the resilience work are informal businesses with few options of formal finance. These smallholders, small producer cooperatives, and cash￾crop farmers became extremely vulnerable due to repeat shocks, including the drought (late 2017/2018), then pestilence, then COVID. The risk-sharing schemes described above marked a strategic shift to an innovative approach to ensure access to finance, which, in turn, bolstered resilience programming. Because of the risk-share products, these new forms of finance are extended to the savings groups, savings and credit cooperative organizations (SACCOs), and village savings and loan associations (VSLAs). By reaching these groups, because the less literate and most informal are included in such groups, the effect was to span the gap between informal non-banked individuals and semi-formal and financial institution linked groups, then link to banks, and agri-input providers with credit. The access to finance activity which spurred increased lending to groups also inspired the development of basic fintech that benefited the most vulnerable borrowers found in these groups. In one case, with Dahabshiil, the bank developed a phone application for the groups to monitor and manage their credit and accounts, thereby offering better access for the producer groups and reach for the bank. (KII) Learning from earlier programs. Immediately preceding and/or parallel to the implementation of GEEL, the DFID-funded project Promoting Inclusive Markets in Somalia (PIMS) and the World Bank-sponsored project Somali Business Catalytic Fund (SBCF) also engaged private sector/market systems and investment programs through matching grant and technical assistance modalities. As reported through KIIs with implementing partner staff, initially, GEEL had determined it would establish its own unique relationships. However, the slow start led to an adjustment in approach to focusing more on GEEL’s complementarity with the achievements of other projects, such as PIMS and SBCF. GEEL, in collaboration with PIMS, was able to accelerate its knowledge 30 of key sectors and expand its networks based on sector focus and partnerships with PIMS (dairy/livestock, fisheries, sesame). More specifically, GEEL formed a cooperative grant agreement with PIMS and the beneficiary to address larger capital equipment grants for key beneficiaries, for example in sesame. By sharing the grant with PIMS, they were able to meet a bigger demand, and beneficiaries were able to scale production and meet higher standards of processing safety and efficiency. GEEL also participated early on in key convenings such as the Somali Renewable Energy Forum, developed and implemented by NGO Shuraako. As a feature of the forum, Shuraako conducted a labor force needs assessment for energy sector technicians, which pointed to serious deficits of skilled technicians. By attending, GEEL was immediately introduced to key energy industry players, who were keen to establish partnerships around technical training programs for energy workers. This was important to these actors as results of the needs assessment, which highlighted capacity gaps in their respective companies. In response to the call for training, GEEL partnered with BECO in Mogadishu, training technicians who were able to move into gainful employment, which, before the partnership, would not have been offered. The initial partnership with BECO also gave an introduction to a water utility/bottling provider, who made use of grant funding to collaborate with GEEL and offer technical training to build a team for the utility. The training resulted in a reported 85 percent employment rate, as the trainees were promised that if they were successful through graduation, they would be retained after for an internship, with approximately 30 percent of employment opportunities for women, as the utility targeted women for key roles in the company. Provision of targeted technical assistance (vocational training, capacity building, advisory services), in addition to capital yielded positive results. As GEEL solicited the participation of beneficiaries, increasingly more requests were made for targeted advisory services or technical assistance that was specialized, business and/or sector-specific. KIIs from this process evaluation indicate that the types of support services being sought were not readily available in Somalia. For example, there were specialized services needed to conduct banana cultures for propagation that could not be found locally. GEEL therefore was tasked to identify and mobilize providers that had specific knowledge to meet the needs of the businesses. For the businesses in question, the lack of access to technical assistance and business obstacles severely limited growth. One example of this dilemma was in the fisheries sector. GEEL quickly saw that hygienic handling and preservation of catch, as well as proper processing and packaging to any sort of standard, largely did not exist in the market. In an effort to link fisheries beneficiaries to export markets, GEEL identified prominent food fairs, including the Dubai Sea Food Fair, then mobilized participation for fishing firms to attend trade and food fairs. Several companies quickly realized that they did not have the certifications on their products, and most were unable to implement the right processes to meet quality standards such as ISO. Nor could they perform the necessary controls such as HACCP, which resulted in their product being inferior to other regional fishing companies. GEEL was able to mobilize critical study tours and bring in consulting firms (SGS) that could address standards and processes, and over time, were able to assist many of their fisheries beneficiaries to meet quality standards and processes necessary to certify and export their product. This highlights a critical gap in services from the state, in that they are not equipped to provide extension services, guidance, or enforcement of standards and certifications, making this an insurmountable barrier for agricultural export absent the critical technical support of a project such as GEEL. 31 In the case of their resilience work, where it required addressing yields of crops to mitigate food insecurity, the use of certified seeds and quality inputs was critical. GEEL was able to support companies who developed certified seeds and, in the process, worked with the policymakers on specific delivery of key services to support national frameworks that could provide oversight, like Somali Agricultural Regulatory Inspection Services (SARIS), among others. The evaluation was able to note that the assistance provided was very specific, due to the need for SARIS to learn how to manage seed certification processes. In the Somali context, where institutions are weak or nonexistent and growth or reform is slow, any changes to policy is protracted. In the case of food safety and certifications, private actors must look to improve quality and standards of product which is time-sensitive. GEEL recognized these needs and sought to use some businesses as cases to justify a clear need for institutional reforms. Hindering factors. Somalia is a highly complex operating environment, vulnerable to shocks in a variety of forms (e.g., violent extremism, inter-clan or other tribal line conflicts, climate/draught, political volatility, economic uncertainty, public health/pandemic outbreaks, inadequate education, and skills training). Somali business owners are largely unaware and unprepared to comply with international standards necessary to sell to export markets, and Somali institutions are weak in terms of providing guidance, compliance, and structures to certify production and support the private sector in its export activities. Bringing private businesses up to standard through improved processes and equipment is time-consuming, costly, and requires consistency. Finally, humanitarian organizations may inadvertently undermine certain development and/or market systems objectives, which tend to reinforce an overdependence on aid and further tax donor commitments. For this reason, it is critical that donors in the economics growth and resilience space coordinate closely with those in the humanitarian space. 3.3 EQ3: WHAT LESSONS CAN BE DRAWN FROM GEEL’S EXPERIENCE IMPLEMENTING TO4? Task Order 4, Somali Youth Integrated Livelihoods (see textbox) was a component of GEEL implemented in the fifth year. (One extension year was subsequently added to the activity.) The objectives of TO 4 were: “Previously in Mogadishu, there was no job creation, even the literate youth use to immigrate illegally to other countries due to lack of job opportunities In Mogadishu, while they are having with the perception that they are getting better living standard, well-paid job.”— Representative of Somalia 1. Strengthen the ability of Somali youth to successfully engage in economic growth opportunities, specifically in the agriculture, fisheries, livestock, and energy sectors. 2. Leverage the private sector to increase the engagement of youth as entrepreneurs, employers, and employees in the targeted sectors. 3. Increase public- and private-sector engagement in youth workforce development and workforce readiness activities. This TO was jointly funded and managed with the USAID Social Services (now Education) office. According to GEEL’s Youth Assessment, carried out in 2020, almost 70 percent of the population of Somalia is considered youth (defined in USAID’s 2012 Youth in Development Policy as ages 10-29.) This 32 activity targeted people aged 15-24 and this same study indicated that approximately two-thirds of that group were unemployed. The study and other learning efforts undertaken by GEEL before the launch of TO4 identified the key drivers of low participation in GEEL activities by youth as lack of information or access to programs, cultural barriers (e.g., employment through social networks rather than merit-based), and the scale of technical advisory services and other assistance, like financial products not being appropriate for youth-owned businesses, which tend to be smaller. 3.3.1 DID GEEL ACHIEVE ITS YOUTH LIVELIHOOD OBJECTIVE? Despite the challenges presented by COVID-19’s emergence shortly afterthe launch of TO4, the activities’ three objectives above were largely achieved. Data from the soon-to-be-completed Mission Annual Performance Plan and Report shows that targets were largely met. At the time of this writing, equivalent data was not yet available for TO2 indicators. Though targets were largely met, the lack of understanding of how data was collected as well as uncertainty about data quality also means that its’ data that the Mission isn’t able to use as freely (as opposed to if GEEL was still around and we could engage in a back and forth to better understand the data). TABLE 6: TASK ORDER 4 PERFORMANCE INDICATORS Pillar Description Indicator Number Indicator Title Relevant Activity FY 2021 Target FY 2021 Actuals Workforce Development EG.6-12 Percent of individuals with new employment following participation in USG-assisted workforce development programs GEEL TO4 57% 71% Sex: Female 459 (out of 706) Sex: Male 905 (out of 1228) EG.6-12e Number of females newly employed (numerator) 300 459 EG.612f Number of females who participate (denominator) 525 706 Workforce Development EG.6-13 Percent of individuals with improved soft skills following participation in USG-assisted workforce development programs 28.50% 29.47% Sex: Female GEEL TO4 247 (out of 706) Sex: Male 323 (out of 1228) 33 EG.6-13e Number of females with improved soft skills (numerator) 150 247 EG.6-13f Number of females who participate in the activity (denominator) 525 706 Workforce Development EG.6-14 Percent of individuals who complete USG￾assisted workforce development programs GEEL TO4 90% 100% Sex: Female 706 (out of 706) Sex: Male 1228 (out of 1228) EG.6.14e Number of females who complete (numerator) 473 706 EG.6.14f Number of females who participate (denominator) 525 706 Workforce Development EG.6-15 Percent of individuals with better employment following participation in USG￾assisted workforce development programs GEEL TO4 70% 66% Sex: Female 402 (out of 706) Sex: Male 876 (out of 1228) EG.6-15c Total number of female participants reporting better employment 368 402 EG.6-15e Total number of females participating in workforce development programs 525 706 Entrepreneurship and Finance EG.4.2-1 Total number of clients benefitting from financial services provided through USG￾assisted financial intermediaries, including non-financial institutions or actors GEEL TO4 250 262 Number of males 175 176 EG.4.2-1b Number of females 75 86 34 As can be seen in Table 6, GEEL met or exceeded targets for nearly all TO4 indicators, including having at least 35 percent female participation. All participants in both male and female youth focus groups carried out in Mogadishu agreed that the programs were beneficial, and most cited investments in greenhouses as an example of a useful and relevant investment. This included in-kind distribution of greenhouses and training to increase productivity, operate greenhouses, and engage markets. It was apparent from the discussions that male and female experiences were different. Whereas male participants discussed support for scaling￾up operations, and many indicated that they were owners or operators of at least one greenhouse, women participants talked more about the general positive impact on their lives and their incomes after having received training and in-kind support (see textbox). The women’s discussions were more focused on entry and opportunity, highlighting the fact that interventions must be carefully tailored to target groups (this issue is discussed more in the recommendations under EQ 4.) ‘The GEEL project was very helpful, and everyone in the community heard either from social media or from the direct beneficiaries. To give you an example, we as youth graduates have been provided greenhouses, and each of us has a family and shared the community and villages of implementation and that is how the information was spread to the community.” – Youth Greenhouse Operator, Mogadishu Given its short duration, TO4 functioned as a sort of youth livelihoods pilot program, although the ability to draw lessons learned from the experience was somewhat hampered by the fact that no follow-up data was collected. This is still valuable information but without baselines, midterms, or longer term observations post-program, it must be viewed as a snapshot rather than an illustration of trends. One example raised by several male youth respondents was that there were sometimes technical issues with the installation of drip irrigation and since the program had ended, there was no monitoring in place to identify these problems or program resources to address them. As part of its kickoff efforts, GEEL held a Youth Investment Expo from January 27-28, 2020. This provided young entrepreneurs and would-be entrepreneurs with the opportunity to engage private sector representatives, financial institutions, and policymakers. Thirty percent of participants were women. Although this falls short of the overall goal of 35 percent female participation, it far exceeds the target of 10 percent participation at any single event. (GEEL TO4 Final Report, 2021). GEEL then proceeded to partner with 15 private sector companies to deliver demand-based training and job placement, focusing on emerging technologies. This included the use and adaptation of greenhouse technologies. COVID-19 Response Although GEEL’s overall COVID-19 response is discussed in more detail under EQ 1, there were some adaptations that targeted youth in particular. As part of its broader relief efforts, GEEL engaged financial institutions to offer youth-specific credit services, which comprised longer repayment periods with lower rates as well as business development services providing tailored technical assistance. More than 270 youth-owned businesses with innovative ideas were submitted to FIs under this effort. Youth-owned enterprises were also engaged in making PPE, including masks and gloves, which was in short supply early in the pandemic, due to limited imports. According to the TO4 Final Report, GEEL was able to engage 12 partners to make 562,000 masks and 3,000 pairs of gloves, creating 304 jobs (direct and indirect) (TO4 Final Report). The skills taught as part of this effort, including tailoring, distribution, and supply chain management, are all applicable to more general economic activities. 35 While GEEL worked with its partners and other stakeholders to reschedule or adapt training to virtual platforms, staff recognized that practical training was often best done in person, where technologies and their applications could be better demonstrated and access was not limited by access to virtual platforms. Therefore, in-person training for greenhouse technologies was continued once permitted under COVID restrictions, with careful use of PPE and social distancing in place to ensure the health and safety of participants. There was broad agreement in the two youth focus groups that this training and the use of greenhouse technologies in the GEEL approach in general were key elements of what made these interventions relevant and successful. Another example of the importance of partnerships and coordination being key to program success was the memorandum of understanding (MOU) with the Banadir Regional Administration. Although not solely focused on COVID response, the MOU was launched with the intention of ensuring continuity of activities under COVID restrictions, while at the same time, focusing on activities that would be useful to participants in the longer term, particularly those most often targeted by resilience programming. Activities focused on urban poor youth and IDPs and comprised partnerships with the private sector to provide job-skill focused training to meet the demand for certain skill sets in the agriculture, fisheries, energy, and construction sectors. 3.3.2 WHAT WERE THE MAIN SUPPORTING AND HINDERING FACTORS FOR ACHIEVING THIS OBJECTIVE? Another important activity under TO4’s COVID response was the Training of Trainers. RTI’s Youth Employment and CVE Specialist assisted in training local partners in how to train youth on soft skills. Table 7 lays out factors that supported and hindered the success of TO4. TABLE 7: SUPPORTING AND HINDERING FACTORS FOR TASK ORDER 4 SUCCESS SUPPORTING FACTORS 1. Collaboration with stakeholders in COVID-19 response 2. Market-driven training in addition to meeting the need for skills in several sectors, by connecting youth with job opportunities, these efforts helped overcome the influence of social networks in favor of more merit-based hiring 3. Better understanding of skills mismatches gained through TO4/TO2 collaboration-GEEL leveraged VC partners from TO2 including BECO, Somalia’s largest utility. 4. Connections between training, internships, employment opportunities through private sector collaboration. This included career counseling and other services for job placement, all in collaboration with TO2 partners and others. 5. Extensive partnerships with government and other stakeholders, including Agriculture Youth Partners and Banadir Regional Administration. This included capacity building of partner organizations to carry out training a. Literacy and Numeracy Assessment b. Under partnership with City University of Mogadishu, two greenhouses and a poultry farm were provided in order to meet the need for more practical agriculture training. 36 6. Engagement on platforms likely to reach youth, including more digital outreach and social media. These efforts helped increase access to information about job opportunities, making it accessible to a broader range of potential applicants. 7. Youth involved in the program were proactive in engaging others through the Somalia Agricultural Girls Association, which increased awareness and participation among young women (Final Report, TO4) 8. Building on existing skill sets and/or sectoral involvement. Although it is important to expand opportunities in new sectors, it is also important to build on existing viable foundations. With 98 percent of milk distributors being youth and/or women, GEEL focused on training in handling and business management, including the provision of iceboxes to increase shelf-life during transit (TO4 Work Plan, 2020). HINDERING FACTORS 1. Lack of opportunity, even for the educated. This was identified as a challenge by GEEL studies, several youth focus group participants (male and female), and by respondents from both GEEL and USAID in KIIs. 2. COVID-19 restrictions on movement and other impacts such as export restrictions, and the like, that impacted job opportunities and overall economic activity, with young people who are new to their positions or to business most likely to feel the immediate effects of shocks. 3. Short duration of activity and lack of focused effort to capture lessons learned through follow-up data collection. One of the overarching challenges for this TO was difficulties attracting youth to invest their time and efforts with no clear evidence to demonstrate how their participation would have tangible, positive results. Access to information about employment opportunities, assistance with job placement, and other efforts to increase merit-based hiring all implicitly increase the value of investing in training by participants. Market-driven training and youth-appropriate finance options were also noted to increase the number of jobs available to trainees and their ability to enter the economy as entrepreneurs. Both the design and management of GEEL, as outlined in the table above, contributed to this end. Of the three main hindering factors identified in this evaluation, only one was under the manageable interest of the activity, which is the short duration of the TO. Such a short duration can still be useful for a pilot activity, but the ability to properly capture lessons learned and thus reap the full benefit of the investment in the form of lessons learned for future programming is inhibited by the snapshot nature of the data collected. For example, although hundreds of youth-owned businesses were connected with FIs, it is not known how many ultimately received support, nor do we know the outcomes of that support. The KIIs and FGDs conducted through this evaluation provided some insight into how participants viewed GEEL after the conclusion of the program, but they did not allow for detailed data collection on outcomes. 3.4 EQ4: HOW CAN GEEL’S IMPLEMENTATION EXPERIENCE BE LEVERAGED BY USAID IN ITS NEW STRATEGIC PIVOT UNDER ITS 2020 – 2025 COUNTRY DEVELOPMENT COOPERATION STRATEGY (CDCS)? The new CDCS contributes to the overall U.S. Government policy priority of preventing and countering violent extremism. Lessons learned from GEEL will be applied to the upcoming economic growth and 37 resilience promotion activity, which will support both Development Objectives (DOs) in the new CDCS: DO1 – Targeted local institutions govern in a more legitimate manner, diminishing influence of violent extremist organizations and DO 2 – Enable marginalized Somalis to more effectively withstand shocks and stresses. This activity, co-funded and co-created with FCDO, (Resilient and Inclusive Economic Development (now known as KAYD)) will focus on promoting resilience among populations that have been dependent on humanitarian assistance and will involve close collaboration with DFID-funded partners. The focus of this programming will pivot from being primarily a private sector growth objective to strengthening the resilience of households, communities, and systems. The Theory of Change underlying this new focus is that increasing the resilience and self-reliance through the promotion of livelihoods among the populations most vulnerable to climatic and conflict related shocks is critical to overcoming decades of these kind of shocks. When the resilience focus was added for the final year, staff responded by taking an in-depth look at the implications of these changes, including a three-day workshop to address changes to the AMELP and Theories of Change (TOCs). These were updated to reflect resilience and self-reliance priorities, aligning GEEL activities more with the Mission’s 2020-2025 CDCS. Table 8 details the recommendations for applying GEEL lessons learned to programming under the new CDCS. TABLE 8: RECOMMENDATIONS 1. Do the groundwork to determine which skill sets and commodities are the best investment of program resources. Optimal investments depend on a variety of factors, including target populations and geographic priorities. With the focus on IDPs and other vulnerable populations, donor investment in the most promising value chains from a private sector growth perspective, though an important consideration, should crowd out ones that create the most opportunity for target populations. As an example, under GEEL, fish processing and milk distribution were determined to be areas where women and youth were engaged and thus became priorities for investment. Also consider skill sets outside more traditional economic growth sectors. In the focus group discussion among young women in Mogadishu, all participants agreed that training on skills such as tailoring or beauty services would be beneficial to their longer-term economic prospects. Tailoring and beauty services are rather common and are not necessarily scarce or high growth, though beneficiaries see within each of these vocations’ potential for a marketable, potentially portable business that they can feasibly grow into a newer more profitable market or in times of displacement, reestablish elsewhere. 2. Engage host government and other stakeholders on land and other input availability in target areas. Greenhouses were a successful intervention under TO4, and respondents in both male and female youth focus groups indicated that they valued the training and equipment they received through these interventions. Greenhouses and other in-kind support to productive capacity require land, however, which is especially challenging when dealing with IDP populations. Respondents from GEEL indicated that the federal government is exploring options for providing land to IDPs in the Mogadishu area. Any follow-on activity should carefully explore the status of those efforts and any alternatives when considering how best to support those populations. This is also likely to be true for urban poor youth. 3. Timing of interventions is critical to success. 38 As a high-level government official in Baidoa emphasized in an interview, technical assistance and in-kind support that do not sync up with growing seasons can reduce the benefits to participants. With proper timing, participants can apply learning and inputs in real time, maximizing the benefits of training. In the worst cases, missed opportunities due to poor timing or delays can affect the ability of smallholders and small business owners to repay loans. 4. Maintenance and operation of capital investments should be considered during design. As noted in EQ 3, although FGD participants were generally happy with the in-kind support they received, one participant noted that there were problems with some of the drip irrigation systems GEEL installed and that the lack of follow up by the program led to delays in fixing these problems. Although they did not report similar problems with greenhouses, greenhouses also require maintenance, and any program that supplies these types of infrastructure investments must consider longer term upkeep and operation, if the program’s benefits are to be sustainable. 5. Look for lessons in other programs, including USAID's and those of other donors. As noted in EQ 2, GEEL benefited from coordination with PIMS and other existing programs and from their respective lessons learned. With an early shift in GEEL’s approach, stepping away from its initial go-it-alone approach to instead collaborate with PIMS rather than competing with each other in support of mutually identified key value chains, each was able to contribute according to their comparative program strengths, thereby achieving more impactful outcomes. Likewise, GEEL not only improved its ability and willingness to learn from other development partner programs, but also became a main source of partnering and collaboration. GEEL was known for sharing and supporting for example the SSF II efforts, BRICS and SomREP efforts. For future programming, it is important to evaluate the donor landscape to identify both potential partners and learning opportunities with existing programs to increase coordination, collaboration and partnerships. 6. Know whom you are talking to. Elite capture is always a risk with donor programs and is especially difficult to avoid in complex, dynamic environments, where donors are relatively inexperienced. It is critical to do the groundwork to make sure that engagement processes are truly reaching representative samples of target populations. Annex IV provides a preliminary clan analysis that may be useful for future programming efforts. 7. Map out a focused learning agenda based on a Theory of Change and AMELP. The learning agenda can evolve, as it did with GEEL during the pivot to resilience, but it is important to have a clear learning agenda that has been shared with both the implementer and any third-party monitoring implementers. This can help guide reporting and ensure that activities are collecting data on the outcomes that are most important to the learning agenda. As adaptations are made to the activity, the learning agenda, AMELP, and TOC should be updated as needed, ideally through discussions with relevant stakeholders. 39 8. Make technical training more accessible to urban poor youth and IDPs. This is especially challenging under COVID restrictions, as these populations are least likely to have access to virtual learning options. As was the case with greenhouse training, activities should identify which, if any, training is most critical to conduct in-person, and ensure that proper facilities and PPE are available for participants. Make use of tools, such as the literacy and numeracy assessment tool developed by GEEL, to identify the most urgent training needs of target populations. 9. Keep it simple. For resilience-focused activities, it is not necessarily innovation that counts. Focusing on existing market activities and how they can be adapted to vulnerable or marginalized populations is the best place to start. For example, we know that access to finance is a major barrier to entry for marginalized groups. Group lending activities are a tried￾and-true method for bolstering resilience and saw success under GEEL. They are especially helpful for groups that do not have access to finance through more mainstream financial institutions. 10. Fortify group lending and mutual support group effectiveness through creative risk sharing schemes with local FIs and organizational strengthening efforts for self-help groups. Beneficiaries requiring resilience support are often informal earners pursuing basic livelihoods or informal businesses with few options for formal finance. GEEL found that its work with local financial institutions during their COVID response and resilience work phase made rapid impacts and lasting success. GEEL supported lending to ‘groups’ such as self-help Ayuda/Hagbad, SACCO'S and VSLA’s and worked with local banks to provide a risk sharing grant that banks were to match X a multiple in lending capital and target those most vulnerable communities. This fortified access to operating funds for businesses of marginalized communities and reinforced the mutual support ‘ecosystem’ of group members taking care of other group members in a shock when these groups could continue to cycle critical funds in a time of most needs. When supporting groups, it also allows the less literate or less formal benefit from others in the group who are, and by virtue of group association, on a basic level begin to bridge a gap from informality into at least basic formality (linkage to a bank or lender) and thereby increasing the likelihood of formal savings hence a reserve in future shocks. Source: Authors 1 40 4. EVALUATION DELIVERABLES TIMELINE The period of performance for this assignment is September 27, 2021 – June 26, 2022. Specific tasks and deliverables are outline in the schedule below. Using Task Order Section F.5 as a guide, with completion of dissemination presentations weeks 24-27, Integra has slightly modified the timeline as presented below and as discussed with the Mission for review. Weeks S e p t 2 7 O c t 4 O c t 1 1 O c t 1 8 O c t 2 5 N o v 1 N o v 8 N o v 1 5 N o v 2 2 N o v 2 9 D e c 6 D e c 1 3 D e c 2 0 D e c 2 7 J a n 3 J a n 1 0 J a n 1 7 J a n 2 4 J a n 3 1 F e b 7 F e b 1 4 F e b 2 8 F e b 2 8 M a r 7 M a r 1 4 M a r 2 1 M a r 2 8 Award 1. Initial meeting 2. Draft Inception Report 3. Stakeholder workshop 4. Revised Inception Report Fieldwork prep / scheduling Fieldwork / KIIs 6. Analysis / Preliminary findings workshop 7. Draft Evaluation Report USAID review 8. Debrief 9. Final Evaluation Report 10/11. 508 compliance 12. Dissemination * Week 13 accounts for holidays / office closure 41 ANNEX I: DRAFT DATA COLLECTION TOOLS KEY INFORMANT AND FOCUS GROUP INTERVIEW GUIDE To be concise, this draft contains questions for each type of informant. Prior to the beginning of fieldwork, and once the questions have been finalized in consultation with USAID, the guides will be separated into different versions by informant type and possibly by geographic location, if necessary. It is not anticipated that different guidance will be needed for remote and in-person interviews. That determination will be made once all locations have been chosen. Additional guidance for FGDs will also be produced once topics are chosen at the Stakeholders Workshop. Note to interviewer: This is intended to be a discussion guide, not a formal questionnaire. Interviewers should deviate from the listed topics and questions when appropriate; for example, to further pursue an interesting point brought up by the interviewer or to skip questions that are deemed irrelevant or were addressed in some other manner during the discussion. As this is a process evaluation, while we are considering performance of the activities, the primary focus of the discussion will be on process components, examining change management, targeting of participants, and work plan adherence and adaptation, among other key process and management factors. They included identifying bottlenecks, delays, and inefficiencies and how those could have been avoided or mitigated. The discussion should include how processes can be improved moving forward in future USAID programming. Facilitator: Note taker: Location: Date: Participants: (include age, sex): Male/Female: Respondent Affiliation (USAID, implementing partner, program participant, etc.: Script for Start of the Interview Hello, __________. My name is __________and I am working with Integra to conduct an evaluation of USAID GEEL Activity. The purpose of this evaluation is to help you and USAID gain a better understanding of how the Activity has worked during implementation and what areas might be improved going forward. Consent to the Interview and Recording the Interview Your participation is voluntary. No one will know your responses to the questions. me know if you want to pause or stop the interview at any time. Would you be willing to allow the interview to be recorded? Y/N 42 You have the right to participate in the interview without being recorded, Do you have any questions? Thank you. Questions for Program Participants: 1. Which activities did you participate in under GEEL (e.g., youth entrepreneurship, value chain strengthening, etc.)? Please provide location and approximate dates of participation for each activity. (For the interviewer-responses should be tagged by TO to indicate whether the activities fell under TO2 or TO4) 2. How were you chosen as a participant? What in particular made you interested in participating in the program? For youth/women, do you feel that the efforts to engage (youth/women) were adequate? Did they create any impediments to participation? How would you improve engagement for future activities? 3. Do you feel there was sufficient outreach and engagement with the community during preparation for the activity? 4. Were sufficient efforts made to engage different clans, age groups, genders, and ethno￾linguistic groups? Do you feel that the type or level of those efforts affected participation by those groups? 5. Are you aware of others in the same area who were interested in participating but were not chosen? Do you know why they were not chosen? 6. What are your perceptions/observations about the approach taken by GEEL? Specifically, were the activities relevant to your needs? Were they accessible? Were trainings useful and provided in a timely manner? What strengths and weaknesses did you observe during your participation? For TO2 participants, do you feel the approach put Somali actors at the forefront and sufficiently empowered you and other participants to be drivers of change? For GEEL Staff 1. How were program participants chosen? Who had input into both geographic focus and project participant selection? 2. Please describe your approach to engaging potential program participants, in particular youth and women. In hindsight and based on your experience with implementation, would you make changes to that approach for future programming? Please discuss. 3. Was a gender and/or youth livelihoods assessment carried out during development or work planning? If not, what was the basis for devising a strategy for engagement of women? Youth? 4. Were adaptations made to the work plan to meet changes in context or project priorities or changes observed in potential causal pathways between the implemented activities and expected results? How was the implementation of those adaptations carried out? To what extent were the MEL systems and processes pivoted to inform iterative and adaptive programmatic decision making? 43 5. Were adaptations made to the work plan to meet changes in context or project priorities? How was the implementation of those adaptations carried out? 6. What was attendance like at activity events such as trainings or other capacity building events? What was the opinion of participants about the relevance and quality of these events? Did participants express any opinions or observations about the level of empowerment for participants, regarding GEEL’s intended approach to ensure that transformation was Somali led? For Local Authorities 1. Do you feel there was sufficient outreach and engagement with the community during preparation for the GEEL activities in your area? Did it differ by activity? Do you agree with the way engagement was done and participants were chosen? Do you feel that USAID /IRG did a good job of supporting local leadership of project activities? 2. Were sufficient efforts made to engage different clans, age groups, genders, and ethno￾linguistic groups? Do you feel that the type or level of those efforts affected participation by those groups? 3. What are your perceptions/observations about the approach taken by GEEL? Do you feel the approach put Somali actors at the forefront and sufficiently empowered participants to be drivers of change? 4. Were schedules adhered to when agreed upon? If not, were you provided with notice and information about updated timelines and the reasons for change? Do you think that changes in schedules or planned activities affected project outcomes? 5. Were adaptations made to the work plan to meet changes in context or project priorities? How was the implementation of those adaptations carried out? 6. What was attendance like at activity events such as trainings or other capacity building events? What was the opinion of participants about the relevance and quality of these events? Did participants express any opinions or observations about the level of empowerment for participants, regarding GEEL’s intended approach to ensure that transformation was Somali led? For CSOs 1. Do you feel there was sufficient outreach and engagement with the community during preparation for the activity? Do you agree with the way engagement was done and participants were chosen? 2. Were sufficient efforts made to engage different clans, age groups, genders, and ethno￾linguistic groups? Do you feel that the type or level of those efforts affected participation by those groups? 3. What are your perceptions/observations about the approach taken by GEEL? Do you feel the approach put Somali actors at the forefront and sufficiently empowered participants to be drivers of change? 44 4. Were you aware of or did you have the opportunity to attend activity events such as trainings or other capacity building events? What was your opinion about the relevance and quality of these events? In particular, did you have any perceptions or observations regarding GEEL’s intended approach to ensure that transformation was Somali led? For Private Sector Firms 1. Do you feel there was sufficient outreach and engagement with the private sector during preparation for the activity? Do you agree with the way engagement was done and participants were chosen? Do you agree with the choice of supply chains that GEEL has chosen to focus on? 2. Was sufficient effort made to engage a diversity of firms and investors? Are you aware of efforts to engage businesses owned by female, young, or entrepreneurs from marginalized clans or groups? 3. What are your perceptions/observations about the approach taken by GEEL? Was the technical assistance relevant to the challenges faced by the private sector? 4. Did you receive any grants through from Value Chain Competitiveness Funds? Whether yes or no, can you comment on the application process? Were you aware of this opportunity? If you did receive funds, what was the grant management and reporting experience like? 45 ANNEX II. ILLUSTRATIVE EVALUATION MATRIX TABLE 8: EVALUATION MATRIX RESEARCH QUESTION DATA SOURCE METHODOLOGY EVALUATION QUESTION 1 (including sub￾questions) Performance data and project reporting-in particular, documentation of changes and change management processes KIIs and FGDs- GEEL team, Policy makers, Lead Entities, select small holders or associations. Performance data will be used to examine outcomes and whether targets were met. Project documentation combined with KII and FGD qualitative data will then be used to examine processes and external factors that impacted those outcomes and identify lessons learned for future programming. EVALUATION QUESTION 2 (including sub￾questions) Performance data and documentation of approach to private sector objectives. Secondary data relevant to outcomes of private sector focused activities KIIs and FGDs with GEEL team, Desk review, Local Banks and Gargaara, Lead Entities, representative sample of small holders, specific ministries/agencies such as SOMINVEST, MOCI Performance data for private sector objectives will be used to examine outcomes and whether targets were met. Project documentation combined with KII and FGD qualitative data will then be used to examine processes and external factors that impacted those outcomes, including beneficiary targeting and engagement, internal change management of the project, and external shocks impacting the private sector. 46 QUESTION EVALUATION QUESTION 3 (including sub￾questions) Performance data and documentation of approach to youth livelihoods and engagement strategy. Secondary data relevant to outcomes related to youth activities KIIs and FGDs with GEEL team, Youth groups, beneficiaries of the youth livelihood sub activities, Ministry of Labor and Social Affairs, youth livelihood focused NGOs, youth associations. Performance data for the youth livelihood objective will be used to examine outcomes and whether targets were met. Project documentation combined with KII and FGD qualitative data will then be used to examine processes and external factors that impacted those outcomes. The analysis will focus in particular on engagement strategies and processes, how those impacted participation, how that affected achievement of the objective, and what lessons can be learned on beneficiary engagement for future activities. See draft data collection tool in Annex I for more information on specific questions. EVALUATION QUESTION 4 Project reporting, including reporting in annual and quarterly report on administrative issues and challenges and how they affected performance. Outcome surveys CDCS and RFI (RIED) Beneficiary data and related performance data, especially regarding engagement and participation of populations and geographic areas targeted by CDCS. KIIs and FGDs Lessons learned relating to performance and engagement on TO1, TO2, TO4, focusing on target populations under the new CDCS, will be translated into recommendations for both programming and activity management. This will take into account both the new CDCS and the information from the KAYD RFI which better informs the learning needs. 47 ANNEX III. REFERENCES ● Mission’s 2021-2025 CDCS and Performance Management Plan (PMP); ● Statement of work (SOW) for the GEEL activity; ● Implementing partner (IP) response to GEEL activity SOW; ● Activity work plans, semi-annual and annual reports, activity monitoring, evaluation and learning plan, management reviews developed as part of routine monitoring, relevant sections of the Project Appraisal Document and miscellaneous thematic reports from other sources; ● 2019 and 2021 GEEL Outcome Surveys; ● Monitoring and Verification reports conducted by USAID’s third-party monitoring contractor; ● Data Quality Assessments (DQAs), conducted by USAID’s third-party monitoring contractor; ● Promoting Inclusive Markets in Somalia evaluation; ● Investment Promotion or other Materials developed in partnership with SOMINVEST, MoCI, MoAI, or other agencies of FGS or Government of Somaliland; ● Trade missions and/or Investment Conference attendance, study tours or other similar mission reports; and 48 - ANNEX IV. SAMPLING APPROACH TABLE 9: KEY INFORMANTS (NON BENEFICIARY) RESPONDENT TYPES KEY INFORMANT POOL # LOCATION(S) USAID/Somalia Mission Staff Office Directors and Deputies 5 Remote Contracting Officer’s Representative (CORs) Sectoral and technical experts Project Staff Chief of Party 5 Remote Technical staff M&E team Government Representatives Ministry of Planning Investment and Economic Development (MoPIED) 5 Min. 1 per location: Mogadishu, Jowhar, Baidoa, Afgoye Ministry of Commerce and Industries (MoCI) Ministry of Agriculture and Irrigation (MoAI) Somali Bureau of Standards (SOBS) Ministry of Fisheries Civil Society Organizations Chamber of Commerce 5 Min. 1 per location: Mogadishu, Jowhar, Baidoa, Afgoye Co-operatives Business and trade organizations Investment promotion companies Private sector, non-beneficiaries Financial institutions 2 3 Mogadishu Private sector - applied not selected as beneficiaries Location(s) TBD Total Key Informants (Non-Beneficiary) 25 TABLE 10: KEY INFORMANTS (BENEFICIARIES) VALUE CHAIN LOCATION SAMPLING # Dairy and Livestock Minimum 2 locations: Mogadishu, Balcad, Afgoye, Baidoa 5 Sesame Minimum 2 locations: Jowhar, Afgoye, Baidoa 5 Fisheries Mogadishu, Kismayo 3 Banana Minimum 2 locations: Jowhar Mogadishu, Balcad, Afgoye, Baidoa 2 Fruit and Vegetable Minimum 2 locations: Jowhar Mogadishu, Balcad, Afgoye, Baidoa 5 Total Key Informants (Beneficiaries) 20 49 TABLE 11: ANALYSIS OF CLANS BY LOCATION FOR FOCUS GROUP DISCUSSION BAIDOA CLAN CLASSIFICATION GROUP RATIONALE Dominant clan(s) Rahanwayn - Mirifle - Leysan Majority clan district-wide (both urban and rural). Leysan occupy more influential positions than Hariin and are more numerous, though power is contested between Leysan and Hariin. Other main clan(s) Rahanwayn - Mirifle - Hariin and Hadamo Second major clans, whereby the Hariin are much more strongly represented in terms of numbers and positions than the Hadamo, who dominate Xudur district. However, Hadamo hold a number of important positions within the security apparatus. Marginalized clan(s) Rahanwayn - Mirifle - Elay, Reer Dhimaal and Moalin Wayne, Boqol hore The Elay are strongly represented in Buur Hakaba district, but less so in Baidoa, where their claims to indegeneity are largely ignored and the group relatively sidelined politically. Elay, Reer Dhimaal, Mo'alin Wayne, and Boqol Hore all lack political power, compared to Leysan, Hariin, and Hadamo. However, there is still considerable variation in degree of marginalization within these clans, and those that are displaced (whether from other parts of the district or elsewhere) are among the most marginalized in Baidoa. JOWHAR CLAN CLASSIFICATION GROUP RATIONALE Dominant clan(s) Hawiye - Abgaal Abgaal was the leading clan in the district, but after a long period of conflict with the Shiidle over political control, in which Abgaal often excluded the Shiidle, both clans now hold an equal share of power, in which none of the two has tangibly less influence than the other. Other main clan(s) Jareer (Bantu) - Shiidle Original founders of the district and indigenous to Jowhar, the Shiidle were long excluded from power by the Abgaal. They are now a numerous clan with an equal share in decision-making positions. 50 Marginalized clan(s) Hawiye - Gal Jecel, Hawiye - Mobileen, Xawaadle, Baadi Cad and Murusade The Mobileen often associate themselves with the Abgaal. Gal Jecel, conversely, is strongly involved in Al Shabaab in the area, but are also included in the Hirshabelle government providing the parliament's speaker, the Minister of Defense, and the District Police Commander. They are thus a crucial group of Jowhar's conflict dynamics. Baadi Cade are also involved with Al Shabaab. Xawaadle are a minority, and most numerous in Hiraan. AFGOYE CLAN CLASSIFICATION GROUP RATIONALE Dominant clan(s) Rahanweyn-Digil-Gelledi Largest clan in district; holds DA position; does not dominate local positions, but dominant in terms of population certainly Other main clan(s) Hawiye-Wacdaan Hawiye-Wacdaan is the 2nd largest clan in district; some representation in local government (general secretary of district). Marginalized clan(s) Jareer-weyn-Shiidle; Sab-Eyle; Hawiye-Abgaal; Hawiye￾Murusade Jareer-weyn-Shiidle is 3rd largest clan (including all Bantu groups), but relatively limited representation in government (one position in district administration). Sab-Eyle are an occupational minority (i.e., discriminated caste) and have no representation in local government; Hawiye-Abgaal have some representation in security services (SNA commander) but are generally viewed as marginalized relative to the Gelledi and Wacadaan; Hawiye-Murusade have some representation in the district administration (vice DA for social affairs) but generally are viewed as marginalized relative to the Gelledi and Wacdaan. DAYNIILE (E.G.) CLAN CLASSIFICATION GROUP RATIONALE Dominant clan(s) Hawiye-Murusade Largest clan in district; holds DC position, vice DC, general secretary, taxation office, a few other prominent positions in district Other main clan(s) Hawiye-Abgaal-Reer Mataan; Hawiye-Habar Gedir Second and thirst largest clans in district; represented in administration, but not widely represented Marginalized clan(s) Rahanweyn (Digil and Mirifle); Jareer-weyne; Boon Smaller group; does not hold any major local positions 51 ANNEX V. SAMPLE DATA CODING MATRIX TABLE 12: DATA CODING Demographics/Organization Characteristics Themes and Evaluation Questions (responses to specific interview questions will be coded according to the Evaluation Questions they address) Respo ndent Identi fier Age Sex Affiliation Value Chain Type of Organization Location EQ 1 EQ 2 EQ 3 EQ 4 Evaluation Objectives addressed Perform ance Indicato rs address ed 1 Beneficiary 1.1 Individual 1.2 Firm 2 Non￾Beneficiary 2.1 Individual 2.2 Firm 2.3 CSO 2.4 Local Government 2.5 National Government 2.6 USAID 2.7 GEEL 1. Dairy 2. Sesame 3. Banana 4. Fisheries 1 Private Sector 1.1 Financial Institution 1.2 Agricultural (subtypes will be coded based on KIIs) 1.3 Other (to be coded based on KIIs); 2 CSO (coded by type based on KIIs) Y/N Reco rd sub￾questi ons as Y/N Y/N Y/N No sub￾questi ons 1) Process analysis 2) Inform future programming List by indicator number 52 follo ws: 1-1; 1-2; 1-3 53 ANNEX VI. COVID PROTOCOLS FOR IN-PERSON DATA COLLECTION The safety of our team (consultants and staff) is Integra’s top priority. If at any point during your fieldwork period you feel that you are unsafe, please do not continue the work and alert your Operations Manager, Associate, and Team Lead immediately. If you are having a medical emergency or are in immediate danger, please contact your local emergency and/or medial service provider and alert your team at your next available opportunity. For the protocols below, “staff” includes any Integra-hired employee or consultant, including support staff such as drivers and logisticians. Integra will make the decision whether to conduct fieldwork and in-person interviews on a case-by-case basis. Every attempt will be made to conduct fieldwork remotely before deciding to engage in fieldwork that requires travel during the pandemic. When fieldwork is required, team members should follow the protocols and procedures outlined in this document. BEFORE FIELDWORK Prior to beginning any approved fieldwork, consultants and staff are encouraged to report if they become aware of any changes in the health or security environment in their locations of travel. Integra will continue to monitor the COVID-19 situation and follow-up on any alerts received through our information service, International SOS (ISOS). In addition, all team members should follow the specific protocols and procedures outlined below: ● Staff should make every attempt to quarantine for 10 days before and after fieldwork to ensure the safety of themselves and their team. Integra will reimburse for COVID testing with receipts as necessary to meet travel requirements and restrictions. ● Your Operations Manager or Associate will provide a brief on current medical and security risks based on our health and security service provider, ISOS. Please review and confirm receipt of that brief with your Operations Manager / Associate before commencing travel. ● If you have tested positive or knowingly have been exposed to someone that has tested positive for COVID-19 in the past 14 days, please alert your team immediately. ● If using air transport, Integra will make every attempt to reduce the number of flight connections to reduce the time spent in airports. When flying, if possible and within your booked class of travel (i.e., economy in most cases), staff should select seats that are not carrying passengers on either side in line with social distancing precautions. DURING FIELDWORK During fieldwork, Integra will continue to monitor the COVID-19 situation and receive alerts through their information service, ISOS, though consultants and staff are encouraged to report if they notice any changes in the health or security environment in their locations of travel. In addition, all team members should follow the protocols and procedures outlined below: 54 ● Integra will make every attempt to limit the use of hotels, and the number of hotels utilized in the event they are required, to reduce the level of human contact for our team. As is standard Integra practice, rooms will not be shared amongst team members. Integra will also make every attempt to identify hotels with reputations for following COVID-19 protocols but encourages staff to further sanitize their rooms as necessary. ● All staff are required to wear face masks at all times unless actively engaged in eating or drinking. This includes drivers who will be hired by Integra to reduce the level of human contact usually experienced via taxis and/or ride sharing applications (e.g., Uber). ● All staff should consistently wash their hands for at least 20 seconds, using soap and water. If hand washing is not available, please use hand sanitizer. ● Social distancing (i.e., 2 meters or 6 feet between you and another person) should be practiced whenever possible. If traveling within a car with other team / support staff, all staff should continue to wear masks while traveling. ● Avoid any forms of close contact such as shaking hands or sharing food, and avoid touching areas of your face (e.g., eyes, nose, mouth, etc.). ● When conducting interviews or focus group discussions, teams will bring additional masks (reimbursable by Integra) in the event informants do not have masks or are not following recommended precautions. ● Every attempt should be made to conduct interviews and/or focus group discussions outside or in open areas, if respondents are amenable to it. Participants should practice social distancing at all times. In the event there is no option to conduct interviews outside or social distance, you may decline to conduct the interview and attempt to reschedule. In this event, please make every attempt to find a setting that will allow you to conduct the interview. Related, your team and Logistician will make every attempt to ensure informants and associated staff are informed in advance to avoid this issue. ● If interviewers suspect an interviewee may have COVID-19, they should not conduct the interview. They should attempt to interview a different respondent or reschedule via phone or videoconference. ● After working hours, please continue to practice social distancing, wear masks, and avoid crowded places. Team meetings outdoors are encouraged whenever possible. If staying in hotels, it is encouraged that you eat outside or in your own rooms. FOLLOWING FIELDWORK ● Upon return from fieldwork, it is recommended that you quarantine for 14 days to ensure your safety. Please note that time spent quarantining before and/or after fieldwork, while not working, is NOT allowable on your timesheet but highly recommended as a precaution. ● If required to meet necessary government travel requirements, Integra will cover the cost of COVID-19 testing upon or to facilitate your return to your home base. 55