YEMEN MONITORING AND EVALUATION PROJECT, PHASE II (YMEP II) PERFORMANCE EVALUATION OF THE USAID YEMEN ECONOMIC STABILIZATION AND SUCCESS (YESS) PROGRAM Submitted January 16, 2020 Resubmitted February 11, 2020 This publication was produced for review by the United States Agency for International Development. It was prepared by International Business & Technical Consultants, Inc. (IBTCI) under AID-279-C-15-00001. YEMEN MONITORING AND EVALUATION PROJECT, PHASE II (YMEP II) PERFORMANCE EVALUATION OF THE USAID YEMEN ECONOMIC STABILIZATION AND SUCCESS (YESS) PROGRAM February 11, 2020 CONTRACT AID-279-C-15-00001 Prepared for Jessica R. Pomerantz, YMEP II Contracting Officer’s Representative Monitoring and Evaluation Specialist Middle East Bureau United States Agency for International Development Prepared by International Business & Technical Consultants, Inc. (IBTCI) 8618 Westwood Center Drive, Suite 400 Vienna, VA 22182 USA Tel: (703) 749-0100 DISCLAIMER The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. TABLE OF CONTENTS Acronyms ...................................................................................................................................i Executive Summary .................................................................................................................1 Evaluation Design 1 Summary of Findings and Recommendations 1 Evaluation Purpose & Evaluation Questions .........................................................................8 Evaluation Purpose 8 Evaluation Questions 8 Project Background .................................................................................................................9 YESS Program Theory of Change 9 YESS Components 10 Evaluation Methods & Limitations .......................................................................................13 Full Findings Conclusions & Recommendations..................................................................15 QUESTION 1 15 QUESTION 2 25 QUESTION 3 30 Annexes.................................................................................................................................................... Annex 1 Annex 1: Evaluation Statement of Work Annex I – 1 Annex II: YESS Utilization of CLA Principles Annex II – 1 Annex III: Data Collection Summary Annex III – 1 Annex IV: Changes to Evaluation Design Annex IV – 1 Annex V: Inception Report (Full Evaluation Design and Data Collection Instruments) Annex V –1 Annex VI: Sources of Information Annex VI – 1 Annex VII: Disclosure of any Conflicts of Interest Annex VII – 1 TABLES Table 1. YESS Beneficiary Data Collection ..........................................................................................................................1 Table 2. Evaluation Questions................................................................................................................................................8 Table 3. Progress toward Trade Outcomes as Reported.............................................................................................. 11 Table 4. Progress toward Livelihood Outcomes as Reported...................................................................................... 12 Table 5. Progress toward Income Generation Outcomes as Reported ..................................................................... 13 Table 6. YESS Beneficiary Data Collection ....................................................................................................................... 13 Table 7. SMEs Included in Data Collection....................................................................................................................... 14 Table 8. "Overall, to what extent has the YESS assistance helped to improve port management and operation?"............................................................................................................................................................. 18 Table 9. "To your knowledge, have port operations been streamlined due to YESS assistance?"......................... 18 Table 10. "To your knowledge, have port capacities increased due to YESS assistance?" ...................................... 18 Table 11. "Do you feel that humanitarian assistance is being delivered more effectively due to YESS assistance?"..................................................................................................................................................... 19 Table 12. Utilization of YESS Trade Training ................................................................................................................... 20 Table 13. "Overall, to what extent has the YESS assistance helped to increase your farm yields?"...................... 26 Table 14. "If you have experienced an increase in farm yields resulting from your interactions with YESS, how has YESS helped increase your yields?"........................................................................................................... 26 Table 15. "Do you feel this increase in yields will be the same in five years?"........................................................... 26 Table 16. "Have you experienced new market linkages as a result of YESS? Meaning, have you experienced new relationships with suppliers, buyers, cooperatives, etc.? ............................................................................. 27 Table 17. "If 'yes,' have these new relationships been beneficial?" ............................................................................... 27 Table 18. "Were you trained on any of these agriculture practices through YESS? Bi-ridge planting method?" 27 Table 19. "Were you trained on any of these agriculture practices through YESS? Drip irrigation system?"..... 27 Table 20. "Were you trained on any of these agricultural practices through YESS? Greenhouses?".................... 28 Table 21. "How likely are you to use your new training on a weekly basis for the foreseeable future?" ............ 28 Table 22. "Is there a negative impact of the ongoing conflict on your ability to use this training on a regular basis?"............................................................................................................................................................................. 28 Table 23. Overview of SME Data Collection ................................................................................................................... 34 Table 24. Respondent Length of Current Employment ................................................................................................. 35 Table 25. Pre-existing Employment Status........................................................................................................................ 35 Table 26. YESS Employment Generation, SME Employment Transition ..................................................................... 37 FIGURES Figure 1. YESS Theory of Change....................................................................................................................................... 10 Figure 2. MEG/YESS Feedback Loop Approach ............................................................................................................... 12 Figure 3. YESS High-level Theory of Change.................................................................................................................... 15 Figure 4. Port of Aden Trade Flows................................................................................................................................... 19 Figure 5. USAID Standard Development Hypothesis for Trade................................................................................... 21 Figure 6. Performance Indicator Reference for YESS, Indicator 3b ............................................................................. 35 ACRONYMS AEO Authorized Economic Operator AMEP Activity Monitoring and Evaluation Plan AMELP Activity Monitoring, Evaluation, and Learning Plan ASYCUDA Automated System for Customs Data CDCS Country Development Cooperation Strategy CLA Collaboration, Learning, and Adaptation CSA Climate Smart Agriculture CVA Customs Valuation Agreement DFID UK Department for International Development EG Employment Generation FFP Office of Food for Peace FGD Focus Group Discussion FTE Full-time Employment GOY Government of Yemen IBTCI International Business and Technical Consultants, Inc. IP Implementing Partner KII Key Informant Interview MEG Middle East Economic Growth Best Practices MEL Monitoring Evaluation and Learning NCTF National Committee on Trade Facilitation PA Partnership Agreement PAD Project Appraisal Document PIR Performance Indicator Reference Sheet ROI Return on Investment SME Small and Medium Enterprise TANA Technical Assistance Needs Assessment TF Trade Facilitation ToC Theory of Change TTF Trade Task Force TTWT Trusted Trader Working Group UNVIM United Nations Verification and Inspection Mechanism for Yemen USAID U.S. Agency for International Development USG United States Government WASH Water, Sanitation, and Hygiene WTO World Trade Organization YCA Yemen Customs Authority YESS Yemen Economic Stabilization and Success Program YESS Final Evaluation Report 1 EXECUTIVE SUMMARY This report captures the results from a performance evaluation of the Yemen Economic Stabilization and Success Program (YESS) funded by the United States Agency for International Development (USAID). This section includes a brief outline of the evaluation findings and recommendations, with reference to a more complete discussion of the program background, evaluation questions, methodology, and full findings in the full report. This summary is intended to be a quick reference to evaluation conclusions and has limited reference to data collection and discussion of findings. Evaluation Design This was a mixed-method performance evaluation comprised of document review, key informant interviews (KIIs), focus group discussions (FGDs), and surveys, both in-person (“live”) and on the telephone. This is a summary of Annex V, which contains the full Inception Report. Table 1. YESS Beneficiary Data Collection YESS Component KIIs Live Surveys Phone Surveys FGDs Trade 24 54 - 2 Livelihoods 10 - 120 - Economic Growth 18 418 - - Total 52 472 120 2 The conflict in Yemen created high levels of insecurity, which prevented data collection in the north of Yemen and face-to-face surveys with livelihood beneficiaries in the south. The live surveys were then replaced by telephone surveys, which necessitated shorter surveys. Data was collected only in Aden and Hadramout. Summary of Findings and Recommendations QUESTION 1: In what ways has YESS assistance contributed (or not), in a meaningful if limited way, to the economic stabilization of Yemen? Findings • Stabilization is not adequately defined. The YESS program has no consistent or measurable definition of “economic stabilization” as a goal for YESS. This goal statement of YESS combines multiple programming areas into a single “Program,” resulting in a disjointed theory of change (ToC) with an overly broad goal statement. • Beneficiaries see success. YESS has had success in creating “stability,” subjectively defined by beneficiaries/respondents, in the ecosystems in which the various YESS beneficiaries operate. Conclusions • There is no consistently used or documented definition of “economic stabilization” against which it may be measured. • The three YESS Components (Trade, Livelihoods, Employment) all have their own context￾specific definitions of “economic stabilization.” • The use of FGDs as a complexity-monitoring tool to assess the YESS goal of economic stabilization may not have been correctly applied. YESS Final Evaluation Report 2 Recommendation: “Economic Stabilization” may be an appropriate outcome for YESS, but was contextually defined for each YESS Component with a local systems or similar approach. Each YESS Component should be measured by its own contextualized “Economic Stabilization” indicator. These Component level indicators should then feed into an aggregate goal indicator that would simply be a summation. QUESTION 1.1: To what extent has the YESS assistance directly resulted in improvements in customs and port operations to include World Trade Organization (WTO) trade agreement provisions and YESS performance indicators? Findings • Output targets have been met, but evidence for outcomes is lacking. The Trade component of YESS met all of its output-oriented performance indicators (i.e. trainings held, policies developed, etc.) but there is insufficient evidence to validate the accomplishment of targeted outcomes. This is not to say that these outcomes have not been achieved, just that there is a lack of evidence to validate them. Much of the evidence that could have validated outcomes, such as reduced processing times, would come from administrative records and other management documentation that is not available. This type of record-keeping is a standard operating procedure of a port or customs authority and the documentation should be readily, if not publicly, available. Upon discussions with USAID, it appears they are seeking to address this in follow-on programming. In spite of the lack of evidence, YESS trade beneficiaries reported high levels of satisfaction with their trainings and the level of partnership, professionalism, and expertise of the YESS team. • Lack of government mitigates outcomes. Any outcome dependent on the active, sustained, and constructive engagement of a central Yemeni government cannot be completely achieved during the life of the project. Hence, progress and sustainability toward outcomes such as the World Trade Organization (WTO) agreement, which is largely dependent on a functional and responsive Yemeni government, is unobtainable at present. • There is no evidence of increased trade flows. There is no evidence of change in trade flows, or any type of cargo, that can be attributed to YESS. There are fluctuations in the level of trade flows, even more so for specific types of cargo, but more time will be needed to determine if these fluctuations will lead to an overall increase over time. Even if they do, it will be extremely difficult, if not impossible, to attribute these increases to YESS, given a high level of interfering variables. Conclusions • YESS Trade met all of its output indicators, but there is a lack of evidence to determine that training led to improved performance. • An overreliance on output-related objectives and indicators means that behavior-change outcomes were not as well articulated. • The lack of a functional government and other external influences prevent the full accomplishment of the targeted outcomes. • There is no increase in trade flow that can be directly attributed to YESS; lack of evidence makes it impossible to determine what may have contributed to increased trade flows. Recommendations: USAID should consider a greater emphasis on articulating, assessing, and documenting the factors influencing the intermediate behavior changes necessary to achieve higher￾level outcomes such as reduced processing times, etc. In the case of Trade, for example, the YESS YESS Final Evaluation Report 3 staff established Action Plans1 with different beneficiary groups to chart progress on the uptake of trainings. These Action Plans could be expanded upon, given that they 1) may be the highest-level outcome achieved during the programming; and 2) provide the most valuable evidence on behavior change that could influence subsequent programming. QUESTION 1.2: Has the YESS collaborative partnership/dialogue with key Government of Yemen (GOY) and other stakeholders identified gaps/weaknesses in current assistance which should be addressed in future programming? If so, what kinds of assistance are needed? Findings External factors played a very prominent role in mitigating Trade results. The primary external factors were the absence of a functional GOY – but there were also delays in cargo shipments due to interference of the Saudis and Emiratis in cargo inspection and approval. Per feedback from YESS staff as well as from beneficiaries, these external factors must be resolved if YESS outcomes are to be achieved and sustained. Relationships drive outcomes. In interviews with YESS staff, the importance of “trust” in various relationships was frequently highlighted as critical to outcomes. Trust was a common success cited by all Trade stakeholders. Evidence indicates this as a positive unanticipated outcome of YESS, as these types of relationships are often neglected in capacity-development programming of this type. There was an effective focus on “function” as opposed to simply “form.” • Future programming is needed. The YESS Trade team made several observations regarding the need for Aden port officials to attract future investments to modernize capital equipment. Specifically, there is a need to replace outdated tugs that consistently break down and to establish an updated training center for the Customs Authority. YESS produced and updated a report entitled Fuel Market in Yemen: Current Status and Strategic Recommendations, which provided a series of recommendations for short-term interventions that would improve the current humanitarian crisis in Yemen. This report also contains recommendations for future programming that can be updated and adapted. Conclusions • There is evidence suggesting that the gains made under this component of YESS may not be sustained, given the number of respondents who indicate that there has been no change in practice. • Despite the absence of a functional government, YESS enjoyed enthusiastic support from those government officials and other stakeholders who were involved. • YESS produced a report entitled Fuel Market in Yemen: Current Status and Strategic Recommendations which provided a series of recommendations which still hold true. Recommendations: YESS should make better use of conflict-sensitivity analysis to highlight those significant external factors that hindered the accomplishment of targeted YESS Trade outcomes. Given the success of YESS in building relationships of trust, additional programming could be informed by identifying the relationships that influence outcomes. Tools such systems-mapping could 1 The evaluation attempted to obtain these Action Plans but was un-successful in doing so. “The ‘Reliable Trader’ is considered to be the most surprising result. We had never imagined that a trader would come with goods and have them cleared immediately.” —YESS Trade Beneficiary YESS Final Evaluation Report 4 help future programming identify, monitor, adapt to, and document these relationships to provide a transparent operating picture for internal program management and learning. QUESTION 2.1: To what extent did YESS technical assistance/training achieve its relevant indicators and result in increased agricultural productivity and market access improvements in key value chain sectors. Findings • There is risk to the use of Climate Smart Agriculture (CSA). Despite the prominent role of CSA practices being trained in through YESS, extremely low numbers of beneficiaries reported having received CSA training and being familiar with specific CSA practices such as bi-ridge planting or drip irrigation. Use of CSA was a documented YESS objective. Survey evidence found that only 2.5 percent of respondents reported receiving bi￾ridge planting training, while 5 percent said the same for drip irrigation and 1.7 percent for fertigation. The fact that there were such low numbers of beneficiaries who reported receiving the CSA training indicates problems in the targeted uptake of CSA practices. • There were increases in yields. Conversely, 81.7 percent of beneficiaries answered “significantly” or “somewhat” when asked about the extent to which “YESS assistance helped to increase your farm yields.” Respondents reported that YESS contributed to this increase in yields by teaching “how to feed and water bees/livestock and how to keep them clean.” Bee/livestock training was cited by many respondents as the biggest contribution to the increases and 77 percent of those respondents feel these increases will be sustained in five years. • There is a lack of market linkages. Only 30 percent of respondents reported that they have experienced new market linkages since YESS and only 27.5 percent of these respondents reported that the new market linkages were beneficial. • Trainings are highly sought. YESS livelihoods trainings were very popular and resulted in adding additional trainings and trainings where attendance was much higher than expected. There were, however, disruptions to trainings due to the security situation. Conclusions • An extremely low knowledge of CSA practices was reported, which poses significant risks to the application of CSA practices. • Eighty-two (82) percent of farmers noted a “significant” or “somewhat” increase in their yields due to YESS. • Seventy-three (73) percent of farmers think it is highly likely they will be using the techniques trained on by YESS in the foreseeable future. • Only 30 percent of respondents reported a new market linkage due to YESS and only 27 percent of those indicated that the linkage was a beneficial one. • The ongoing conflict has the biggest effect on Livelihood outcomes. Recommendations: It is very difficult to perform one-time trainings and expect sustained outcomes, unless the intervention and targeted outcomes are very simple, predictable, and designed/implemented based on evidence driven best practices.2 YESS Livelihoods had very good results given that it was largely comprised of a series of one-time trainings. However, evidence from 2 See the 3ie Evidence Gap Map, specifically “Adoption of Inputs and Practices.” Impact Evaluations at https://gapmaps.3ieimpact.org/evidence-maps/agricultural-innovation. YESS Final Evaluation Report 5 other evaluations shows that these one-time trainings are the least-effective method of achieving targeted outcomes – hence the emphasis on demonstration farms (which YESS also included) and alternative methods where there is more opportunity for back and forth with trainees.3 QUESTION 3.1: To what extent did YESS small- and medium-sized enterprise (SME) job-generation assistance achieve its goal and relevant indicators of improving business performance and self-sufficiency for relevant firms? Findings • Indicators were met, but there is a sustainability risk. In terms of the YESS Activity Monitoring, and Evaluation Plan (AMEP) indicators regarding full-time employment (FTE), YESS is a great success. However, the sustainability of these gains is questionable, given that the primary employee beneficiaries of YESS are highly qualified and have also largely already left the employment created by YESS. The war in Yemen has created a great demand for goods and services; whether these gains are sustainable after this immediate demand is alleviated cannot be answered with a one-time data collection. • FTE was successful, but there is a risk to sustainability. YESS “Job Generation” is measured by FTE, which is defined by 260 days of labor per year. The evaluation found strong evidence that these jobs were indeed created by YESS. However, this does not mean that there are over 3,000 people employed; it means that over 3,000 units of 260 days-of￾labor-per-year were created. There is reason to question the sustainability of these results. • There is large employee turnover. Of the 812 employees hired as a result of YESS that were included in the evaluation data collection, 246 employees had left the beneficiary SME. In Aden, among the 524 employees hired, 205 had departed employment by the time of data collection. In Aden, 42 percent of YESS employees have already left their employment. This finding may be the result of YESS primarily benefiting individuals who do not face substantial barriers to employment. The evaluation found that 54 percent of “newly employed” individuals had employment immediately prior to their employment through YESS. While this could be “better employment,” when considered in conjunction with turnover rates it could also be that YESS provided temporary employment for a substantial share of beneficiaries who were already previously employed and/or have already transitioned out of the YESS employment. • Action Plans add value. “Improved performance” and “self-sufficiency” of SMEs were articulated through Action Plans developed with YESS assistance. These Action Plans were meant to identify barriers to improved performance and self-sufficiency and provide road maps to their alleviation. In reviewing these Action Plans and through interviews with YESS staff and SME leadership, evidence suggests that the majority of SMEs successfully completed their Action Plans. • Customer preferences are an increasing priority. As evidenced by KIIs, customer preferences appear to be a high priority for SMEs, in line with the Buyer-Led Growth approach implemented. • There are limited data on competitiveness and similar topics. Perceptions of SMEs are mixed regarding the sustainability of YESS employment and sales growth gains. YESS collected sales data on 24 firms which all showed sales improvement. However, there are 3 Ibid. YESS Final Evaluation Report 6 no available data beyond sales to calculate “financial sustainability,” “profit rate,” or “competitiveness;” hence, these constructs cannot be measured. • Limited definitions lead to limited measures. The constructs of “new and better employment” are not sufficiently defined to be measurable. However, the evaluation did find that the majority of employees placed in employment under YESS were already employed and could come under a definition of “better employment,” depending on the definition. Conclusions • The evidence suggests that YESS benefited employees who already enjoy a strong competitive advantage in obtaining and keeping employment. Most employees placed by YESS already have jobs and a significant portion of employees placed by YESS have already left those jobs. • YESS achieved all of its relevant output indicators, but the sustainability of these gains is highly questionable. • “New and better employment” is not sufficiently defined to measure in a way that results in actionable evidence. • There is no way to determine the profitability and financial self-sufficiency of the SMEs included in YESS, given that only sales data were collected. These data suggest increases in sales. Recommendations: YESS economic growth could have benefited from more careful targeting of beneficiaries, both in terms of SMEs and employees. Targeting employees with higher barriers to employment could result in more collective job creation as opposed to possible job substitution, in which the already employed obtain different employment under YESS for a short period of time before they transition to other employment. QUESTION 3.2: To what extent did YESS business performance assistance to improve access to finance, improve or introduce new management practices, etc., result in enhanced competitiveness of SMEs? Findings • Action Plans provide clarity. The evaluation team asked SME leaders who received YESS assistance what they perceive had the biggest impact on their management practices. Forty￾five (45) percent responded with answers that fit within their respective Action Plans, suggesting that YESS had success in developing and utilizing Action Plans. Thirty (30) percent of SME leadership respondents felt that YESS made their business more competitive. Seventy-five (75) percent of SME leaders felt that these Action Plans correctly reflected their primary obstacles to increasing profits. • There is no increase in access to finance. No SME leader noted an increase in access to finance. “We are caring a lot about our services’ quality as we do all efforts to have clean product [the quality of bread and flour] with high quality to make customers satisfied.” —SME Owner “Work was going less smoothly and normally before [YESS support] but after [its] establishment we were provided help in facilitating delivering services. Now things are done smoothly and with less frustration.” —SME Leader YESS Final Evaluation Report 7 • Conflict threatens gains. When asked about the sustainability of progress made under YESS, 85 percent of respondents gave answers related to effects of the conflict, including lack of long term-financial partners, energy shortages/outages, poor water service, inflated prices (especially for fuel), and inflation. Overall, SME leaders reported that YESS had limited success in the areas of increasing access to finance, competitiveness, etc. This limited success was more due to the external effects of the conflict as opposed to the type and manner of assistance. Conclusions • YESS was a short-term program that correctly spent a significant portion of its time identifying and unpacking the problems each SME faced (i.e., the SME “market failure”). This left little time for technical assistance to overcome these problems which, assuming a steady state, could still expect to see impact. • SME-specific Action Plans seem to be very valuable tools in monitoring progress toward the integration of new management practices and other arenas. These Action Plans also serve the function of being a collaborative tool where the SME takes an active role in meeting the objectives and articulating barriers. • Action Plans have some “steady-state” assumptions; the plans were meant to address firm￾level market failures but did not look at external variables such as reliable electricity, fuel, or inflation (as these variables are safe assumptions in a steady state). However, data collected show that these assumptions did not apply given the conflict in Yemen, and thus the targeted plan objectives could not be reached. Recommendations: Given the low level of increases in access to finance, future programming could benefit from more formative monitoring that tracks the development of relationships between SMEs and other value-chain actors, whether financial or other. This monitoring would help to map out context-specific barriers that need to be mitigated in order to achieve market linkages. More attention should be paid to external variables that can diminish relevant results. In the case of YESS, an updated conflict-sensitivity assessment could possibly have identified many of the interfering variables that prevented SMEs from achieving an increase in competitiveness. YESS Final Evaluation Report 8 EVALUATION PURPOSE & EVALUATION QUESTIONS Evaluation Purpose This report captures the results from a performance evaluation of the Yemen Economic Stabilization and Success Program (YESS) funded by the United States Agency for International Development (USAID). The primary audience is USAID/Yemen and the USAID/Middle East Economic Growth Best Practices (MEG) project. Other primary intended users include the implementing partner (IP) of the YESS activity (Pragma) and other USAID missions in similar contexts. Overview This performance evaluation seeks to: 1. Inform USAID and stakeholder decision-making 2. Contribute to learning on similar theories of change in similar contexts 3. Provide a transparent and well-documented measure of accountability These objectives are met by producing evidence that can be leveraged into new programming through collected quantitative data (surveys) and qualitative data (key informant interviews (KIIs), focus group discussions (FGDs)). Evaluation Questions This evaluation was guided by six questions covering the three primary components of YESS: Trade, Livelihoods and Job Creation. Activities involving the Central Bank of Yemen were not included in this evaluation per guidance from USAID. Table 2. Evaluation Questions Evaluation Questions Data Sources Analytical Methods Overall QUESTION 1: In what ways has YESS assistance contributed (or not), in a meaningful if limited way, to the economic stabilization of Yemen? KIIs, FGDs Qualitative information derived from KIIs will be assessed for level of effectiveness of YESS activities in achieving the overall MEG goal. Component 1: International Trade Facilitated QUESTION 1.1: To what extent has the YESS assistance directly resulted in improvements in customs and port operations to include World Trade Organization (WTO) trade agreement provisions and YESS performance indicators? GOY port management data to the extent available and verified, KIIs, FGDs YESS data will be analyzed in context with relevant available Government of Yemen (GOY) port/trade data. Information will be assessed for level of effectiveness of YESS assistance. QUESTION 1.2: Has the YESS collaborative partnership/dialogue with key GOY and other stakeholders identified gaps/weaknesses in current assistance which should be addressed in future programming? If so, what kinds of assistance are needed? KIIs, FGDs Assessment of perceived gaps as to how and to what extent additional types of assistance could be effective in the challenging context of the conflict in Yemen. Recommendations for future programming YESS Final Evaluation Report 9 Component 2: Agricultural Productivity Improved/Livelihoods Expanded QUESTION 2.1: To what extent did YESS technical assistance/training achieve its relevant indicators and result in increased agricultural productivity and market access improvements in key value chain sectors.? Surveys, KIIs Perceived increases against evidence in the field of increased production and/or sales to the extent relevant data is available. Component 3: Employment Generation/SME Competitiveness Improved QUESTION 3.1: To what extent did YESS SME job generation assistance achieve its goal and relevant indicators of improving business performance and self-sufficiency for relevant firms? SMEs documentation, GOY sector data for comparison, surveys, KIIs, FGDs Triangulation across available data points to identify growth and expansion attributable to YESS to the extent that relevant data is obtainable/generated. QUESTION 3.2: To what extent did YESS business performance assistance to improve access to finance, improve or introduce new management practices, etc., result in enhanced competitiveness of SMEs? SME revenue/sales data, KIIs, FGDs, financial institution data, sector data if available Assess relative levels of SME financing for YESS assisted SMEs and general level of sector SME growth/performance. PROJECT BACKGROUND YESS Program Theory of Change As stated in the YESS Activity Monitoring, Evaluation and Learning Plan (AMELP), the objective of MEG/YESS is to increase economic stabilization in Yemen, by achieving three primary outcomes: (1) productivity in selected value chains increased; (2) employment opportunities expanded; and (3) integration of private sector into international trade and finance networks improved. An overview of the YESS Theory of Change (ToC) is represented in the Figure 1 below. YESS Final Evaluation Report 10 Figure 1. YESS Theory of Change The YESS AMELP outlines three underlying assumptions that are critical to YESS success and are relevant to the evaluation findings: • Political stability – that there is sufficiently good governance to enable Yemenis to make decisions with the expectation that things will not change significantly over the next 12 to 18 months; • Macro-economic stability – that economic indicators either remain stable or improve, so that Yemeni firms and job seekers can make decisions with the expectation that the economy will not deteriorate significantly over the next 12 to 18 months; and • Security conditions remain the same or improve – in particular, that movement around the YESS operating environment does not affect the ability of the program staff to implement the activity. YESS Components YESS is comprises three primary components, which include Trade Facilitation, Agricultural Livelihoods, and Employment Generation, with additional activities that are outside the scope of this evaluation. The three primary components are summarized below. Trade Facilitation The Trade component made up of various analytical and technical assistance sub-activities meant to address key trade policy/institutional barriers hampering humanitarian assistance inflows to the population and limiting the integration of Yemen’s private sector into international trade and finance networks. The higher outcome of these activities is “Integration of Yemen’s private sector into international trade and finance networks improved” (Outcome #1 in the YESS Results Framework). This higher-level outcome is contributed to by several intermediate outcomes, including: • Reduce trading costs and delays caused by external factors. • Undertake a trade-facilitation pilot activity at the Port of Aden. • Based on prior mapping analysis, develop and initiate implementation of detailed recommendations for streamlining import/export procedures at key ports of entry. YESS Final Evaluation Report 11 • Build customs capacity in key areas. • Launch the implementation of the WTO Trade Facilitation Agreement. • Support the modernization of the Customs Legal Framework. • Improve access to fuel at competitive prices. Reported Results:4 Progress toward these outcomes was tracked by two primary indicators, hours of assistance and WTO facilitation, in the quarterly progress reports (Table 3). Table 3. Progress toward Trade Outcomes as Reported Indicator Target Actual Person hours of U.S. Government (USG)-supported training completed in trade and investment 500 2,670 Number of WTO Trade Facilitation provisions supported by USG assistance 5 6 Livelihoods The YESS Livelihoods component comprised numerous training and demonstration farm activities meant to increase the productivity of small farmers. This higher-level outcome of increased productivity (Outcome #2 in the YESS Results Framework) has several intermediate objectives with correlated activities to include: 1. Objective: Improved Climate Smart Agriculture (CSA) productivity in selected value chains • Technical Assistance Needs Assessment (TANA) • Technical Assistance Work Planning based on TANA • Farmers’ Field Schools • Demonstration plots • Capacity building for private and public agriculture service providers 2. Objective: Increased supply chain competitiveness • Capacity building for farmers on post-harvest and processing best practices • Training for targeted processors on ‘green’ processing methods/practices • Contract farming between value chains actors • Training for targeted processors on product development and certification • Capacity building for private certification bodies • Establish a national platform for each targeted value chain • Develop export marketing linkages with specialty markets 3. Objective: Increased small-producer access to financing • Link actors to microfinance institutions and banks • Support microfinance institutions and banks to develop new CSA-financing products 4 All reported indicator results are from the March 31, 2019 MEG Quarterly Progress Report, which is the last quarterly report for the period of performance for YESS that is covered under this evaluation. YESS Final Evaluation Report 12 • Support women’s empowerment in agriculture Reported Results: The Livelihoods component of YESS was tracked by three primary indicators in quarterly reports that cover application of new innovations, individuals receiving USG supported training and individuals belonging to some larger association (Table 4).5 Table 4. Progress toward Livelihood Outcomes as Reported Indicator Target Actual Number of farmers that have applied improved technologies or management practices with USG assistance 500 208 Number of Individuals who have received USG￾supported short-term agricultural sector productivity or food security training. 3,000 3,538 Number of targeted farmers and others who are members of a cooperative association, network or social group/value chain cluster 3,000 3,218 Employment Generation This YESS component aimed to increase viable employment opportunities across and promote Yemeni job seekers (Outcome #3 in the YESS Results Framework). This YESS Component used a Buyer Led Growth approach that was implemented across three strategic objectives: (1) Employment generation (EG) to promote growth and improving SME competitiveness by increasing small and medium sized enterprise (SME) market penetration; (2) Workforce development with a specific focus on vulnerable youth (especially those at risk of being recruited as combatants) and women; and (3) Access to financing in the form of institutional support to formal and informal financial providers to help them better understand SME credit needs. The YESS EG component utilizes a feedback loop approach that allows for the integration of different YESS components; including technical support, training/workforce development, institutional capacity building for financial service providers, and intensive support to leading value chains. The graphic below from the YESS Employment Work Plan illustrates this envisioned feedback. Figure 2. MEG/YESS Feedback Loop Approach 5 This does not include indicators tracking access to finance. YESS Final Evaluation Report 13 Reported Results: This outcome was measured by five indicators that assess performance on full￾time employment (FTE) creation, “new and better employment” opportunities and training outputs (Table 5). Table 5. Progress toward Income Generation Outcomes as Reported Indicator Target Actual Number of private sector jobs (FTE) created with USG assistance 1,500 1,235 Number of individuals with new or better employment following completion of USG-assisted workforce development programs 1,500 1,363 Number of individuals who complete USG-assisted workforce development programs 500 1,617 Number of firms receiving USG funded assistance for improving business performance 100 111 Number of days of USG funded technical assistance provided to SMEs 2,800 4,448 EVALUATION METHODS & LIMITATIONS This was a mixed-method performance evaluation comprised of document review, KIIs, FGDs, and surveys. Annex V contains the full Inception Report, which outlines the evaluation design, sampling, but is summarized here. Due to the ongoing conflict, there were limitations the evaluation team experienced. There were high levels of insecurity that prevented data collection in the north of Yemen and in-person surveys with Livelihood beneficiaries in the South. These surveys with Livelihood beneficiaries were replaced by telephone surveys, which necessitated shorter surveys. Data was collected only in Aden and Hadramout. A full description of all data collection scope changes is included in Annex IV. Table 6. YESS Beneficiary Data Collection YESS Component KIIs Live Surveys Phone Surveys FGDs Trade 24 54 - 2 Livelihoods 10 - 120 - Economic Growth 18 418 - - Total 52 472 120 2 Respondents included YESS implementation staff, USAID stakeholders, and beneficiaries of all three components. For Livelihoods, training attendees whose phone numbers were included in attendance sheets were randomly selected for a telephone survey. To identify Trade respondents, the evaluation used references from YESS staff and training attendance sheets to ensure recipients and supervisors were included. Employment Generation comprised the biggest scope for data collection, where every beneficiary SME in Aden and Hadramout was included. The evaluation used hiring-lists from SMEs to identify employees for surveys (all employees included on the lists were surveyed). The full list of SMEs is in Table 7 below. The full sample table with correlated references to the respective instruments is included in the Inception Report, Annex V. YESS Final Evaluation Report 14 Table 7. SMEs Included in Data Collection Hadramout Aden Employment Generation Al Morooj Hotels Hyper Panda Be Beautiful Center Al Reyada International Hospital Al Wasat Aden Gulf International Hospital Moj Hotel Al Shahba Automatic Bakery Al Rayan Hospital Alqotrah Healthy Water Al Nahal Shops EG72_Hadhramout Factory for Plastic Workforce Development Air Conditioning Cluster Aden Coral Hotel Al Ahram-Al Mukalla Career Fair Aden German International Hospital Hadhramout Microfinance Program Aden Cuban Hospital Muala Plaza Hotel Buraihi Hospital Halap Restaurant National Tobacco Company Walid Holding Group AISCO Gaz Company SBC Mall Yemen Soft Aden Beach Restaurant The analysis of YESS data collection was carried out in accordance with the approved Inception Plan, including: • Surveys: All surveys were statistically tabulated with disaggregation by gender, age, location, and component/sub-component. Microsoft Excel was used for quantitative analysis. • KII’s/FGD’s: All qualitative data collection were coded and assessed using content analysis (code book available upon request). The content analysis utilized a phased approach where additional codes were produced with each iteration, double-checked with another coder before a final code book was produced, and was then used for analysis. • Document Review: Document review utilized the same protocol for content analysis as the other qualitative data analysis outlined above with a phased approach and development of a code book. The evaluation did encounter limitations that stemmed from the design of YESS including: • Poor problem definition, unclear chain of causation in the ToC, and lack of definition of outcomes • Lack of contextual awareness of external influences on YESS outcomes • Improper use of outcome measures YESS Final Evaluation Report 15 • Inconsistent focus on behavior change to achieve outcomes • Unclear end-use of evaluation findings for Learning and Adaptation There were other limitations the evaluation faced that stemmed from the conflict. There were high levels of insecurity that prevented data collection in the north of Yemen and in-person surveys with Livelihood beneficiaries in the South. The live surveys with Livelihood beneficiaries were replaced by telephone surveys, which necessitated shorter surveys. Data was collected only in Aden and Hadramout. A full description of all data collection scope changes is included in Annex IV. FULL FINDINGS CONCLUSIONS & RECOMMENDATIONS QUESTION 1 In what ways has YESS assistance contributed (or not), in a meaningful if limited way, to the economic stabilization of Yemen? Evaluation Findings USAID program-design processes (identification of need, alternatives analysis, feasibility of proposed solution, and final design as outlined in ADS 201) are critical to evaluations, as they establish the problems to be addressed through programming. Without understanding the due-diligence process that creates programs, evaluations do no not have the ability to measure the solutions or the problems they are meant to help solve. YESS, like many international development programs, incorporates components that vary widely in their technical areas, objectives, and geographic areas of focus. YESS includes components on trade facilitation, agricultural livelihoods, and job creation among SMEs. Integrating these various components under an over-arching goal to which they are all intended to “contribute” is a difficult planning and design problem, which leads to overly broad and loosely defined goals that are difficult to measure. Figure 3. YESS High-level Theory of Change The three outcomes impact separate populations operating in largely distinct ecosystems. The USAID Learning Lab has published useful guidance on (1) what should comprise the causal relationships between outcomes and goals, as well as (2) what the attributes of a “valid” outcome are.6 Per this guidance, there should be clear explanations for why and how each lower-level outcome is a necessary precondition for the goal to be accomplished. Additionally, in order to 6 See website: https://usaidlearninglab.org/lab-notes/what-thing-called-theory-change YESS Final Evaluation Report 16 adequately measure it, each outcome should have clear attributes that define it, including population of interest, timelines for expected results, and behavior changes required. The YESS AMELP does not clearly define “economic stabilization” as the YESS Goal. It states only that “The objective of MEG/YESS is to increase economic stabilization in Yemen, which will be accomplished through the achievement of three component results.” There is no causal pathway defined from YESS component results to the overall YESS goal of Increased Economic Stabilization, which would provide clarity on how “economic stabilization” is defined. Hence, the ability to evaluate this is limited. For example, what is the population of interest for “Increased economic stabilization in Yemen?” Is it all of Yemen? Is it only the treatment populations of the three components? Without being able to define these populations, the first steps toward developing adequate measurement is not possible. The first step in identifying this population is during the program design process. For the three component-level outcomes, these attributes are much more clearly defined. USAID program design processes are meant to define the tentative parameters of the project, building upon the Country Development Cooperation Strategy (CDCS) Results Framework, and to provide a plan to complete the Project Appraisal Document (PAD). There was no PAD for YESS; PADs are not always produced. In mid-November 2017, senior executives from Pragma, Pragma’s partner Deloitte, and USAID travelled to Amman to meet with USAID/Yemen and donors operating in Yemen to better understand the economic and security conditions in the country and to define economic growth and related priorities. These discussions produced what were referred to as “concept maps” that broadly outlined needs and served as the beginning of program design for YESS. The evaluation team attempted to locate these “concept maps,” but they could not be found. Thus, the evaluation team had to rely on observations and available options of the YESS implementer, USAID, and other stakeholders to get a sense of the attributes of outcomes/goals that would normally be included in a PAD or “concept map.” This was a very helpful exercise, because it brought to light a wide variety of opinions on what YESS was supposed to do and for whom. Different stakeholders (e.g., USAID, Pragma, beneficiaries) defined “Economic Stabilization” in different ways, sometimes depending on the component of YESS they are speaking to. During interviews, some USAID personnel took more macro definitions of Economic Stabilization, specifically macro policies that were envisioned as outcomes of the YESS work with the Central Bank of Yemen. Perceptions of YESS beneficiaries of “economic stabilization” were gathered to help provide contextual understanding of the YESS goal, resulting in a wide variety of opinions on its definition. Opinions among beneficiaries were largely defined by the systems in which they operated. For example, farmers defined “economic stabilization” as it relates to input costs, agricultural prices, and/or ability of customers to pay for good/services. Hence, depending upon a respondent’s role as YESS staff, USAID representative, or beneficiary, “economic stabilization” is context-specific to a certain group of stakeholders – and their discrete component of YESS. Per the YESS AMELP, the goal of Economic Stabilization was to be measured through perception data with at least one FGD completed after a training event, usually towards the end of the period of performance for that specific Component. As outlined in the YESS Performance Indicator Reference (PIR) sheet: “Because YESS is a USAID pilot activity and will be implemented for only 12 months or so, the normal measures of economic stability are outside of the program’s manageable interest. As a result, the impact of the pilot activity will be measured through this perception indicator. The perceptions of YESS direct beneficiaries will be collected through an innovative ‘stakeholder feedback’ approach promoted by USAID’s Complexity Aware Monitoring brief.” YESS Final Evaluation Report 17 USAID’s guidance on Complexity Aware Monitoring defines it as a “type of complementary monitoring that is useful when results are difficult to predict due to dynamic contexts or unclear cause-and-effect relationships.”7 The criteria for when these tools should be applied was further defined as where: • cause-and-effect relationships are uncertain (a high degree of un-certainty about how to solve the problem); • stakeholders bring diverse perspectives to the situation, making consensus impractical; • contextual factors are likely to influence programming; • new opportunities or new needs continue to arise; and • the pace of change is unpredictable.8 These criteria are significant, as the ultimate measure of YESS’s achievement is dependent upon their successful application. The YESS use of Complexity Awareness Monitoring does highlight a lack of clarity on the attributes of “economic stabilization” and the necessary cause-and-effect relationships contributing to it. This point was substantiated in KIIs and FGDs with YESS staff, who noted that they mostly saw “economic stabilization” as a side effect of obtaining their individual component outcomes – but only within the populations of interest for their individual components. For example, YESS Trade respondents commented that “economic stabilization” largely meant more consistent (i.e., predictable) flows of goods through ports, while Livelihood staff responded that it meant more stable economic conditions in beneficiary households due to more consistent agricultural markets. This variation in opinion suggests that Complexity Awareness Monitoring tools are appropriate to help alleviate the confusion around what the relevant problems/outcomes should be. These tools are meant to be used to gather data from stakeholders to build the ToCs and the programs to achieve them. In the case of YESS, the data-collection tools were primarily FGDs among beneficiaries. These data were not intended to inform the ToC and subsequent program design (i.e., they did not have a formative function), but instead were meant to measure the accomplishment of the goal of a ToC and program that were already designed (i.e., a summative or accountability function). This is not to say that YESS did not require Complexity Awareness Monitoring, but that the tool – in this case FGDs – appears to be misapplied as a measure of accountability instead of as a more formative, “monitoring” tool. Thus, the FGDs were not used to build an understanding around the necessary cause-and-effect relationships that enable accurate measurement. There is no documentation or evidence of any YESS programmatic changes made as a result of these FGDs (a key function of Complexity Aware Monitoring). Conclusions • There is no consistently used or documented definition of “economic stabilization” to measure against. • The three YESS Components (Trade, Livelihoods, Employment) all have their own context specific definitions of “economic stabilization.” • The use of FGDs as a Complexity Monitoring Tool to assess the YESS Goal of Economic Stabilization may not have been correctly applied. 7 USAID. Discussion Note: Complexity-Aware Monitoring. 2018. 8 Ibid. YESS Final Evaluation Report 18 Recommendations Programs with a variety of activities that address different problems for different populations in different sectors, as is the case with YESS, can have well-defined goals but these goals are usually centered around a common attribute that can be consistently measured. For example, a goal of household income could be consistently measured across program components similar to YESS but “economic stabilization” is comprised of numerous dimensions that are contextually defined. Thus, if “economic stabilization” is a desired outcome, it could be defined and measured according to these context specific dimensions, which could be done at the component level. QUESTION 1.1: To what extent has the YESS assistance directly resulted in improvements in customs and port operations to include WTO trade agreement provisions and YESS performance indicators? Evaluation Findings The evaluation was able to collect subjective perception data of YESS Trade beneficiaries to gain insight that could be combined with objective performance data on port operations. Ninety-five (95) percent of respondents indicated that they have not received any other assistance, so there is a high level of confidence that the results reported here are not the result of other programming. Table 8. "Overall, to what extent has the YESS assistance helped to improve port management and operation?" Response Percentage Significantly 15% Somewhat 50% Minimally 22% Not Applicable/ Decline to Answer 13% These results suggest that YESS was limited in its ability to improve port performance; subsequent questions attempted to determine YESS performance on port processes versus overall capacity to effectively perform port functions. Table 9. "To your knowledge, have port operations been streamlined due to YESS assistance?" Response Percentage Yes 48% No 35% Not Applicable/ Decline to Answer 17% Table 10. "To your knowledge, have port capacities increased due to YESS assistance?" Response Percentage Yes 35% No 50% Not Applicable/Decline to Answer 15% Some of the biggest problems impeding more efficient flow of trade in and out of the Aden port were outside the scope of the Trade facilitation team to address. The YESS Trade team reported YESS Final Evaluation Report 19 significant interference from Saudi Arabia and the Emirates in cargo inspection and approval processes. The Trade facilitation team documented these issues and communicated them to USAID, but they remained significant obstacles to more effective port processing times and trade flows. For example, as part of the prioritization of problems (according to impact), there was a general private￾sector request to address prolonged clearance time due to Saudi interference. This is a demonstration of the external factors influencing YESS outcomes that lie outside the scope of YESS programming. The influence of Saudi actions at the Port of Aden are a possibly insurmountable barrier to accomplishing YESS objectives there, despite the best practices of YESS Trade staff. The 13 percent gap between respondents who report that YESS improved port processes versus those who report YESS improved overall capacity is interesting – but not large enough to suggest a large variance between “form” (i.e., process) and “function” (i.e., capacity). One possible explanation for this gap is that YESS had a short timeline, where changes in process are easier to achieve than changes in performance, especially in the case of highly influential external factors that mitigate performance. However, it should be noted in the YESS Trade ToC, improvements in process are a precursor to improvement in “capacity” and “performance.” Thus, it could be that these subsequent outcomes have yet to materialize. Given the significant mitigation of these outcomes by external variables stemming from the conflict, however, there is a very high risk to outcomes beyond changes in process. Increasing the flow of humanitarian assistance was a key objective of YESS. In assessing the trade flow numbers, the evaluation could not find any significant difference in trade flows, to include humanitarian supplies, due to YESS assistance. However, this is not a summative judgment of Trade activities since these flows are dependent on numerous other variables outside of those influenced by YESS Trade activities. Table 11. "Do you feel that humanitarian assistance is being delivered more effectively due to YESS assistance?" Response Percentage Significantly 9% Somewhat 35% Minimally 22% Not at all 34% The Trade team, in its reporting to USAID, did highlight changes in overall trade flows as captured in Figure 4 below. Figure 4. Port of Aden Trade Flows9 9 Source: YESS monthly reporting to USAID YESS Final Evaluation Report 20 There was inherent difficulty in answering the “extent” of the Trade component of YESS per Evaluation Question 1.1 due to a more output-based design of Trade programming. For example, the results for economic growth are defined by a clear outcome with specific targets (number of FTE), whereas Trade has a more openly defined outcome definition (i.e., “facilitate the flow of humanitarian relief supplies, fuels, and commercial trade, and reduce trading costs and delays…”). Trade was primarily a training and dialogue package of activities. There are only a few Trade outcomes that are behavior-change oriented. For example, “develop a road map for Single Window” is output-based, while “risk management guidelines are adopted” is an outcome that requires behavior change. Generally speaking, YESS Trade achieved all of its output-based results, but there is a lack of evidence on outcomes that required behavior change. This is not because these outcomes were not achieved, but because either (1) these behavior-change outcomes are not identified/defined or (2) there is a lack of evidence that there was behavior change. One example of this is the time to clear customs for incoming cargo. While not a specifically targeted outcome, it is included as an objective in YESS Annual Reports and falls under this evaluation question. In interviews and FGDs with the YESS Trade team, it was noted that “Through the application for risk management and reduction in physical inspection of cargo and promotion for trade facilitation measures (through events), the clearance time by customs was reduced from five to two days” based on observations from the Yemen Customs Authority (YCA). Such a result would be a significant contribution to improvement in port operations. However, this result could not be validated with any type of empirical data, such as customs logs. Although this outcome may have been achieved, it cannot be substantiated beyond observational data. Additionally, while the Trade team performed various technical assistance and trainings around building capacity to build a WTO valuation standardized approach, due to the conflict there is no sitting government that can approve the necessary legal framework for its valuation in accordance with WTO rules. Per the 2018 Annual Report, “YESS evaluated the customs law for consistency with the WTO CVA and found the approach used by Yemen – which was agreed by the WTO – to be acceptable. Due to the conflict, however, Yemen has not adopted the necessary sublegal acts to complete the legal framework for supporting valuation in accordance with WTO rules.” Nevertheless, there were positive responses from beneficiaries on the value of the training they received (Table 12). Table 12. Utilization of YESS Trade Training “How effective do you feel YESS is in helping you use your new training?” “How likely are you to use your new training on a regular basis for the for￾seeable future?” Response Percentage Response Percentage Very effective 57% Very likely 87% Somewhat effective 37% Somewhat likely 8% Not very effective 4% Not very likely 3% Not effective at all 0% Not at all likely 2% Not applicable 2% Not applicable 0% Conclusions • YESS Trade met all of its output indicators, but there is a lack of evidence about the usage of that training that leads to improved performance. YESS Final Evaluation Report 21 • The lack of a functional government and other external influences prevent the full accomplishment of the targeted outcomes. • There is no evidence of increase in trade flows that can be attributed to YESS. Recommendations USAID has developed a Trade Indicators Handbook to help to “organize projects and their activities around clusters of highly researched development hypothesis.”10 This handbook outlines a standardized theory of change (development hypothesis) that includes the outcomes targeted by YESS as alluded to in Figure 5. Figure 5. USAID Standard Development Hypothesis for Trade This development hypothesis includes a variety of possible indicators for each level of outcomes. All outcomes in this standardized ToC are relevant, some more than others, to YESS. YESS targeted similar outcomes. The same can be said for the associated standardized indicators from the USAID guidance, most of which come from indexed indicators (i.e. World Bank “Doing Business” Indicators, Logistic Performance Index, etc.) collected on an annual basis. These indicators do not completely fit YESS outcomes, partly because they are collected annually and would thus not be an adequate measure of change for an 18-month timeline (the period of performance for YESS). However, there are more iterative indicators included in this guidance which could be informative for programming, like YESS, with shorter timelines. These indicators include those sourced from the relevant country Customs Authority that deal with time and cost to clear customs, number of documents needed to clear customs, open ended measures around risk management, etc. Some of these measures could have been used in YESS and provided a richer description for USAID learning. Indicators are only valuable, however, if the outcomes they measure are properly articulated. The USAID indicators referenced here primarily measure behavior-change outcomes, meaning that in order to use these indicators YESS would need to articulate outcomes around the attributes. This includes the who, what, when, where of the specific behavior changes needed to improve port management. YESS made good use of Action Plans to monitor the progress towards targeted improvement and these actions plans could create targeted behavior changes and chart progress towards these behavior changes. And given that YESS Trade applied a learning-by-doing coaching approach, there is great opportunity to assess field-level barriers (and successes) towards these behavior changes. 10 Stephanie Monshein and Molly Hageboeck. Trade Indicators Handbook, USAID. 2013. YESS Final Evaluation Report 22 QUESTION 1.2: Has the YESS collaborative partnership/dialogue with key GOY and other stakeholders identified gaps/weaknesses in current assistance which should be addressed in future programming? If so, what kinds of assistance are needed? Evaluation Findings A helpful way to look at YESS Trade is through accepted best practices, especially given largely qualitative evidence. For this the World Customs Organization Effective Customs Capacity Development (2003) is a helpful way to frame some of the evidence-driven best practices (e.g., Coordination, Commitment, Common Standards) integrated into YESS Trade, given that there is ample evidence that YESS facilitated effective relationships among stakeholders using these practices. Practice #1- Coordination: Trade helped to establish a Trade Task Force (TTF) involving representatives of key ministries and agencies of the Yemeni legitimate government, private sector, humanitarian aid organizations, the United Nations Verification and Inspection Mechanism (UNVIM) for Yemen, and other donors (e.g., DFID, World Bank). There is observational data from beneficiaries and YESS staff that the information on trade flows and port operations collected by YESS was helpful to stakeholders on the TTF and beyond, given the lack of information previously available. The Trade team reported that port officials are engaging/interacting on a daily basis with the private sector. The Trade team facilitated private-sector engagement with government officials in two public-private roundtable discussions. Generally, both sides report higher levels of collaboration. The support from high-level Yemeni officials like the Deputy Minister of Agriculture, Deputy General Director of Import, and Vice Minister of Finance have assisted in building these relationships but, as noted elsewhere in this report, while YESS Trade enjoyed enthusiastic support from available government officials, the lack of a functioning government effectively mitigates many of the targeted outcomes. Practice #2- Commitment/Ownership: The Trade team enjoyed good working relationships with Yemeni stakeholders at all levels of operation – from those with whom they worked side by side in a “learning-by-doing” capacity all the way to various Deputy Ministers. Pragma reported that they never lacked access to Yemeni officials when they voiced a need. It was noted that even with the high levels of support and collaboration that existed with the Yemeni stakeholders, the absence of a functional government is a substantial impediment to additional gains and possibly sustainability of current ones. This is compounded by the fact that once government functions return, it will take a significant period of time before those responsible for managing port operations can be stood up, trained, etc. This is partly mitigated by the training of trainers approach of YESS but given the largely complete absence of a functioning government, the only real mitigation is the presence of a valid government meeting its responsibilities for a consistent period of time. Further evidence of ownership is seen in the establishment of the Trusted Trader Working Group (TTWT) to support roll out of AEO and the support from YCA and others for the pilot at the port of Aden. The TTWT is up and running while the National Committee on Trade Facilitation (NCTF,) meant to serve as an overall forum for various trade related issues to include a WTO Trade Facilitation Agreement, is still in the process of being formed. YESS enjoyed broad support and these officials have taken additional actions to build on the success of YESS. During data collection, it was observed that “high-profile state authorities decided forming an economic committee to organize imports by the private sector and provide [hard] currency funds to importers.” There is evidence among government officials and other available stakeholders that the value in YESS Trade objectives was perceived. Practice #3- Common Standards: The Trade team worked with YCA to complete the guidelines for risk management which were subsequently approved by the Acting YCA Chairman. YESS Final Evaluation Report 23 The Trade team further reported an increased use of streamlined processes, such as more selective container inspections at the Port in Aden. Additionally, the YCA with Trade support was able to complete the necessary draft regulations for launching the “Authorized Economic Operator” (AEO), which provides for trusted traders and processes that result in more efficient clearance (without a functional government, these regulations may not be acted upon). Additionally, YESS helped Al Mukalla customs to integrate to the ASYCUDA system (the automated system for customs data). This will hopefully allow for more streamlined processing and shortened transition times. The Trade team noted that their work was greatly facilitated by their collaborative approach, which combined with the YESS team being well known among the relevant Yemen stakeholders, contributed greatly to building relationships and trust which were critical to the gains achieved. The emphasis on “trust” was an interesting finding of the evaluation which is further discussed below. Practice #4- Systematic Capacity: “Trust” is a construct that is difficult to measure but critical to success, as targeted Trade outcomes require trusting relationships. While YESS Trade technical assistance meant to install procedures and processes to facilitate trust – such as the Reliable Traders Program – these procedures and processes cannot themselves accomplish improvements in port operations. The evaluation found that the YESS Trade team took advantage of their role as facilitator, trainer, and expert to cultivate the necessary relationships to utilize the new processes and procedures to their full ability. Feedback from Trade beneficiaries noted the interplay between processes, procedures, and the new relationships that facilitate their use, especially in the case of Reliable Traders. “We experience easier work procedures due to an increase in mutual trust between Customs staff and traders.” “Traders’ manipulation of data and papers. Now, with introduction of ‘reliable trader’ criteria, things have got better.” “The ‘Reliable Trader’ is considered to be the most surprising result. We had never imagined that a trader would come with goods and have them cleared immediately.” “The surprising result is creation of mutual trust between the trader and the Customs Authority.” Practice #5- Needs Assessment: The overall YESS approach to Trade combined a standardized approach that team members have implemented in numerous contexts, with a Yemen-specific approach that identified barriers that could be mitigated in a learning-by-doing fashion. The Trade team took a problem-driven approach to their work, with initial problem identification completed from Dec 2017–Jan 2018. Daily workshops were conducted in Amman with different stakeholders for two weeks to capture the initial problems. These were followed by diagnostic studies on identified problem areas per the YESS work plan and quarterly reporting to USAID. Given the iterative “learning-by-doing” approach used, problems were consistently identified and resolved as part of the technical assistance process. As problems emerged, the Trade team noted that it never lacked support from the appropriate stakeholders to address the issue – however, the negative impact of the lack of a functioning government cannot be overstated. This iterative process of problem identification led to consistent adaptations to the technical assistance provided, resulting in a tailored approach to building the specific capacity that was needed. However, as further elaborated on in the “Recommendations” section, the fact that the Trade team was problem-driven means that some measures of progress are only defined by the alleviation, or at least partial mitigation, of identified problems. While Trade had its formal indicators, its problem￾driven approach creates an opportunity for at least qualitative measures around specific problems and how they were solved (whether it be through technical assistance or some other method). YESS Final Evaluation Report 24 Practice #6-Measurement: In the case of Trade, there is no doubt that all the envisioned activities were accomplished in a very transparent and measurable manner. Trade had high levels of reporting in its quarterly and monthly reports to USAID; the content of this reporting covered activities (workshops, trainings) and descriptive statistics (trade volumes). The reporting did not, however, include documentation of the operational-level problems involved (barriers) in meeting the various Trade outcomes. On the other hand, numerous respondents, both beneficiaries and YESS implementers, noted that the reporting of various dimensions of port performance (trade flows, etc.) was very useful in providing the data needed to inform stakeholder discussions. In the past, these data had not been available. Future Programming: The Trade component provides learning to develop overall “Port Infrastructure Modernization” projects that can be communicated to other donors for future assistance. Regarding future programming, the Trade team made several observations that Aden port officials require future investments to modernize equipment and training facilities/curriculum. Specifically, they currently have outdated tugs that consistently break down and need an updated training center for the Customs Authority. YESS produced and updated a report entitled Fuel Market in Yemen: Current Status and Strategic Recommendations which provided a series of recommendations on short-term interventions that would improve the current humanitarian crisis in Yemen. Figure 4 demonstrates that flow of fuel through the port of Aden has generally not increased and discussions with the YESS Trade team indicate that all of the recommendations contained in this report are still valid. There were additional suggestions among beneficiaries on potential future programming. One observer noted that “The [AEO] System has been implemented only in the Free Zone. Our work in the airport is mostly manual, particularly after the war because the airport’s server and systems were damaged by bombardment.” Future programming need not expand into new areas. Approximately 60 percent of Trade beneficiary respondents reported that they did not feel some of the technical assistance has led to changes in practice. This sentiment was reported as “participants gained personal benefits from the training only. There was no success because the program (AEO) is not applied on the ground.” In following up on this general observation, however, it was observed by beneficiaries that this was not limited to AEO and that there were no significant or clearly apparent barriers to “application on the ground.” The precise (not merely observational) reasons for this lack of application require additional research best suited for the formative work done as part of future programming preparation. Conclusions • There is evidence suggesting that the gains made under this component of YESS may not be sustained given the number of respondents who indicate that there has been no change in practice. • Despite the absence of a functional government, YESS enjoyed enthusiastic support from those government officials and other stakeholders who were involved. • YESS produced a report entitled Fuel Market in Yemen: Current Status and Strategic Recommendations which provides a series of recommendations that still hold true. Recommendations Documentation of Problem Solving: YESS followed an emerging best practice in using a technical assistance approach where problems are iteratively identified, solved, then new ones identified, solved, and so on. While this approach embraces a higher level of flexibility and informality, the problem-solving process should be carefully documented. YESS Final Evaluation Report 25 There was a significant amount of reporting by the Trade team, not only as a part of regular quarterly reporting but extensive monthly reporting on trade flows and activities at the various ports. However, per our discussions with USAID, this reporting did not lead to any informed decision-making;11 thus, its utility could be questioned. The quality of the reporting was high. The evidence from the emerging best practices around iterative problem-solving approaches notes the importance of reporting around what specific problems were identified and how they were solved, which produces a strong evidence base around how outcomes are or are not achieved. This is the type of evidence base that is most easily leveraged into replication/duplication of similar approaches in varying contexts. As noted, many of the Trade “outcomes” are actually outputs (e.g., developing materials, conducting workshops). These outputs are a necessity, but they are a necessity in order to leverage a behavior change that leads to improved port management (an increase in capacity). It is the behavior change, and from whom the behavior change is needed, that is critical to understanding the extent of outcomes as asked by the evaluation questions. YESS Trade indicators only measure outcomes, while there is a need for mapping the causal pathways between outcomes that becomes more critical for short-term programming like YESS. For example, YESS Trade aimed to increase the skills of government stakeholders (RF 2.3.1.1) in order to modernize clearance procedures (RF 2.3.2), but the link between these two outcomes is unclear in terms of what the operational uptake of the skills looks like and how it can be assessed. In short, the YESS Trade activities did what they were designed to do and if more evidence-driven results are required, the design of activities must shift to more behavior-change outcomes with correlated monitoring data that can iteratively gather evidence on the necessary change. QUESTION 2 QUESTION 2.1: To what extent did YESS technical assistance/training achieve its relevant indicators and result in increased agricultural productivity and market access improvements in key value chain sectors? Evaluation Findings The evaluation team used the Agricultural Livelihood’s training sheets to administer telephone surveys from October 1 to November 30, 2019. This resulted in 120 valid phone surveys among beneficiaries, in addition to interviews among YESS Agricultural Livelihoods staff and document reviews. To assess the contribution of YESS to agricultural productivity, respondents were asked about their yields (since “yield” translates across agricultural practices whether it is bee keeping or tomato growing). It should be noted that this is observational data, namely the opinion of the respondent, which while valued (given the respondent is the most qualified person to answer the question) needs to be qualified. Usually surveys dealing with farm yields ask not only about yields but level of inputs (fertilizer, seed, etc.) as well as proxy measures for income that usually assess consumption levels. However, as these phone surveys had to be very limited in duration, the evaluation team could not gather all data on these additional questions. But as over 75 percent of respondents claimed that YESS had “significantly” or “somewhat” increased their farm yields, it is highly likely that YESS did increase farm yields – possibly not at such a high rate. 11 It should be noted that the trade volume data has been used by USAID in conversation with other donors and international actors but no “decision” could be identified where the information was utilized. YESS Final Evaluation Report 26 Table 13. "Overall, to what extent has the YESS assistance helped to increase your farm yields?" Frequency Percent Significantly 33 27.5 Somewhat 65 54.2 Minimally 4 3.3 Not at All 18 15.0 Total 120 100 To help explain the “how” behind the “what,” the evaluation team asked the reason for the increase in yield. Table 14. "If you have experienced an increase in farm yields resulting from your interactions with YESS, how has YESS helped increas e your yields?" Reason Frequency Percent We learned how to feed and water bees/livestock and how to keep them clean 37 30.8 We followed all we learned in the training 6 5.0 We learned how to purify/clean honey/cells/animals 8 6.7 We learned how to use the modern methods/tools/materials to increase the production 18 15 We learned how to keep bees/livestock healthy/learnt how to know illness and the right treatment 13 10.8 I followed some instructions from beekeepers on how to keep honey clean and valid (honey sellers) 2 1.7 I have not worked yet since the training/have not seen the results yet 3 2.5 We learned how to choose the right male and female for fertilization to increase and improve the production 5 4.2 Other - - These findings correlate roughly to the composition of YESS Livelihood training curriculum. This means that the evaluation results do not offer conclusions on what aspects of the YESS training were more conducive to increasing yields over others. For those that did self-report an increase in yields, the vast majority felt that their yields would be the same in five years. Table 15. "Do you feel this increase in yields will be the same in five years?" Frequency Percent Yes 93 77.5 No 9 7.5 Other 18 15 Another primary objective of YESS Livelihoods assistance was to improve market linkages for beneficiaries to mitigate the problem (outlined in the Livelihoods workplan) of “Lack of effective linkages between farmers, exporters, and food processors.” The evaluation team asked about all potential market linkages (not just linkages to exporters) to better capture other possible linkages. YESS Final Evaluation Report 27 YESS took care to develop specific market linkage objectives for each cohort of beneficiaries, whether it be livestock (access to veterinary services, linkages with dairy processers), honey (exporters), coffee (upstream linkages), or others. Identifying the market-linkage constraints of each value chain is critical and a best practice; it demonstrates that YESS Livelihoods took the time to tailor activities to the specific needs of its beneficiaries. Table 16. "Have you experienced new market linkages as a result of YESS? Meaning, have you experienced new relationships with suppliers, buyers, cooperatives, etc.? Frequency Percent Yes 36 30 No 84 70 Other - - Table 17. "If 'yes,' have these new relationships been beneficial?" Percent Yes 27.5 No 2.5 Other 70 With only 30 percent reporting new linkages, and 27.5 percent of those reporting that they have been beneficial, there is strong evidence that the market linkages objective of YESS was not as successful as it could have been. However, there was no target for market linkages (e.g., a percentage of beneficiaries), so the level of results versus intentions cannot be determined. The evaluation team worked with USAID and YESS to identify the three main CSA practices that were trained on during the various Agriculture activities. These consultations resulted in three priority CSA practices that USAID and Pragma wanted feedback on from beneficiaries. These three practices included Bi-ridge Planting, Drip Irrigation, and Greenhouses. The results from data collection are summarized in the following tables. Table 18. "Were you trained on any of these agriculture practices through YESS? Bi-ridge planting method?" Frequency Percent Yes 3 2.5 No 117 97.5 Table 19. "Were you trained on any of these agriculture practices through YESS? Drip irrigation system?" Frequency Percent Yes 6 5 No 114 95 YESS Final Evaluation Report 28 Table 20. "Were you trained on any of these agricultural practices through YESS? Greenhouses?" Frequency Percent Yes 3 2.5 No 117 97.5 Table 21. "How likely are you to use your new training on a weekly basis for the foreseeable future?" Frequency Percent Very Likely 88 73.3 Somewhat Likely 30 25 Not Very Likely 2 1.7 Total 120 100 These questions were refined from a more general series of questions around whether beneficiaries received CSA training and if they are using it, to questions where specific CSA practices like bi-ridge planting are the focus of specific questions. This was done to avoid any confusion by the respondent beneficiary on what “CSA” is. During data collection, the enumerators followed up with respondents to ensure they understood the question in full. This context is important when considering the results from data collection, since at first glance it would appear that there is some confusion by respondents on the CSA. This is because there are very low response rates on having received training on these specific CSA practices, but very high response rates on the forward-looking applicability of the training. Since the evaluation took steps to mitigate this error, both in designing the questions and in administering them, confusion by respondents does not explain these data. Table 22. "Is there a negative impact of the ongoing conflict on your ability to use this training on a regular basis?" Frequency Percent Yes 51 42.5 No 69 57.5 If “Yes” is the impact: Very Significant 33 27.5 Somewhat Significant 15 12.5 Not Very Significant 2 1.7 Not at all Significant 1 .8 Other 69 57.5 Fifty-one respondents consider the ongoing conflict to negatively affect their ability to use YESS training and 48 percent of those consider the effect to be “somewhat” or “very significant.” This should be taken within the context that 81 percent of respondents indicated that YESS increased their farm yields. When considered together, the evidence indicates that the ongoing conflict is a possible significant risk to sustaining YESS results. YESS Final Evaluation Report 29 Conclusions • There is an extremely low reported knowledge of CSA practices that were trained on, which poses significant risks to the application of CSA practices. • Eighty-two (82) percent of farmers noted an “significant” or “somewhat” increase in their yields due to YESS. • Seventy-three (73) percent of farmers think it is highly likely they will be using the techniques trained on by YESS for the foreseeable future. • Only 30 percent of respondents reported a new market linkage due to YESS and only 27 percent of those indicate that the linkage was a beneficial one. • The ongoing conflict has the biggest effect on Livelihood outcomes. Recommendations Additional programming could use formative research to help alleviate the confusion around why the CSA training was less effective than hoped for, and then integrate results into the subsequent programming. The lack of success in establishing market linkages could stem from a variety of causes. YESS was a short-term program with little value chain or other system mapping used to design the activities. It may be that YESS could have benefited from a better understanding of the relevant market systems and value chains for the targeted beneficiaries, which may require more time and other resources for planning and designing subsequent programming. The effects of the conflict are substantial, according to the reports of beneficiaries. The potential disruption to results is significant and could be better understood by performing a conflict sensitivity analysis on the relevant theory of change and how the conflict dynamics influence those causal relationships. Without this understanding, which can be formatively tested and built upon during any subsequent programming, sustaining any short-term success is at risk. There is an evidence now being developed around other secondary and unintended effects of programming similar to YESS. For example, there is mounting evidence that training on innovations like CSA practices or facilitating market linkages leads to food security outcomes before economic outcomes. This means that agricultural households often experience an increase in caloric intake on a more consistent basis before they experience an increase in farm/household income or value chain competitiveness. These food security improvements often correlate to an increase in yields, meaning that either (1) the household is consuming farm surplus or (2) they are selling the surplus and using the revenue to for food security purposes.12 Hence, while YESS focused on “competitiveness” for value chains, there are numerous other outcomes along the causal pathway to competitiveness that could be targeted, measured, and learned from. Given that value-chain “completeness” has numerous outcomes that could be achieved on that pathway to competitiveness, subsequent programming could ensure that these outcomes have their own evaluation questions and a program design that sets up a test for each outcome. Some additional evaluation questions could include: • What are the intermediate outcomes that could be achieved and measured on the pathway to improved completeness? • Which training approaches are most effective in introducing specific food products or other innovations to farming communities? 12 https://www.3ieimpact.org/sites/default/files/2019-01/sr19-africa-smallholder-farming-review.pdf YESS Final Evaluation Report 30 • What approach to value-chain mapping is most conducive to making changes to programming during implementation? • Which type of farmers benefit most from agricultural training? • Which type of training is most effective in encouraging the consumption of more nutritious food? • What is the impact of training on male versus female farmers? • What are the indirect benefits of smallholder farming interventions, and who are the potential beneficiaries? YESS did include demonstration farms in addition to the trainings, but there were only a few (plus “Farmer Field Schools”). One-off trainings composed the bulk of Livelihood activities. Without any evidence that could explain the decision making, there are no PAD or “concept maps” from the YESS design process; it is therefore impossible to test the rationale for this composition of activities. There is strong evidence, though, that activities like demonstration farms or farmer field schools are much more conducive to the application of new, knowledge and practices like those taught by YESS, because they offer beneficiaries multiple chances to retain knowledge, ask questions, and receive more specific instruction or feedback on their application of the new knowledge.13 This does not mean that Livelihood activities should consist solely of demonstration sites and farmer field schools, but that the application of new knowledge by beneficiaries is facilitated by multiple opportunities for knowledge transfer and feedback. There should be a lower dependence on one-off trainings and more allowance for iterative trainings with the same beneficiaries to provide more opportunity for knowledge capture. QUESTION 3 QUESTION 3.1: To what extent did YESS SME job generation assistance achieve its goal and relevant indicators of improving business performance and self-sufficiency for relevant firms? Evaluation Findings From the YESS AMELP indicators regarding full-time employment (FTE), YESS is a great success and it achieved all of its performance indicators. Likewise, there is solid evidence that YESS beneficiary￾SMEs have improved their performance in the short to possibly medium term. Whether these gains are sustainable cannot be conclusively answered with a one-time data collection, although there is strong evidence to doubt the sustainability of these gains. Sustainability is an issue further discussed below regarding YESS’s utilization of a Buyer-Led Growth Strategy. YESS “Job Generation” is measured by FTE, which is defined per the Performance Indicator Reference sheet (PIR) as 260 days of labor a year. The evaluation found strong evidence that these jobs were indeed created by YESS, but this does not mean that there are over 3,000 people employed; it means that over 3,000 units of 260-days-of-labor-per-year were created. Generally speaking, there is some evidence that YESS improved the competitiveness of beneficiary SMEs in Yemen. During the course of designing and carrying out the evaluation, it become apparent that “Buyer-Led Growth” was integrated into YESS and was of great interest, not only to USAID but other 13 https://www.3ieimpact.org/sites/default/files/2019-01/sr19-africa-smallholder-farming-review.pdf YESS Final Evaluation Report 31 stakeholders as well. In answering the relevant evaluation questions (3.1 and 3.2), this report will use the principles of Buyer-Led Growth to organize the evidence. Buyer-Led Growth is a general strategy with a wide variety of applications. Most applications focus on cultivating demand as measured through the number, size and other attributes of transactions. With this focus on “transactions” as the ultimate objective, the best monitoring for a Buyer-Led Growth Strategy is often the customers themselves – since the business is defined by the demand for its goods and services. The ToC for the EG component of YESS is somewhat unclear on the causal relationship between supply and demand. Job creation is supply, transactions are demand; this means job creation is only sustainable given a proportional creation, or meeting, of demand as measured by transactions. With this relationship, there can be a temporary increase in job creation, but this increase will be alleviated over time if there are insufficient transactions to sustain the “new or better employment.” This is the rationale for the Buyer-Led Growth general ToC that begins with transactions and works backward to supply. YESS focused on increasing jobs by addressing firm-level market failures. YESS targeted firms that had pent-up demand, but could not meet the demand for a variety of reasons (a firm-level market failure). YESS delivered the technical assistance needed to meet the demand as the SMEs hired new employees (or “improved the jobs of current employees”). This YESS economic growth causal chain can be summarized as: 1. Firms are selected that have pent-up demand, but have a firm-level market failure which mitigates meeting that demand; 2. Technical assistance addresses the market failure which helps the firm meet the demand; 3. Meeting this demand creates the revenue to sustain the jobs that the SMEs commit to creating or improving. Selected SMEs agree to hire a certain number of people before the technical assistance begins and the hiring of these employees is tracked as a YESS indicator. YESS employed a Buyer-Led Growth Strategy in different way, due to the priority focus on “jobs” with a lesser focus on “cultivating demand” centered around transactions. This “jobs” focus could be more centered to humanitarian efforts as opposed to long-term development. If part of the purpose of this evaluation is to help determine whether/how USAID can perform development work in Yemen, YESS EG has limited ability to answer that question, because YESS EG could be considered at least partly a humanitarian program, due to the supply-centered focus on job creation. This is not to say that Buyer-Led Growth is only for development programs. YESS EG provides an opportunity to learn from how a Buyer-Led Growth strategy that focuses on jobs in preference to cultivating demand, in a humanitarian context, can lead to success. To provide evidence, the evaluation team presents some of its findings below on YESS EG organized by Buyer-Led Growth principles. This is done to make the evidence more actionable in speaking directly to principles in practice. Principle #1: Make “Demand Driven” Real. Guidance on Buyer-Led Growth indicates that customer profiles should be articulated and that “demand” should be defined “as buyers with first and last names, not statistical or market studies.”14 Implicit in this definition is (1) careful selection of beneficiary firms (SMEs in the case of YESS) and (2) this selection is driven by articulation of “demand” as specific buyers. 14 We do Know Now: A Buyer Led Approach to Creating Jobs for the Poor, James Riordan. New Academia Publishing, 2014. Pg. 45. YESS Final Evaluation Report 32 This starts with a transaction from a buyer outlining in detail the required product quantities, specifications, and quality standards. As Riordan states, “The buyer-led approach to job growth is based on the idea that there can be no sustainable creation of jobs or increase in income without accounting for and designing around the continuous and definitive judgment of consumers. In a certain sense, these consumers—external to the project, unmanageable, and largely unpredictable— are the only ‘project monitors’ who really matter. The buyer-led approach looks beyond individual transactions to the sustainability of commercial relationships between buyers and sellers over time.”15 YESS used the criteria below to assess SMEs for eligibility to receive assistance.16 These criteria were part of a larger process of developing individual Action Plans for each beneficiary SME that laid out milestones and measures of progress for mitigating the firm-level market failure constraining their competitiveness. The YESS Job Creation component targeted specific SMEs that were facing specific constraints to growth, including: • Low level of productivity • Unskilled workforce • Inadequate management • Inability to innovate • Inadequate quality controls • Ineffective marketing strategies • Underdeveloped value chains • Lack of effective branding • Restricted access to capital Criteria for beneficiary selection are critical to program success in general and Buyer-Led Growth strategies in particular. Because Buyer-Led Growth is generally “demand driven” it is important that beneficiary businesses are selected based on demand for their goods and services. However, such an analysis requires data around “demand” that is difficult to gather in Yemen. Yemeni SMEs are not working in an organized regulatory economic system where contracts are enforced. Generally, companies do not disclose their revenues and sales history to outside parties, making “demand” hard to articulate. While these criteria are clearly documented, the terms are vague and do not transfer easily to objective assessment. In addition, there is no documentation of this assessment and no score cards exist that capture how each SME scored against the criteria, further obscuring beneficiary targeting thresholds. Once an SME is accepted, the Partnership Agreement (PA) is drafted and rationale for the decision is included in the PA itself, reviewed by Pragma senior management. During this process, some firms may be dropped, as they may not meet enough of the criteria for selection. In reviewing these PAs, it is easier to get an idea of the firms that were selected for inclusion but because the application of the criteria was not documented (for understandable reasons given the 15 Ibid., 215. 16 The original YESS EG Workplan called for working with SMEs as well as “Medium-sized to larger companies or so called ‘Anchor Firms’ with a large network of upstream SMEs as partners and suppliers.” However, initial assessments by the YESS team found that these Anchor Firms were largely absent from Yemen, so the work was confined to SMEs; no Anchor Firms were included. YESS Final Evaluation Report 33 short timelines, large number of SMEs reviewed, and limited YESS resources) it is impossible to identify a threshold for a qualifying versus disqualifying SME. Principle #2: Focus on Specific Problems. Some guidance on Buyer-Led Growth indicates that the primary problems to be mitigated are transaction-specific, in essence, to solve the problems that stand in the way of meeting specific buyers’ requirements for specific transactions. YESS did take a problem-driven approach, but in way that transaction-specific problems would only be addressed if they were also a problem for job growth. YESS EG would be “buyer-led” only when the problems that mitigate job creation (the YESS objective) are also the same problems that stand in the way of meeting specific buyers’ requirements (Buyer-Led Growth objective). It is possible that an assessment of how “buyer-led” YESS is would require assessing the level of overlap between (1) the problems hampering job growth that YESS addressed with (2) the problems that stand in the way of specific transactions. As part of the PAs, YESS developed SME-specific Action Plans that outlined timelines around specific milestones that alleviate barriers to job growth. These Action Plans are a rich source of data for learning (discussed below under the Principle #3, below). Principle #3: Manage for Results. Managing for results is also problem-driven, since “results” are the alleviation of problems. The evaluation team found ample evidence that YESS did a very thorough job of managing for results. YESS did this by implementing flexible technical assistance that was responsive to emerging issues in meeting the milestones outlined in the various SME Action Plans. Likewise, YESS was very careful to work with the SMEs in building ownership of the Action Plan and the corresponding roles and responsibilities of the SME in terms of reporting on job growth, among other categories. Principle #4: Facilitate Buyer-Seller Relationships. Because Buyer-Led Growth focuses on cultivating demand as its primary goal, Buyer-Led Growth programs tend to facilitate relationships between buyers and sellers. These relationships help better understand buyer needs relating to specific products or to expand into new markets. YESS did not facilitate buyer and seller relationships in this manner, but it did facilitate relationships between job seekers and employers through the Workforce Development activities of YESS EG that met all its performance indicators. Principle #5: Organize Geographically. Organizing geographically takes advantage of organically formed relationships of trust. These relationships could drive business transactions that, hopefully, mature to the point that “clusters” form in business corridors. In these cases, mutually aligned incentives drive down transaction costs, spur innovation, etc. Thus, it is critical that Buyer￾Led Growth approaches let the market choose what businesses to work with so that they can sustainably mature into clusters. Many Buyer-Led Growth approaches select businesses to work with based on (1) demand for products and (2) geographic location in an economic corridor that has high likelihood for maturation into a cluster. YESS selected SMEs with the most potential to grow (where “growth” is defined as job creation). During interviews with YESS staff, it was noted that as part of the initial planning, SMEs usually provided a “wish list” but YESS fine-tuned barriers to growth to those that were evidence￾driven. Evidence shows that there are SMEs that YESS worked with which do have potential to mature into clusters, but it is not clear that this was a specific criterium that was applied in SME selection. YESS was a pilot project with a short timeline, which creates the necessity for YESS to demonstrate results in short order. This necessity has direct implications for the selection of SMEs and objectives pursued. Perhaps due to these parameters, YESS placed a higher priority on SMEs that could create jobs during the YESS period of performance than it did on market-driven demand or potential for SMEs to mature into clusters. YESS Final Evaluation Report 34 To validate the FTE reported by Pragma (per Evaluation Question 3.1), YESS provided PAs and employment lists for all beneficiary SMEs, as well as aggregations of individuals placed in new and improved jobs per SME (used to calculate FTE numbers for indicator 3a). The team randomly selected 15 Job Creation and 10 Job Matching SMEs and their reported numbers from the aggregated sheets and then checked these against the employment lists submitted by the SME. For all selected SMEs, the team found no discrepancies between the submitted employment lists and the aggregated numbers used for indicator calculation. When the evaluation team collected data on these identified employees at the various SMEs, YESS field officers facilitated introductions to SME managers.17 These employee lists were reviewed by the SME manager or with the SME human resources (or employee affairs) officer who reviewed the lists and provided updates on which individuals were still employed, who had left employment or were never employed by the SME at all. Table 23. Overview of SME Data Collection SME # of Respondents SME # of Respondents Aden Germany Hospital 54 Hyber Banda 9 Aden Gulf International Hospital 22 Mooj Hotel 13 Al Borehi Hospital 12 Mu'la Plaza Hotel 5 al-Ahram Company 1 Qatra Water 5 Al-Rayan Hospital 46 SBC MALL 60 Al-Riyadah Hospital 45 Yemen Soft 2 Al-Shohaba Bakery 17 Air Conditioning Cluster_Basaa Industrial Complex 23 Al-Wasat Company 4 Air Conditioning Cluster_as-Sediq as￾Seefi 25 Coral Hotel 14 Be Beautiful Center 18 Cubi Hospital 8 Hadhramawt Microfinance Program 2 Halab Restaurant 32 Aden Beach Restaurant 1 Total: 418 Of the 418 who responded to the survey question, the average age is 27.5 years old. Ninety-three (93) percent of respondents reported that they were employed full time at the time of the survey. Fifty-five (55) percent of respondents indicated they had been offered a contract with their current employer. 17 To identify employees to be sampled, the evaluation team used hiring and training lists provided by YESS that had been validated by the SMEs (as the SMEs had been reporting on the employment of these individual employees). Enumerators collaborated with YESS field staff, who facilitated introductions to SME leadership who confirmed the employment results reported. The full sampling methodology can be found in Annex V. YESS Final Evaluation Report 35 Table 24. Respondent Length of Current Employment Length of Current Employment % of Overall Respondents Less than one month .2 Between one and six months 6.5 Between six months and a year 39 Between a year and 1.5 years 16.7 More than 1.5 years 37.6 There is a 20-percent literacy gap between the average Yemeni population and our sub-sample. This assumes that the measure is accurate and that our question and the way we phrased did not introduce significant bias. Bias is unlikely given that the same population of survey respondents also has much higher levels of education obtained than the general Yemeni population, hence are more likely to have higher levels of literacy. Forty-six (46.1) percent of those surveyed completed secondary school, compared to an average of 30 percent for the general Yemeni population,18 with an additional 20.3 percent of those surveyed having completed a post-secondary institute certification/degree. This suggests that those who obtained “new or better” employment through YESS have much greater human capital than the average Yemeni. This is also consistent with the findings that 54 percent of these beneficiaries had preexisting employment before YESS (i.e., they had full-time work up to the time they transitioned to the “better” employment created under YESS). Table 25. Pre-existing Employment Status Were you employed immediately before your current employment? % Was this previous employment full-time? % Yes 54.1 Yes 81.4 No 45.9 No 18.6 This last finding is significant since it challenges the construct of “better employment” in the YESS outcomes as measured by indicator 3b (Figure 6). Figure 6. Performance Indicator Reference for YESS, Indicator 3b Number/Name of Performance Indicator 3b: Number of individuals with new or better employment following completion of USG-assisted workforce development programs (EG 6-1) PERFORMANCE INDICATOR DESCRIPTION USAID Standard Definition: This indicator will be used to monitor change in employment of individuals following participation in workforce development activities. It will provide, along with other assessments, a general sense of the scale and reach of programs aimed at generating new or better employment. It is also a good proxy indicator for increased income and reduced poverty. It will be used, along with other indicators, to describe progress toward USAID Education Strategy goals related to workforce development and Agency-level priorities in several areas of interest including economic and youth development. 18 See website: https://data.unicef.org/topic/education/secondary-education/ YESS Final Evaluation Report 36 The possible issue with this indicator definition is that the indicator is attempting to measure “new or better employment…of individuals following participation in workforce development activities.” There could be problems with the use of the word “following” because, while understandable from an “attribution” perspective, it creates a scenario where individuals could already be fully employed and merely experience a neutral change in employment.19 Hence the baseline for any Return on Investment (ROI) calculation would necessitate a much higher baseline to assess “better employment” against, not a baseline that assumes under or un-employment. This is further muddled by the inconsistency of how “following” is used in the definition where it is initially used to say “following completion” then “following participation” (as bolded in the figure above). For comparison, when taking baseline measurements for a WASH intervention, enumerators ask about incidence of water-borne illness in the previous few weeks, not on the day the intervention begins. If a similar baseline approach was taken for this indicator, then the baseline would include the preexisting employment status of the beneficiaries and the majority of beneficiaries under YESS EG were already employed. If these previously employed individuals were only included in the count for “better” employment as opposed to “new,” then only 18.6 percent of these employees could be counted, as the remaining 81.4 percent were fully employed.20 The construct of “better employment” is never defined. During data collection among the YESS EG implementation team, it was observed that “better” meant more hours of labor per week. If this definition is applied, then only the 18.6 percent that were previously employed can be counted under “better employment,” since they were not fully employed. This is with the caveat that they are fully employed now, or at least have more employed hours than before, which may not be true. Not all of YESS employment is full employment; some is part-time employment. Survey results do seem to indicate that YESS EG placed individuals in employment that were largely already employed and, given their education level and employment history, do not seem to have significant barriers to finding employment. This is not meant to obscure the fact that 45.9 percent of individuals placed in employment by YESS did not have a job immediately prior to YESS (and hence they can be counted as “new employment”). However, unless the jobs vacated by the other 54.1 percent were filled by previously un- or under-employed individuals, then collectively there is no “job creation” per se, only job substitution. USAID has noted the importance of beneficiary targeting, especially in dynamic environments,21 while leading development actors like the World Bank22 and industry thought leaders23 have also emphasized its importance in achieving envisioned outcomes. This means that outcomes happen for specific people with specific characteristics and not abstract constructs outlined in a ToC. There is some lack of clarity about who exactly the EG component of YESS was targeting. Project materials suggest the focus was on firms with relatively low “growth” barriers that could be overcome with 19 A possible flaw with this argument is its assumption that the individual did not choose to change employment based on any perceived benefit (e.g., closer to residence, better working hours, less physically demanding labor) which seems unlikely. The question remains whether these reasons for the change in employment are included in the definition of “better” under the indicator. 20 This is self-reported full-time employment as assessed by the respondent and not FTE as measured by this indicator. 21USAID, Beneficiary Targeting and Verification Guide for FFP Emergency Food Security Activities. Retrieved from: https://www.usaid.gov/sites/default/files/documents/1866/Beneficiary_Targeting_Verification_guidefor Emergency_April2017.pdf. 22 World Bank, Reaching the Poor and Vulnerable: Targeting Strategies for Social Funds and other Community-Driven Programs. Retrieved from: http://siteresources.worldbank.org/SOCIALPROTECTION/Resources/SP￾Discussion-papers/Social-Funds-DP/0711.pdf 23 See website: https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/833.pdf YESS Final Evaluation Report 37 technical assistance. Hence the “targeting” was primarily oriented towards SMEs that had unmet demand, or in other words, were experiencing a SME-level market failure. The lack of clarity around the targeting of SMEs for inclusion (which type of SME was needed to achieve the targeted outcomes) in YESS has been noted. The YESS team built and utilized a network of professionals that included business leaders, Chamber of Commerce, etc., to identify potential SMEs for inclusion in YESS. These SMEs were then engaged and assessed according the criteria outlined above for final decision on inclusion. YESS took a Buyer-Led Growth approach to job creation. Such an approach is meant to “sustainably increase sales.” The idea is to stimulate employment opportunities under the theory that an increase in demand for business services/products will result in an increase in demand for labor to produce those services/products. This typically starts with outlining in detail the demand for specific product quantities, specifications, and quality standards. Referring back to Riordan, “The buyer-led approach looks beyond individual transactions to the sustainability of commercial relationships between buyers and sellers over time.”24 Sustainability is a component of YESS Indicator 3.2.d, which measures “firms that are more profitable or associations that are more financially self-sufficient due to USG assistance.” YESS collected sales data for baseline (2018) and actual (2019) from 24 SMEs based in Sana’a, Aden, and Al Mukalla. These data have different units of analysis for sales, ranging from bottles, to tons, cases, patients, and actual revenue to name a few. These different units of analysis make a correlated comparison impossible; the sample makes any extrapolation to the larger YESS SME beneficiary population very limited. However, it should be noted that in all 24 cases, SMEs reported an increase in sales in their respective units (cases, tons, etc.). However, YESS Indicator 3.2.d is meant to measure “profitability” and “financial self-sufficiency,” which require more than sales or revenue data to calculate. There are no data available on cost, depreciation values, or other information needed to assess “profitability” and “financial self￾sufficiency.” Thus, any conclusions on “financial self-sufficiency” and “profitability” are not possible; there are only sales data available for a small portion of beneficiary SMEs. While financial self￾sufficiency is an attribute of sustainability, looking at the turnover of those employees placed under YESS can offer evidence on the sustainability of the “new or better employment.” Table 26. YESS Employment Generation, SME Employment Transition Governorate Total # of Employees # of Employees who have left SME % of employees who have departed # of Employees who never worked at SME % of employees who never worked at SMEs Aden Job Creation 195 84 43% 1 -- Workforce Development 329 121 37% 0 -- Total 524 205 -- 1 -- Hadramout Job Creation 197 34 17% 13 7% 24 James Riordan. We do Know Now: A Buyer Led Approach to Creating Jobs for the Poor. New Academia Publishing, 2014. YESS Final Evaluation Report 38 Workforce Development 81 7 9% 0 -- Total 288 41 -- 13 -- There is no readily available explanation for the high turnover rate in Aden as opposed to Hadramout. It should be noted both governorates have relatively high turnover rates for such a short period of time in a conflict environment where there is extremely high unemployment. However, this finding does seem to reinforce the probability that YESS benefited employees who have much greater capacity to find employment than the average Yemeni. Increasing evidence shows that in conflict environments, short-term job growth interventions should be designed with an integration of conflict dynamics in the short run, and on longer-term goals for economic transformation. Job growth in fragile situations is difficult and interventions with limited reach are crucial tools for emergency relief, stabilization, and starting momentum in private-sector recovery. However, they must be designed to facilitate larger structural stabilization and reform. YESS was affected by conflict dynamics (e.g., work stoppages) but it is unknown to what degree conflict dynamics influenced the demand for goods, services, and labor. This is partly due to the lack of a tailored conflict sensitivity analysis. There is no evidence that YESS or any follow-on programming to YESS is aiming at the larger goals of economic transformation or structural reform which may be necessary preconditions to facilitating sustainability of YESS EG outcomes. This said, YESS did target and complete reform-related activities with the Central Bank of Yemen, but these activities were not completed as originally envisioned. While YESS did focus on the immediate ecosystems surrounding the SMEs with which they worked (see YESS indicator 3a), these ecosystems are not independent and rely on larger macro conditions. The gains made under YESS in terms of job creation have built the momentum but, given evidence from similar interventions, these gains may be difficult to sustain without more macro-economic stabilization. This is especially informative given the conversation around the YESS goal of “economic stabilization.” Conclusions • The evidence suggests that YESS benefited employees who already enjoy a strong competitive advantage in obtaining and keeping employment. Most employees placed by YESS already have jobs and a significant portion of employees placed by YESS have already left those jobs. • YESS achieved all of its relevant output indicators but the sustainability of these gains is highly questionable. • “New and better employment” is not sufficiently defined to measure in a way that results in actionable evidence. • There is no way to determine the profitability and financial self-sufficiency of the SMEs included in YESS given that only sales data were collected, despite the fact that these data suggest increases in sales. Recommendations While YESS did articulate and document the criteria it used to select SMEs for inclusion, the application of these criteria was not documented, hence there is limited evidence on the adequacy of these selection criteria. A process for applying the criteria to SMEs for their inclusion that includes an assessment of the adequacy of the criteria could provide valuable evidence for which SMEs should be targeted. YESS Final Evaluation Report 39 More precise definitions of constructs like “new and better employment” can provide a threshold of comparison for more actionable evidence in future evaluations. Other constructs like “profitability” and “self-sufficiency” should only be used as targeted outcomes if there is an appropriate monitoring and evaluation approach to collect the necessary data to assess how well these outcomes were achieved. In the case of YESS, only sales data were collected to assess “profitability” and “self￾sufficiency” when additional data around costs, among others, were required. There could be improvement in beneficiary targeting, given that YESS might have helped those employees who were in the least amount of need. This raises the question of whether this was a SME- or employee-focused program. Likewise, if Buyer-Led Growth is a preferred strategy, there should be more deliberate structuring of how the program is designed and implemented, with a focus on demand by specific people for specific products/services with specific attributes (cost, quality level, etc.). QUESTION 3.2: To what extent did YESS business performance assistance to improve access to finance, improve or introduce new management practices, etc., result in enhanced competitiveness of SMEs? Evaluation Findings The evaluation asked SME leaders who received YESS assistance what they felt the biggest impact of YESS was on their management practices. Forty-five (45) percent responded with answers that fit within their respective Action Plans, suggesting that YESS had success in developing and utilizing the Action Plans. Thirty (30) percent of SME leadership respondents felt that YESS made their business more competitive. Seventy-five (75) percent of SME leaders felt that these Action Plans correctly reflected their primary problems in increasing profits. Twenty-five (25) percent of SME leaders observed that they had been introduced to financial service providers, but no SME leader reported that these introductions have resulted in a lasting relationship. Thus, no SME leader indicated that they have benefitted from an increase to finance under YESS. This is not to say that YESS did not result in an increase in access to finance for some SMEs, just that no SME included in the evaluation reported this. When asked about the sustainability of progress made under YESS, 85 percent of respondents gave answers related to effects of the conflict, including lack of long term-financial partners, energy shortages/outages, poor water service, inflated prices (especially for fuel), and inflation. Overall, the perceptions of SME leaders seem to indicate that YESS had limited success in the areas of increasing access to finance, competitiveness, etc. This limited success could be more due to the external effects of the conflict than to the type and manner of assistance. Conclusions “Competitiveness” and “Increase access to finance” are not immediate-term outcomes. These constructs take time and a reasonable level of stability to materialize. YESS was a short-term program that, correctly, spent a significant portion of its time identifying and unpacking the problems each SME faced (SME-level market failure). This left little time for the technical assistance to overcome these problems which, assuming a steady state, could still expect to see impact. However, it could be that while YESS collaborated with SME to identify barriers that became the focus of the various Action Plans, these Action Plans could have had some steady-state assumptions that do not apply in this case. The Action Plans were meant to address firm-level market failures but did not look at external variables like reliable electricity, fuel, or inflation as these variables are safe assumptions in a steady state. Given that Yemen is not a steady state, these variables become critical assumptions that may not have been correctly accounted for in the objectives for YESS Employment Generation component in general, and individual SME Action Plans in particular. YESS Final Evaluation Report 40 Given this, the SME-specific Action Plans are very valuable tools in monitoring progress towards the integration of new management practices. These Action Plans also served the function of being a collaborative tool where the SME takes an active role in meeting the objectives and articulating barriers. Recommendations If SMEs are the focus of the intervention, then more time spent on diagnosing the firm-level market failures for each SME would benefit subsequent Action Plans. YESS staff did an incredible job of working with SMEs to ensure that the Action Plans were not merely a way to fulfill SME “wish lists” and were instead evidence-driven plans. This is a notable success given the limited amount of time YESS staff had to develop these Action Plans for such a large number of SMEs – time limitations could have easily incentivized the Action Plans becoming “wish lists.” Given the low level of increases in access to finance, future programming could benefit from more formative monitoring that tracks the development of relationships between SMEs and other value￾chain actors, whether financial providers or others. This monitoring would help to map out the context-specific barriers that need to be mitigated to achieve these market linkages. More attention should be paid to external variables that can diminish relevant results. In the case of YESS, an updated conflict-sensitivity assessment may have possibly identified many of the interfering aspects of the conflict that prevented SMEs from achieving an increase in competitiveness. ANNEX I. Evaluation Statement of Work Annex I – 1 ANNEXES Annex 1: Evaluation Statement of Work Proposed Technical Scope of Work for the Evaluation of USAID/Yemen Yemen Economic Stabilization and Success Program Draft – May 22, 2019 ANNEX I. Evaluation Statement of Work Annex I – 2 Acronyms ADS Automated Directive Systems AMELP Activity Monitoring, Evaluation and Learning Plan CLA Collaborating, Learning and Adapting DQA Data Quality Assessment FGD Focus Group Discussions GOY Government of Yemen IP Implementing Partner KI Key Informant KII Key Informant Interview M&E Monitoring and Evaluation MEG Middle East Economic Growth Best Practices Project MSC Most Significant Change SME Small and Medium Enterprises USAID United States Agency for International Development USG United States Government YESS Yemen Economic Stabilization and Success Program ANNEX I. Evaluation Statement of Work Annex I – 3 INTRODUCTION The Yemen Economic Stabilization and Success Program (YESS) is a two-year $15 million USAID activity designed as a “pilot project” to test innovative approaches to promote economic stabilization in Yemen. YESS is being implemented by the Pragma Corporation. It is funded under the broader USAID Middle East Economic Growth Best Practices Project (MEG). This Proposed Technical Scope of Work lays out the preliminary objectives, general observations on methods, tentative schedule and operational considerations for both an evaluation of the performance of YESS to date. It will also evaluate the effectiveness of the activities of this pilot program. It will examine what has, or has not, worked and why. Background Yemen is a country in conflict and identified by the UN as the world’s worst humanitarian crisis. It is one of the least advanced countries globally in terms of open and accountable governance, economic development, social equality, and citizen health and safety, as reflected in Yemen’s FY2019 USAID Country Roadmap25. The Roadmap shows a country suffering from significant political, social, and economic upheaval after four years of ongoing conflict. The conflict has greatly impacted the Yemeni economy, resulting in high rates of unemployment and increased levels of poverty. This highly political operating environment adds layers of complexity to the programming context. USAID implementing partners face many challenges in the field, not the least of which is measuring results. Thus, there is a need for a robust evaluation that can provide evidence-based data and information on which implementing partner (IP) and USAID managers can base future programming decisions. Evaluation Purpose The primary purpose of the evaluation is to independently measure performance, i.e., achievements against indicator targets, and assess key challenges that constrained successful implementation. Key Objectives The broad objectives of the evaluation are to understand: • The degree to which the assistance has been consistent with USAID and Government of Yemen (GoY) expectations (as defined by the program’s targeted indicators); United States Government (USG) economic growth policies; and the needs of project beneficiaries, i.e., the businesses and new employees assisted by YESS (Relevance). • The ways in which the assistance, directly and indirectly, resulted in expanded job/business opportunities in Yemen and improved trade flows at Yemen ports (Causality). • The extent to which assistance directly and indirectly resulted in expanded job/business opportunities in Yemen and improved trade flows at Yemen ports (Effectiveness). More specifically, it is the objective of this evaluation to assess the effectiveness of the various activities of the program’s three primary components, as follows: Component 1: International Trade Facilitated: To what extent and in what ways has YESS provided trade facilitation support that has enhanced trade flows? Component 2: Agricultural Productivity Improved/Livelihoods Expanded: To what extent and in what ways has YESS increased small farmer productivity and access to markets 25 See https://selfreliance.usaid.gov/docs/roadmaps/USAID_FY_2019_Yemen_Country_Roadmap.pdf ANNEX I. Evaluation Statement of Work Annex I – 4 with a view to growing agricultural livelihood opportunities, particularly for vulnerable populations? Component 3: SME Competitiveness Improved: To what extent and in what ways has YESS been effective in improving SME competitiveness as evidenced primarily by its ability to expand formal sector employment opportunities? Engagement with USAID The evaluation team will engage USAID at each step of the process (see Evaluation Schedule, Table 5 later in this proposal). Upon mobilization, the team will meet with USAID to review this proposal, revise as necessary and ensure that all activities of the evaluation are consistent with USAID’s Evaluation Policy (January 2011) and ADS 201. EVALUATION METHODOLOGY Approach The evaluation will use a mixed methods approach, including quantitative surveys of beneficiaries, and qualitative research via Key Informant Interviews (KIIs), and focus group discussions using Most Significant Change (MSC) protocols. Beneficiary survey data, and, for small and medium enterprises (SMEs), a corresponding on-site check of business and employment records (materials necessary to verify outcomes like revenue, expanded employment, etc.), will provide a basic verification of program achievements and serve as one tier of a triangulation of reported IP data, and stakeholders perceptions of program performance. Depending on the availability of relevant data, the evaluation will assess how well YESS identified and addressed the most critical constraints to SME business and job growth by analyzing patterns of growth against sector trends as an inference of improved competitiveness. These analyses will be supplemented by Key Informant Interviews where stakeholders will rate and provide perspective on which types of YESS assistance best helped to build business competitiveness, i.e., assistance to remove technical constraints in new product development, enhance marketing, strengthen financing, management, etc. Quantitative data from YESS employment beneficiary surveys (for both farm workers and SME employees) and GOY sector employment surveys, if available, will complement qualitative information from KIIs and focus group discussions to provide a more complete understanding of the extent and ways that different types of assistance effect job growth and sustainability. Where other relevant quantitative data, from USAID, GOY, academic institutions, development partners, the private sector or other independent sources, is available it will be used in conjunction with YESS data for a finer grain of analysis on YESS assistance effectiveness. “Most Significant Change”26, a participatory monitoring and evaluation technique, will be used to develop “storylines” of success, that is, how program interventions changed things for the better. Initially, through a process of field focus group discussions, change storylines will be identified and articulated. Evaluation team members will then lead follow-up discussions with key stakeholders as to the value of these change stories in addressing questions as to why and how certain types of YESS assistance have induced change on the ground in SMEs, with agricultural producers, for the unemployed, in port management and in trade facilitation. Stakeholders and beneficiaries come with different expectations, and carry with them different perspectives, about donor assistance. Knowledge on how best to achieve desired outcomes in the 26 The ‘Most Significant Change’ (MSC) Technique: A Guide to Its Use, Rick Davies and Jess Dart, April 2005 ANNEX I. Evaluation Statement of Work Annex I – 5 challenging context of a country in strife will necessarily be incomplete or uncertain if the evaluation does not engage with all actors in a meaningful way. The MSC protocol helps stakeholders to develop a shared understanding of the most critical issues and allows for a deeper discussion of the priorities in going forward. Steps and Sequence of Activities The steps of evaluation activities in relative sequence will be as follows: 1. Perform an Evaluability Assessment (to include RFI’s to USAID for data, sample frame sources, etc.) through: a. Desk review of IP reports, context data from donor sources, and other independent sources of context information. b. Document key issues identified in desk reviews. Search for “gap” field data that would be useful in the evaluation. c. Identify, collect and input into an appropriate data base management system (DBMS) (SPSS or other USAID preferred system), including critical indicators and other key performance data. d. Present Evaluability Assessment to USAID for feedback 2. Develop an Inception Report that captures all design elements of the evaluation to include sampling methodology, implementation approach and questionnaires for surveys of businesses and new employees and others that received YESS assistance, seek USAID approval. a. Present Inception Report to USAID for review and approval b. During development of Inception Report, IBTCI will produce a Data Collection TOR that will be used to procure the enumeration firm in Yemen to perform all data collection tasks. This TOR will be approved by USAID before it is used for an open procurement but because procurement of the enumeration firm will take approx. 5 weeks, we will seek approval of this TOR before approval of the Inception Report as a whole. 3. Conduct Data Collection (including quality control protocols) to include: a. Surveys b. Key Informants Interviews (KIIs) c. Focus Group Discussions (FGDs) d. MSC focus group discussions. 4. Perform Analysis and Follow up Data Collection a. Identify most significant change storylines from focus group discussions. Lead follow￾up stakeholder discussions about applying lessons learned to future programming. b. Perform necessary qualitative coding and quantitative statistical analysis c. Evaluation Team to reconvene for identification of key issues and discussion of preliminary evidence-based evaluations recommendations/conclusions. 5. Prepare draft report, submit to USAID for comments. 6. Revise draft, prepare materials to present findings to IP and other relevant stakeholders. 7. Prepare and submit final deliverables to USAID. A chart illustrating the timeline of these activities is presented in Table 5, later in this report. ANNEX I. Evaluation Statement of Work Annex I – 6 Evaluation Questions The evaluation design is driven by the evaluation questions. Following are presented the evaluations questions by component which relate to the underlying evaluation knowledge objectives listed: Overall 1. Has the YESS assistance contributed in a meaningful, if limited, way to the economic stabilization of Yemen? Component 1: International Trade Facilitated 1. To what extent has the YESS assistance directly resulted in improvements in customs and port operations? 2. Has the YESS collaborative partnership/dialogue with key GOY and other stakeholders identified gaps/weaknesses in current assistance which should be addressed in future programming? If so, what kinds of assistance are needed? Component 2: Agricultural Productivity Improved/Livelihoods Expanded 1. To what extent did YESS technical assistance/training result in agricultural productivity and market access improvements in key value chain sectors with a view to increasing farm household income? Component 3: Employment Generation/SME Competitiveness Improved 1. To what extent did YESS SME job generation assistance – job preparedness, job matching, vocational training, etc. – match SMEs needs with a view to supporting sustainable private sector job growth and business expansion? 2. To what extent did YESS business performance assistance to improve access to finance, introduce new organization technologies, improve management practices, etc., result in enhanced competitiveness of SMEs? DOCUMENTATION REVIEW Documentation review will be conducted as a first screen of identifying information required to address the evaluation questions. Each team member will be assigned documents from USAID, the IP and other independent sources and will review and summarize the findings. DATA COLLECTION, QUALITY AND LIMITATIONS For each data source provided by USAID, the IP and independent sources, reviewing team members will complete a form. This will summarize information related to the individual evaluation questions and any issues regarding data quality/completeness in order to ensure a full understanding of available and relevant quantitative data and qualitative information. This process will inform the design of additional data collection efforts that may be needed, i.e., surveys on post-assistance outreach to assisted firms. Particular attention will be given to prior Data Quality Assessments (DQAs) if available. The distinguishing features of quantitative and qualitative research will drive internal discussions of the usefulness, completeness and relevance of IP and other available data and potential new data to be developed in the evaluation. Qualitative data can help provide an understanding of the program’s theory of change, that is, the assumptions underlying the type and sequence of assistance activities meant to induce business and job growth. Qualitative information can provide insights into the context of a problem, generating ideas to test with available quantitative data. ANNEX I. Evaluation Statement of Work Annex I – 7 However, the quality of the analyses is contingent on the quality of available data. Limitations, such as survey beneficiary selection bias, recall bias in KIIs or FGDs, unobservable differences between comparator groups, will be noted. It is beyond the scope of this evaluation to conduct in-depth Data Quality Assessments. However, IP internal DQA and monitoring site visit reports as noted in the YESS AMELP will be reviewed for completeness and general accuracy. Since the submission of the 2018 Annual Report, it does not appear that the IP has yet conducted any baseline assessments to collect data against which to measure improvements as per a number of AMELP outcome indicators. The evaluation team will seek customs data and enterprise records to the extent such information is available and relevant to assess improvements in port and customs management, trade facilitation, SME business growth and job expansion, and agricultural productivity and market access, and will supplement the data assessment with qualitative perceptions of progress from beneficiaries and KIs. While it will not be possible to conduct an overall conflict sensitivity analysis of Yemen, the evaluation will necessarily identify critical security issues impacting the implementation of YESS activities. Surveys Surveys of beneficiaries will be conducted either on-site or by phone as the circumstances on the ground in Yemen allow. IBTCI will contract with a local firm who has relevant technical expertise and experience, logistical capacity and situational awareness to guide the final choices of how and where the surveys will be implemented. The evaluation team will collaborate with USAID for the final selection of sites to be visited. SURVEYS-KIIS-FGDS FIELDWORK PROTOCOLS The evaluation team included local staff and contractors will visit selected sites where YESS assistance had been provided. Field visits will be guided by the protocols below: 1. Sites to be identified in collaboration with USAID. 2. Where needed, notices to be sent by the evaluation team with an introduction from USAID. 3. Local survey partners and their enumerators will be trained. 4. Survey to businesses/employees disseminated (in person, by phone, and/or by mail/email as appropriate), with a sample supplemented by interviews during the field visits. 5. Each KI/FGD will be summarized in a report highlighting key issues. 6. Daily conferrals of team members will take place before/after each days’ activities. 7. Each team member will be required to prepare daily notes of key issues. 8. One member of each team will be responsible for follow-up and tracking down data that could supplement material derived from the interviews. Data collection will be conducted by a contracted data collection firm. The evaluation team will collect and analyze both qualitative and quantitative data in the cities of Sanaa, Aden and Al Mukalla, along with rural populations in Amran and Hadhramaut Governorates. A sampling frame will be developed based on YESS deliverables for each component, to be informed by conversations with USAID. The sampling frame will be included in an evaluation inception report ANNEX I. Evaluation Statement of Work Annex I – 8 Gender The evaluation will address gender issues and the robustness of the program’s attention to such issues in its documentation review, KI interviews, survey design and implementation, and its various analyses. Gender should not be a separate component of review, but rather an integrated cross-cutting theme addressed throughout the evaluation to the extent of available data. EVALUATION DELIVERABLES The deliverables for the evaluation are listed below. Deadlines are noted in Table 3 following. 1. An inception report including evaluation design matrix, analysis plan, and work plan; 2. A desk review report (to be included as an appendix in the draft evaluation report); 3. Draft and final evaluation reports; 4. A 5-page CLA summary of best practices and lessons learned; 5. A short, 2-3-page dissemination plan; 6. A PowerPoint presentation for stakeholders from site visits; 7. Follow-up recommendations tracking matrix. Dates Evaluation Activity Week 1 • Mobilization Brief with USAID to review/modify/approve evaluation plan • Finalization of methodology/schedule of activities • Complete Desk Review • Prepare/submit Inception Report to USAID Week 2-3 • Develop contact lists/interview guides for KIIs • Initial meeting with IP • Collect/ review IP and independent data sources • Finalize survey instruments Weeks 4-5 • Field interviews/focus group discussions in selected sites in Yemen Weeks 6-7 • Team reconvenes to analyze data and formulate findings and conclusion • Initial draft chapters relating to four key questions Weeks 8 • Draft Report prepared,submitted to USAID COR at start of week • Internal discussion of draft with USAID COR and key tech staff end of week Week 9 • USAID develops and submits comments to evaluation team Week 10 • Draft Report revised and submitted to USAID • USAID disseminates reportto IP and other stakeholders. Week 11 • Evaluation Team prepares/conducts presentation for all stakeholders • Final Report prepared/submitted to USAID Week 12 • Evaluation team prepares/submits all final deliverables including Presentation, Dissemination Plan and Recommendations Follow-Up Tracking Matrix • Evaluation Team has exit meeting with USAID ANNEX I. Evaluation Statement of Work Annex I – 9 OPERATIONAL CONSIDERATIONS Local Assistance Yemen consultancies will be hired to conduct surveys, KIIs and FGDs. The consultancies will be responsible for training enumerators, conducting surveys on-site and where feasible by phone, uploading data into SPSS, and providing a first level set of tabulations to the evaluation team/ Security IBTCI will ensure that appropriate security protocols are utilized for all field activities in Yemen. ANNEX II. CLA in YESS Annex II – 1 Annex II: YESS Utilization of CLA Principles This annex assesses the integration, implementation and effectiveness of CLA principles into YESS design and execution. The assessment is organized according to thematic areas:27 • Collaborating: Are we collaborating with the right partners at the right time to promote synergy over stove-piping? • Learning: Are we asking the most important questions and finding answers that are relevant to decision making? • Adapting: Are we using the information that we gather through collaboration and learning activities to make better decisions and adjustments as necessary? • Enabling Conditions: Are we working in an organizational environment that supports our collaborating, learning, and adapting efforts? These thematic are further broken down into principles for each area that are applicable to YESS and provide a structure to consider the assessment results. It is the opinion of the evaluation team that YESS was partially successful in CLA with the biggest barrier to better use of CLA being the design of YESS. The design of YESS created parameters that resulted in an opportunity cost for CLA. We say “opportunity cost” because while there was effective CLA practices within YESS, the design of YESS ensured that the full potential for CLA within YESS was not possible. However, the evaluation found that even with the constraints imposed by YESS design, the YESS implementation team performed exceptionally well across all components to maximize CLA and often had to develop creative practices to do so. COLLABORATION • CLA Principle: Link and coordinate associated mechanisms toward a common goal28 The evaluation report noted the lack of an effective definition around “economic stabilization” as a goal for YESS, but this lack of a common goal mitigated much of the coordination that possibly could have happened between the YESS components. Without a clearly understood common goal that requires coordinated action between components in order to achieve it, the YESS components were largely autonomous with limited coordination/cross-pollination. The most active area of coordination between components was related to the “feedback loop” approach that sought to take advantage of similar “access” to finance objectives. But the evaluation found limited evidence of this feedback loop approach leading to increases in access to finance. But the lack of a common goal is sometimes the result of a lack of a clearly articulated theory of change that outlines the causal pathway to the common goal with relevant assumptions. USAID has previously noted the harm that limited theories of change have on the ability to generate evidence for action. “When asked about barriers to evaluation quality and use, USAID staff members often cite instances of an evaluation taking place at the wrong time, focused on the wrong questions, or failed to engage stakeholders from the beginning. Similarly, external 27 USAID CLA Toolkit website: https://usaidlearninglab.org/node/14633 28 USAID, Collaborating, Learning and Adapting for Project Design. Retrieved from: https://usaidlearninglab.org/sites/default/files/resource/files/cla_in_pads.pdf ANNEX II. CLA in YESS Annex II – 2 evaluation teams can find themselves challenged to collect and analyze data while simultaneously reconstructing the theory of change and adapting to the evolving needs of stakeholders.”29 This guidance from USAID is very applicable to the YESS. YESS was managed in a way that produced rich data, the evaluation found that there is ample opportunity for learning that has not been fully taken advantage of. The YESS ToC is overly simplistic and does not accurately reflect the cause and effect relationships or external influences involved in YESS. As noted throughout the evaluation report, the lack of an articulated ToC for the separate components and relationships between the components themselves, limits the ability of the evaluation to produce more actionable evidence on more specific chains of causation. • CLA Principle: Engage local actors for consultation, feedback and collaborative action. The evaluation is unable to assess the level and manner in which local actors were consulted during the design of YESS due to a lack of documentation and overall data that explains the details of how YESS was designed. This lack of transparency is in and of itself an important evaluation finding. However, there is ample evidence on how YESS was implemented and the use of actions plans in the Trade and more specifically, the EG component of YESS demonstrate a high level of collaboration with local actors. These EG Action Plans outlined the SME specific objectives to improve competitiveness as well as their barriers. These Action Plans were use as monitoring tools to assess progress, update barriers, etc. In theory these Action Plans are effective and when it comes to YESS they did add value but not on a consistent basis due to the limited duration of YESS and the number of SME beneficiaries reached in that short timeframe. Because YESS was mandated, to work with such a high number of SMEs, in such a short period of time, there was limited levels of collaboration that could happen and there is good evidence that the resources allotted to collaboration may not have been sufficient. This is compounded by the fact that, because of the Action Plans for jointly developed in a collaborative manner, that “wish lists” were prominent in the initial phases of action plan development (meaning SMEs submitted “wish lists” during initial consultations). The YESS team took special care to ensure that the Action Plans were problem driven, meaning that the alleviation of problems that mitigated “competiveness” were the focus of the Action Plans and not possible “wish lists”. During data collection it became apparent that many (but not all) of these Action Plans were not modified after they were signed, meaning that as technical assistance was delivered the Action Plans remained the same. It is impossible to determine how much Action Plans needed to be updated to be “optimal” but given the time constraints and number of SMEs, it would be impossible to update a significant portion of the Action Plans after they were signed even in the face of new evidence that they needed to be updated with emerging problems, shifting circumstances, etc. LEARNING • CLA Principle: Utilize reflection and adaptive management processes to identify, respond to, and use new data and knowledge. Given that part of the purpose of the evaluation is to inform this end-use decision making, we sought to ensure that the evaluation was designed around these articulated end uses to include the 29 USAID, Conducting and Evaluability Assessment for USAID Evaluations. Bureau of Policy and Planning. 2017. ANNEX II. CLA in YESS Annex II – 3 specific timing and evidence requirements (i.e. what is considered to be “actionable” and “credible” evidence thresholds for specific decision-making). The evaluation spent considerable time searching the documentation and holding discussions but did not find any clearly articulated end-use identified, a point included in the YESS Evaluation Inception Report. During qualitative collection with USAID personnel some general end-uses were discussed to but not significantly detailed to mitigate the risk of evaluation findings not being utilized (resulting in a low return on the evaluation investment). These general end-uses focused on informing USAID programming in Yemen following YESS (phase 2 of YESS has already begun so it is assumed that this refers to some future programming that has yet to be determined). However, these end-uses are not specific, they are not articulated as a set of decisions, their timing, their information requirements nor what the successful result of these decisions would be. It could very well be that there is significant use of these evaluation findings, however there could have been improvements in identifying end-use attributes and integrating them into the program (especially given its status as a “pilot”) and evaluation design. The potential for learning was a central design feature of this evaluation of YESS and it is a focus throughout this report. The team involved in this evaluation not only have a professional interest in seeing these evaluation findings utilized but an ethical interest in seeing this evidence lead to social impacts. We make ourselves available at any time not only to USAID but to any actor who feels this evidence can lead to these results. • CLA Principle: Learning processes designed to support continued examination of the validity of the assumptions in the Logical Framework There were three assumptions listed in the YESS AMELP: 1. Political stability – that there is sufficient governance that Yemenis can make decisions with the expectation that things will not change significantly over the next 12 to 18 months; 2. Macro-economic stability – that economic indicator either remain stable or improve, again so that Yemeni firms and job seekers can make decisions with the expectation that the economy will not deteriorate significantly over the next 12 to 18 months; and 3. Security conditions remain the same or improve – in particular, that movement around the YESS operating environment does not affect the ability of the program staff to implement the program. These assumptions are very strategic and high risk given the operating context of Yemen at the present time. The risk associated in these assumptions became apparent during (and after) the implementation of YESS as many outcomes failed to materialize due to these assumptions. The evaluation found no explicit assumptions documented or referred to for the individual YESS components. Likewise, the evaluation found no evidence of any effort to gather evidence on relevant assumptions (i.e. assumption testing) or identify assumptions during implementation. The lack of adequate assumptions and the failure to assess them resulted in a loss of effectiveness for YESS. ADAPTING • CLA Principle: Test and explore assumptions and hypotheses throughout implementation and stay open to the possibility that your assumptions and hypotheses are not accurate Any pilot begs the question of “a pilot for what?”, meaning what is being tested in the pilot and how is that test being conducted. While not formally stated in YESS documentation, the USAID and YESS perceptions gathered in data collection tend to frame YESS as a pilot to test “whether” ANNEX II. CLA in YESS Annex II – 4 development programming is feasible in Yemen. Data on this “test” includes this evaluation and the YESS MEL framework (as outlined in the YESS MEL Plan) but both of these tools have limitations in answering this question largely due to the design of YESS. Meaning that if YESS was meant to explicitly test the ability to implement development programming in Yemen at this time, more specific hypothesis built around perceived barriers to development programming could have been articulated and then designed to. For example, if a conflict sensitivity analysis outlined possible influences of the lack of a central government on YESS targeted outcomes, then YESS activities and MEL could be designed to assess the level and type of influence stemming from the lack of a functioning government. This provides more tailored and actionable evidence for follow on decision making. Hence if YESS is a pilot, then specific hypothesis around what YESS is testing, and how, would add a lot of value to an appropriate MEL strategy to test the hypothesis. This is not to say that successful larger scale development programming in Yemen is not feasible, just that YESS (despite its successes) is possibly not a good pilot to determine this feasibility. The design of YESS does have several attributes of shorter-term humanitarian type programming as opposed to longer term development programming, which further complicates the function of YESS as a pilot for longer term development programming in Yemen. For example, the Employment Generation component of YESS did not target the cultivation of demand that would, in turn, drive job creation. Instead, it targeted job creation itself (albeit in areas where they felt there was sufficient demand to sustain the job creation). Demand-driven outcomes are more characteristic of development programming while “job creation” outcomes tend to be more characteristic of humanitarian programming. There is more on this later in the report when dealing with evaluation questions 3.1. and 3.2. Likewise, the fact that Yemen does not have a functioning central, and often times governorate level, government to act as stakeholders in development outcomes is an issue that cannot be overstated. This is not saying that all development, as opposed to humanitarian, outcomes are impossible in Yemen, just that those requiring active Yemeni government partnership will be much more difficult. The short timeline for YESS (Phase 1 was an 18-month period of performance), is more indicative of a pilot program but the shorter timeline is also common among humanitarian interventions. This is not saying that YESS is a humanitarian program, just that it has many characteristics of one, hence its limited ability to answer questions around feasibility of larger scale development programming in Yemen. ENABLING CONDITIONS • CLA Principle: Account for local systems YESS was a small program that in theory, given its scale, would not significantly contribute to any conflict driver, especially given its geographic distribution throughout the whole of Yemen. But there was no conflict sensitivity analysis conducted or referred to in any program documents that outlined what the conflict drivers were to ensure that YESS would not, even accidently, contribute to them. In the vast majority of interviews with Pragma staff, when questioned about the biggest barriers faced during the program, by far the most cited was related to effects of the conflict. The most prevalent of these conflicts induced barriers was the lack of a functioning government that could provide the necessary partnerships needed for optimal YESS outcomes. The Trade Facilitation (TF) component of YESS enjoyed the most active Yemen government participation but even this was far below what was needed for optimal outcomes since the TF component was the most dependent on active Yemeni government stakeholdership. ANNEX II. CLA in YESS Annex II – 5 Likewise results from data collection in Yemen among beneficiaries found widespread views that the conflict was a significant disrupter to results. For example, 42% of surveyed Livelihood beneficiaries observed that the conflict was a very significant obstacle to sustaining the gains from YESS. Likewise, Trade and EG beneficiaries noted that the conflict was the primary obstacle to achieving the outcomes targeted by YESS. The key lesson of conflict sensitivity is that organizations and their activities will become a part of the context in which they are operating; they will have an impact on the relationships among people in those contexts.30 A conflict sensitivity analysis could have articulated specific influences of the conflict that influenced YESS causal chains and provided YESS implementers and MEL (to include this evaluation) with better information to design, implement and assess the performance of YESS. If YESS was a pilot to help determine if larger scale development programming could be conducted in Yemen, the lack of a conflict sensitivity analysis, and resulting implementation adaptations, hamper the ability to answer to that question. 30 USAID. Conflict Sensitivity in Food Security Programming. 2017. ANNEX III. Data Collection Summary Annex III – 1 Annex III: Data Collection Summary This was a mixed method performance evaluation that consisted of document review, Key Informant Interviews (KII), Focus Group Discussions (FGDs), face-to-face surveys, and phone surveys. The following tables summarize the data collection design among the various data sources as correlated to YESS component. (Full data collection tables will be included in Final Evaluation Report) Table A1: Overview of Data Collection Beneficiary YESS Component KIIs Live Surveys Phone Surveys FGDs Trade 24 54 - 2 Livelihoods 10 - 120 - Economic Growth 18 418 - - Total 52 472 120 2 Table A2: Specific Data Collection from Pragma and USAID Key Informant Interviews Date Respondent August 19, 2019 William Baldridge, (USAID) August 26, 2019 Mayad Yafai, YESS Team Lead August 27, 2019 Mohammad Ilyas, Livelihoods Team Leader August 27, 2019 Heather McHugh (MEL Senior Advisor), August 28, 2019 Michael Wyzan (USAID/Yemen-General Development Officer Yemen Affairs Unit) August 28, 2019 Mohamed Ali (YESS MEL Manager) September 1, 2019 Farhat Farhat (TF Team Leader) September 3, 2019 Kevin Martin, USAID EG, Middle East Bureau September 6, 2019 Ines Allouche (A2F Team Leader) September 9, 2019 Moneef Sultan (A2F Field Team Leader) September 9, 2019 Boutheina Lassadi (EG/Job Matching Team Leader) Focus Group Discussions Date Respondents August 26, 2019 Heather McHugh (MEL Senior Advisor), Mohamed Ali (YESS MEL Manager) August 28, 2019 YESS MEL Team- Abdullah Baidhani (Sanaa), Ammar Moati (Aden), Mostafa Moqaibel (Al Mukalla), and Dana Hardan (Amman) September 1, 2019 Abdul WahabAl Awdi (Senior TF Expert), Roy Facey (Senior Transport Advisor), Farhat (TF Team Leader) ANNEX III. Data Collection Summary Annex III – 2 September 3, 2019 Abdulaziz Saeed Radman (Livelihoods- Sanaa), Mohsen Hubaishi (Livelihoods- Al Mukalla), Dhafer Shehab (Livelihoods- Aden) September 4, 2019 Slim Sayadi (Business Advisor) and Shadin Alhamaideh(Senior Business Advisor)π ANNEX IV. Changes to Evaluation Design Annex IV – 1 Annex IV: Changes to Evaluation Design Changes in Evaluation Design There were two changes of scope in the data collection after approval of the Evaluation Inception Report. These are abbreviated here in the order of change to provide transparency on the evolution of the scope changes. 1. On September 7th USAID approved a change in data collection on YESS Livelihoods activities. The Inception Report originally envisioned that there would be face to face KII’s with Demonstration Site Leaders and Farm Field School Lead Farmers as well as face to face surveys with up to 20 participants for each sampled Demonstration Site and Farmer Field School in Sanaa, Amran and Hadramout. However, given the distances between the farms, and difficulty in locating them, of the Demonstration Site and FFS participant (or clustered) farmers, these face to face surveys were deemed not feasible given security conditions. As a result, the face to face KIIs for Demonstration Site Leaders and FFS Lead Farmers were shifted to KIIs conducted over the phone and the face to face surveys of participating farmers were eliminated. Additionally, the original scope for Livelihood training participants in these three governorates was envisioned to be face to face surveys but these were also shifted to phone surveys due to the same security concerns. It was anticipated, based on available evidence from similar phone￾based surveys of rural respondents, that the response rate would be very low and the quality of responses diminished. However these risks were mitigated, as best they could, by the availability of phone numbers for a large proportion of all trainees which allowed for a high number of reserve respondents who could be contacted in the case of non or incomplete response from the original sampling. 2. On September 17th USAID approved changes that eliminated all data collection outside of Aden and Hadramout. As a result, there were no face to face surveys with Demonstration Site Leaders, Farmer Field School Lead Farmers and Livelihood training participants. There was no data collection on any Livelihood beneficiaries outside of Aden and Hadramout. There were no data collection activities for the Employment Generation component of YESS in all areas outside of Aden and Hadramout. The method of sampling for Aden and Hadramout was changed to include a census of all beneficiary SMEs and a census of their new employees. Instead of face to face surveys with Livelihood training beneficiaries, we performed phone surveys livelihood training beneficiaries in Hadramout (there were no training beneficiaries in Aden). We performed KIIs for the leadership of all SMEs, and face to face surveys for all new employees, in Aden and Hadramout who reported data for YESS indicator 3a and 3b. ANNEX IV. Changes to Evaluation Design Annex IV – 2 Effects of Evaluation Scope Change on Beneficiary Coverage Governorate 31 YESS Component and Beneficiary Numbers Initial Evaluation Design Coverage Final Evaluation Design Coverage Final Data Collection overage Sanaa Trade: NA Trade: NA Trade: NA Trade: NA EG SMEs: 53 EG SMEs: 7 EG SMEs: 0 EG SMEs: 0 EG Employees: 2,130 EG Employees: 70 EG Employees: 0 EG Employees: 0 Livelihoods: 110 Livelihoods:20 Livelihoods:0 Livelihoods:0 Finance: 4 Finance: 4 Finance: 0 Finance: 0 Amran Trade: NA Trade: NA Trade: NA Trade: NA EG: NA EG: NA EG: NA EG: NA Livelihoods: 303 Livelihoods:45 Livelihoods:0 Livelihoods:0 Aden Trade: 238 Trade: 238 Trade: 238 Trade: EG SME: 18 EG SME: 7 EG SME: 18 EG SME: EG Employees: 656 EG Employees: 70 EG Employees: 656 EG Employees: Livelihoods: NA Livelihoods: NA Livelihoods: NA Livelihoods: NA Hadramout Trade: NA Trade: NA Trade: NA Trade: NA EG SME: 12 EG SME: 10 EG SME: 12 EG: EG Employees: 255 EG Employees: 100 EG Employees: 255 Livelihoods: 677 Livelihoods: 50 Livelihoods:659 Livelihoods: 31 For the purposes of this report we are only included governorates that were targeted for data collection in the initial evaluation deign as captured in the approved Inception Report. ANNEX V. Inception Report Annex V – 1 Annex V: Inception Report (Full Evaluation Design and Data Collection Instruments) INCEPTION REPORT PERFORMANCE EVALUATION OF YEMEN ECONOMIC STABILIZATION AND SUCCESS (YESS) AUGUST 22, 2019 CONTRACT NO. AID-279-C-15-00001 Submitted August 22, 2019 Prepared for Jessica Pomerantz, YMEP II Contracting Officer’s Representative General Development Officer/Yemen United States Agency for International Development USAID/Middle East Regional Platform Prepared by International Business & Technical Consultants, Inc. (IBTCI) 8618 Westwood Center Drive, Suite 400 Vienna, VA 22182 USA Tel: (703) 749-0100 DISCLAIMER The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ANNEX V. Inception Report Annex V – 2 INTRODUCTION This Inception Report outlines the evaluation design for the performance evaluation of the USAID Yemen Economic Stabilization and Success (YESS) program. It will provide stakeholders with a common understanding of the design details, limitations and purpose of this evaluation. It will be shared with Pragma and any other stakeholders deemed appropriate by USAID for review, feedback and formal approval or statement of difference if needed. This Inception Report includes all evaluation questions, evaluability criteria, intended sampling, instruments and appropriate data management and quality control protocols. Limitations and Intended Use This performance evaluation has three primary limitations, 1.) Ability to reach intended beneficiaries for adequate representative sampling/data collection (due to security factors), 2.) Lack of a clearly defined end-use of evaluation results with correlated decision-making requirements that could be used to inform the evaluation design and 3.) Lack of a reliable counterfactual. As a result, this evaluation is primarily meant to perform accountability functions on fidelity of YESS programming and generate evidence around YESS’ performance against stated outcomes. OVERVIEW OF THE PROGRAM AND THE INTERVENTIONS EVALUATED The Yemen Economic Stabilization and Success (YESS) Program is part of USAID’s Middle East Economic Growth (MEG) and is meant to provide information, data, best practices and learning to USAID decision makers in order to enhance economic growth outcomes in Yemen. YESS is a collaborative pilot project to test approaches “to promote economic stabilization in the country, and then to identify what works or does not work, and then to leverage best practices or lessons learned to scale up and expand the scope.”32 YESS is meant to be highly adaptive and responsive to the large-scale conflict in Yemen with three primary activities, Trade Facilitation, Employment Generation and Agricultural Livelihood Support. These activities are meant to bring about three main intermediate outcomes: (1) productivity in selected value chains increased; (2) employment opportunities expanded; and (3) integration of private sector into international trade and finance networks improved. YESS was originally designed to be implemented in an 18-month timeframe, from February 2018 to August 2019 – adhering to USAID transitional operating framework. The YESS Employment Generation component shut down at the end of June, 2019. Parts of that program – particularly market linkages and financing for farmers – has been integrated into the Livelihoods component. Phase II of the YESS program will continue until the end of March 2020. YESS COMPONENTS The three primary activities of YESS are summarized here in order to build an understanding of the YESS Theory of Change (ToC). Trade Facilitation: The main objective of this component is to facilitate enhanced access to key ports, air facilities and trade transit corridors that allow for efficient shipping of humanitarian supplies into the country. It also focuses on improvements in the trade facilitation policy/regulatory/institutional framework required to significantly enhance the time- and cost-efficiency of trading activities in Yemen.33 32 YESS AMELP 33 YESS Trade Facilitation Work Plan ANNEX V. Inception Report Annex V – 3 The Trade Facilitation component is comprised of various analytical and technical assistance sub￾activities meant to address key trade policy/institutional barriers hampering humanitarian assistance inflows to the population, and limiting the integration of Yemen’s private sector into international trade and finance networks. The primary goals are to 1) facilitate commercial trade for enabling growth; 2) improve the availability of fuel at competitive prices, and 3) facilitate the flow humanitarian aid. The Trade Facilitation ToC includes several intermediate outcomes: • Reduce trading costs and delays caused by external factors • Undertake a trade facilitation pilot activity at the Port of Aden • Based on prior mapping analysis, develop and initiate implementation of detailed recommendations for streamlining import/export procedures at key ports of entry • Build customs capacity in key areas • Launch the implementation of the WTO Trade Facilitation Agreement • Support the modernization of the Customs Legal Framework • Improve access to fuel at competitive prices Agricultural Livelihoods: Small farmers have been particularly hard-hit by the current conflict, having lost access to traditional markets, agriculture services, and essential production inputs. Much of the agriculture infrastructure in the country was severely damaged, with no measurable new public or private investment. It is estimated that locally produced food supplies dropped by 62% of pre-conflict levels due to crop losses, and agriculture sector employment by 50%.34 YESS aims to increase small farmer productivity, efficiency, and resilience to environmental and market risks by improving the quality and competitiveness of high-potential value chains that will expand and diversify production. It also seeks to strengthen linkages and build effective partnerships with producers, processors, retailers, and exporters to help stimulate the sector and expand access to services and markets. To sustainably improve the livelihoods of 3,000 small producer families, the MEG/YESS Livelihoods Team targets key constraints impeding growth in the agriculture sector to increase small farmer production capacity. These include: 1) Climate Smart Agricultural (CSA) production and resilience (to conserve biodiversity, while enhancing agricultural production); 2) improving competitiveness (through supply chain efficiencies and strengthened market linkages); and 3) increasing competitive small producer access to finance. These targeted interventions have the potential for increasing small producer incomes and generating inclusive employment through scaling up and effectively linking production to internal and regional demand channels. Cross-cutting linkages to the MEG/YESS Employment Generation support and regional trade facilitation components will further support and promote the sustainability of these interventions. The YESS Livelihoods ToC comprised of three outcomes with correlated objectives: Outcome #1: Improved CSA Productivity in Selected Value Chains • Value chain actors organized around a cluster model/platform. • Technical Assistance Needs Assessment (TANA) 34 YESS Agriculture Livelihoods Work Plan ANNEX V. Inception Report Annex V – 4 • Develop Technical Assistance Game-plan • Farmers Field Schools (FFS) • Demonstration plots • Capacity building for private and public agriculture service providers Outcome #2 - Increased Supply Chain Competitiveness • Capacity building for farmers on post-harvest and processing best practices • Training for targeted processors on ‘green’ processing methods/practices • Contract farming between value chains actors • Training for targeted processors on product development and certification • Capacity building for private certification bodies • Establish a national platform for each targeted value chain • Develop export marketing linkages with specialty markets Outcome#3- Increased Small Producer Access to Financing • Link actors to MFIs and banks • Support MFIs and banks to develop new climate-smart financing products • Women’s Empowerment in Agriculture Employment Generation: The primary objective of this activity is to promote viable job opportunities by increasing the completeness of Small Medium Enterprises (SMEs) that have the potential to sustainably increase sales in internal and (in some cases) export markets that stimulate job growth. This was supported by partnership agreements with high-potential firms that are designed to identify, prioritize, and effectively address the key business development constraints which are preventing them from expanding sales and stimulating job creation. This approach is implemented across three strategic pillars: 1) Employment generation—to promote growth and improving SME competitiveness by increasing SME market penetration; 2) Workforce development with a specific focus on vulnerable youth (especially those at risk of being recruited as combatants) and women; and 3) Access to financing in the form of institutional support to formal and informal financial providers to help them better understand SME credit needs. The YESS Employment Generation component utilizes a feedback loop approach that allows for the integration of different activity components; including technical support, training/workforce development, institutional capacity building for financial service providers, and intensive support to leading value chains. The graphic below from the YESS Employment Work Plan illustrates. ANNEX V. Inception Report Annex V – 5 YESS PROGRAM THEORY OF CHANGE As stated in the YESS stab, the objective of MEG/YESS is to increase economic stabilization in Yemen, by achieving three primary outcomes: (1) productivity in selected value chains increased; (2) employment opportunities expanded; and (3) integration of private sector into international trade and finance networks improved. The YESS AMELP outlines three underlying assumptions that are critical to YESS success and thus an important factor in any evaluation: • Political stability – that there is sufficient governance that Yemenis can make decisions with the expectation that things will not change significantly over the next 12 to 18 months; • Macro-economic stability – that economic indicators either remain stable or improve, so that Yemeni firms and job seekers can make decisions with the expectation that the economy will not deteriorate significantly over the next 12 to 18 months; and • Security conditions remain the same or improve – in particular, that movement around the YESS operating environment does not affect the ability of the program staff to implement the activity. LITERATURE REVIEW A literature review was conducted to inform this evaluation design. The YESS ToC, including assumptions, was used to identify relevant evidence. The literature review helped identify numerous criteria for each YESS component, as well as conflict sensitivity issues, that are included in data collection and the analysis plan design for the evaluation. The full literature review is included in Annex 20. EVALUATION DESIGN This evaluation is primarily meant to perform accountability functions on fidelity of YESS programming and generate evidence around YESS performance against stated outcomes. ANNEX V. Inception Report Annex V – 6 The primary audience of the evaluation is USAID/Yemen and USAID/MEG. Other primary intended users include the implementing partner (IP) of the YESS activity covered by this study (Pragma), and other USAID missions with existing and planned programming with similar attributes or in similar contexts. Overview This performance evaluation seeks to: 1. Inform USAID and stakeholder decision making 2. Contribute to learning on similar theories of change in similar contexts 3. Provide a transparent and well-documented measure of accountability These objectives will be met by producing evidence that can be leveraged into new programming. We propose to accomplish this by applying quantitative (surveys), qualitative (Key Informant Interviews, Focus Group Discussions) and document review methods in an integrative and triangulated fashion. The “iterative” comes from our attention to identifying issues that emerge during data collection and analysis for possible follow during the course of the evaluation if possible or for subsequent follow-on if not. EVALUATION QUESTIONS The questions for this evaluation are organized around YESS outcomes of increased economic stabilization, integration of Yemen’s private sector into international trade and finance networks, productivity of smallholder farmers, and viable employment for job seekers. The evaluation questions are included in the below table. They are correlated with relevant data sources and proposed analysis, which will be further explained in the sampling and methods sections below. Table 1: Evaluation Questions Evaluation questions Data Sources Proposed Analysis Overall QUESTION 1: In what ways has YESS assistance contributed (or not) in a meaningful, if limited, way to the economic stabilization of Yemen? KIIs, FGDs Qualitative information derived from KIIs will be assessed for level of effectiveness of YESS activities in achieving the overall MEG goal. Component 1: International Trade Facilitated QUESTION 1.1: To what extent has the YESS assistance directly resulted in improvements in customs and port operations to include WTO trade agreement provisions and YESS performance indicators? GOY Port management data to the extent available and verified, KIIs, FGDs, YESS data will be analyzed in context with relevant available GOY port/trade data. Information will be assessed for level of effectiveness of YESS assistance. QUESTION 1.2: Has the YESS collaborative partnership/dialogue with key GOY and other stakeholders KIIs, FGDs, Assessment of perceived gaps as to how and to what extent additional types of assistance ANNEX V. Inception Report Annex V – 7 Evaluation questions Data Sources Proposed Analysis identified gaps/weaknesses in current assistance which should be addressed in future programming? If so, what kinds of assistance are needed? could be effective in the challenging context of the conflict in Yemen. Recommendations for future programming Component 2: Agricultural Productivity Improved/Livelihoods Expanded QUESTION 2.1: To what extent did YESS technical assistance/training achieve its relevant indicators and result in increased agricultural productivity and market access improvements in key value chain sectors.? Surveys, KIIs Perceived increases against evidence in the field of increased production and/or sales to the extent relevant data is available. Component 3: Employment Generation/SME Competitiveness Improved QUESTION 3.1: To what extent did YESS SME job generation assistance achieve its goal and relevant indicators of improving business performance and self-sufficiency for relevant firms? SMEs documentation, GOY sector data for comparison, Surveys/KIIs/FGDs Triangulation across available data points to identify growth and expansion attributable to YESS to the extent that relevant data is obtainable/generated. QUESTION 3.2: To what extent did YESS business performance assistance to improve access to finance, improve or introduce new management practices, etc., result in enhanced competitiveness of SMEs? SME revenue/sales data, KIIs/FGDs, Financial institution data, sector data if available Assess relative levels of SME financing for YESS assisted SMEs and general level of sector SME growth/performance. METHODOLOGY This is a mixed method performance evaluation and is comprised of five primary methods of collection and assessment: • Evaluability assessment of YESS Theory of Change and program design and relevant evidence from systematic and literature reviews • Document review (cross-cutting) • WhatsApp and face-to-face surveys (primarily for recipients of training under the various activities/sub-activities) • KIIs (conducted with beneficiaries of all three activities and YESS activity teams) ANNEX V. Inception Report Annex V – 8 • FGDs (YESS activity teams and trade, livelihood activity beneficiaries) Evaluability Assessment Evaluability Assessments are an increasingly utilized tool, the value of which has been noted by USAID. 35. These assessments have been performed by internal USAID staff in some instances and by third parties to prepare for, or as part of, an evaluation. For YESS, an evaluability assessment will add value in unpacking YESS into various dimensions (i.e. problem analysis, beneficiary identification, assumptions, etc.) to help provide more detailed, and thus actionable, evidence for USAID decision making and to better understand the results of the other methods of data collection/analysis. The criteria for the evaluability assessment to be completed on YESS is included in Appendix #21. Study Sample We are proposing a multilevel sampling approach where participants from different populations are used to evaluate the components of YESS. This fits well with our timing for data collection where we will be concurrently collecting data from different populations, but keep open the possibility for additional data collection from new sources as issues emerge. Where In the table below, the first column outlines what YESS components/sub-components are being sampled. Column two gives the sample frame (the source material from which we will identify potential respondents). Column three outlines how we will collect data from the respondents, the sample size (the number of times we will administer that form of data collection). The “Sampling Row” for each sub-activity indicates how specific respondent will be identified. The last column outlines where the relevant data collection instruments are in the Appendix to this Inception Report. All data collection instruments will be finalized before data collection begins and enumerators will be trained on collection and observed during piloting for each specific sub-activity to ensure the correct modules are used. Table 2: Sampling Plan # YESS Components/ Sub￾Components Data Source Method/ Number of Collection Instrument Reference Sampling Strategy and Location Data Collector 1 Overall YESS Implementation /Management YESS COP and DCOP (Pragma) YESS Leadership: Two KII YESS Leadership KII: Appendix 1 Full team, Amman IBTCI Team Lead in Amman and Washington D.C. 2 Overall YESS Monitoring and Evaluation YESS M&E Team (Pragma) YESS M&E Team: One FGD YESS M&E FGD: Appendix 2 Full Team, Amman IBTCI Team Lead in Amman and Washington D.C. Component #1: Trade Facilitation 35 Conducting an Evaluability Assessment for USAID Evaluations, USAID (PPL), June 2017. ANNEX V. Inception Report Annex V – 9 # YESS Components/ Sub￾Components Data Source Method/ Number of Collection Instrument Reference Sampling Strategy and Location Data Collector 3 YESS Trade Team (Pragma) YESS Trade Team: One FGD with entire team with possible follow up KIIs YESS Trade Team FGD: Appendix 3 YESS Trade Team (Pragma) The FGD will be with the entire YESS Trade Facilitation team while the KIIs will be with team leadership IBTCI Team Lead in Amman and Washingto n D.C. 4 Authorized Economic Operators (AEO) Relevant YCA, TTWG, Customs Officer personnel, AEO Trainees: WhatsApp Survey for all trainees YCA (pilot participants): 3 KII TTWG: 2 KII Customs Officer: 2 KII Trade Facilitation Survey: Appendix 4 Trade Facilitation KII: Appendix 5 WhatsApp survey of all training attendees, random sampling from trainee sheets for TTWG and YCA pilot participants and Customs Officers KIIs IBTCI data collection team (the Yemen Polling Center) in Yemen 5 ASYCUDA All trainees WhatsApp survey of all trainees Trade Facilitation Survey: Appendix 4 WhatsApp surveys will be sent to all trainees with known emails IBTCI data collection team (the Yemen Polling Center) in Yemen 6 Risk Management: Risk-based Food Control DAHVQ and SMQCO trainees. DAHVQ trainees: 2 KIIs, WhatsApp survey SMQCO trainees: 2 KII, WhatsApp survey Trade Facilitation Survey: Appendix 4 Trade Facilitation KII: Appendix 5 WhatsApp surveys will be sent to all trainees with known emails IBTCI data collection team (the Yemen Polling Center) in Yemen 7 Trade Facilitation: Public-Private Dialog Public-Private dialog attendees and relevant Ministry of Industry and Trade, Customs, and Customs Officials: 2 KII YGPAC Officials: 2 KII Private Sector dialog Attendees: 1 FGD Trade Facilitation KII: Appendix 5 Trade Facilitation Beneficiary KII’s will be randomly selected from Pragma lists of relevant officials and FGD will be comprised of representative sample of private IBTCI data collection team (the Yemen Polling Center) in Yemen ANNEX V. Inception Report Annex V – 10 # YESS Components/ Sub￾Components Data Source Method/ Number of Collection Instrument Reference Sampling Strategy and Location Data Collector YGPAC officials FGD: Appendix 6 sector organizations 8 Port of Aden Operations: Capacity Building Relevant YGAPC personnel who attended trainings and recipients of day-to-day technical assistance Relevant YGAPC personnel: 5 KII and FGD Trade Facilitation KII: Appendix 5 Random selection for KIIs IBTCI data collection team (the Yemen Polling Center) in Yemen 9 Customs Valuation: Customs officials from ACT, Al￾Ma’alla, Aden Airport, Al￾Buraiqa Oil Port, and Dar Saad who attended one or more of the three trainings. YCA Officers at Port of Aden in charge of Customs Valuation. YCA Officers at Port of Aden: 3 KIIs Trained Customs Officials: WhatsApp survey of all trainees Trade Facilitation KII: Appendix 5 Trade Facilitation Survey: Appendix 4 Random selection from training lists for YCA Officer KIIs and all Customs Officials with known emails will be contacted for the survey IBTCI data collection team (the Yemen Polling Center) in Yemen 10 WTO Facilitation Agreement: Conformity Assessment Roundtables Yemeni Customs and Ministry of Industry and Trade Officials who took part in roundtables and technical assistance Yemeni Officials from Customs and Ministries of Industry, Trade: Two KII’s Trade Facilitation KII: Appendix 5 KIIs will be with highest level official from the Customs and Ministry of Industry available who was actively part of the roundtable IBTCI data collection team (the Yemen Polling Center) in Yemen Component #2: Agricultural Livelihoods Support36 36 The data collection for the YESS Livelihoods Component does not cover all value chains and all Livelihood sub￾components due to limited resources and security concerns. Appendix XX contains all the Livelihoods sub- ANNEX V. Inception Report Annex V – 11 # YESS Components/ Sub￾Components Data Source Method/ Number of Collection Instrument Reference Sampling Strategy and Location Data Collector 11 Overall Livelihoods Activity YESS Livelihoods Team YESS Livelihoods Team- 1 FGD with entire team and 2 KII with team leadership YESS Livelihoods Team FGD: Appendix 8 YESS Livelihoods Team KIII: Appendix 7 The FGD will be with the entire YESS Livelihoods team while the KIIs will be with team leadership IBTCI Team Lead in Amman and Washingto n D.C. 12 Amran (Honey and Potato Value Chain Training, Bi￾Ridge Potato Plantation Demo Site/FFS and Potato Processing FFS) Honey Value Chain Trainees, Clustered Farmers at Demo Sites and FFS in Amran Face to Face Survey with Honey and Potato Value Chain trainees (up to 20) and Demo Site (up to 10) and FFS (up to 10) participants. Survey for Demonstration Site and Farmer Field Site leaders/lead farmers Livelihoods Survey (Honey and Potato training beneficiaries): Appendix 9 Demonstration Site/Farmer Field School Leader KII: Appendix 16 Demonstration Site/Farmer Field School Survey: Appendix 17 With references from the Site leader (facilitated by Pragma), surveys conducted for up to 10 clustered farmers. The full sampling protocol is described in Annex 18. Amran Honey and Potato Value Chain Trainees will be randomly selected from trainee lists provided by Pragma where each trainee is randomly assigned an ID# which are then randomly selected for up to 20 respondents IBTCI data collection team (the Yemen Polling Center) in Yemen 13 Hadramout (Honey and Livestock Trainees, Goat Fattening/Herd Honey and Livestock Value Chain Trainees, Clustered Farmers at Face to Face Survey with Honey Value Chain trainees (up to 20) and Demo Livelihoods Survey (Honey, Livestock Value Chain With references from the respective Site leader (facilitated by Pragma) IBTCI data collection team (the Yemen Polling components covered by this data collection. The data collection design attempts to maximize coverage of value chains, governorates and number of beneficiaries given these constraints ANNEX V. Inception Report Annex V – 12 # YESS Components/ Sub￾Components Data Source Method/ Number of Collection Instrument Reference Sampling Strategy and Location Data Collector Management Demo Sites and FFS) Demo Sites and FFS in Hadramout Site (up to 10) and FFS (up to 10) participants. Trainees): Appendix 9 Demonstration Site/Farmer Field School Leader KII: Appendix 16 Demonstration Site/Farmer Field School Survey: Appendix 17 surveys for up to 10 clustered farmers for the Tariba and Boor Demo Sites in Savorun, the FFS in Almagad, Qval and the FFS and Demo Site in Almagad, Qval. The sampling protocol for Demonstration Sites and Farmer Field Schools is described in Annex 18. Hadramout Honey and Livestock Value Chain Trainees will be randomly selected from trainee lists provided by Pragma where each trainee is randomly assigned an ID# which are then randomly selected for up to 20 respondents. Center) in Yemen 14 Sanaa (Coffee Value Chain Training) Coffee Value Chain Trainees in Bani, Matar, Sanaa (trained on harvest and post-harvest practices) Up to 20 face to face surveys Livelihoods Survey Coffee Value Chain trainees): Appendix 9 Coffee Value Chain Trainees in Sanaa will be assigned a random ID# which are then randomly selected for a survey. IBTCI data collection team (the Yemen Polling Center) in Yemen ANNEX V. Inception Report Annex V – 13 # YESS Components/ Sub￾Components Data Source Method/ Number of Collection Instrument Reference Sampling Strategy and Location Data Collector Component #3: Employment Generation 15 YESS Employment Generation (overall) YESS Employment Team (Pragma) YESS Employment Team- One FGD with entire team and two KII with leadership YESS Employment Team FGD: Appendix 11 YESS Employment Team KII: Appendix 10 FGD will be with entire Pragma team while KIIs will be with leadership IBTCI Team Lead in Amman and Washingto n D.C. 16 YESS Employment Generation (Sanaa) Job Creation SME’s and new employees (as selected from employment documentation ) Job Creation SME Firms: Up to seven KIIs per governorate Job Creation SME new employees: face to face survey for up to 10 employees per SME YESS Employment Generation SME (Job Creation) KII: Appendix 12 YESS Employment Generation: Job Creation Survey: Appendix 20 The evaluation will select the top two leadership of each SME for a KII or randomly select and randomly select up to 10 new employees from each SME for the face to face survey. Full sampling procedures for the EG component of YESS are found in Appendix 23 IBTCI data collection team (the Yemen Polling Center) in Yemen 17 YESS Employment Generation (Hadramout) Job Creation SME’s and new employees (as selected from employment documentation ) Job Creation SME Firms: Up to ten KIIs per governorate Job Creation SME new employees: face to face survey for up to 10 employees per SME YESS Employment Generation SME (Job Matching) KII: Appendix 13 YESS Employment Generation: Job Creation Survey: Appendix 20 The evaluation will select the top two leadership of each SME for a KII or randomly select and randomly select up to 10 new employees from each SME for the face to face survey. Full sampling procedures for the EG IBTCI data collection team (the Yemen Polling Center) in Yemen ANNEX V. Inception Report Annex V – 14 # YESS Components/ Sub￾Components Data Source Method/ Number of Collection Instrument Reference Sampling Strategy and Location Data Collector component of YESS are found in Appendix 23 18 YESS Employment Generation (Aden) Job Creation SME’s and new employees (as selected from employment documentation ) Job Creation SME Firms: Up to seven KIIs per governorate Job Creation SME new employees: face to face survey for up to 10 employees per SME YESS Employment Generation SME (Job Creation) KII: Appendix 12 YESS Employment Generation: Job Creation Survey: Appendix 20 The evaluation will select the top two leadership of each SME for a KII or randomly select and randomly select up to 10 new employees from each SME for the face to face survey. Full sampling procedures for the EG component of YESS are found in Appendix 23 IBTCI data collection team (the Yemen Polling Center) in Yemen 19 YESS Employment Generation: Job Matching (Workforce Development) Aden Job Matching (Workforce Development) Partners and new employees (as selected from employment documentation ) Job Matching (Workforce Development Partners): 2 KIIs per partner Job Creation SME new employees: face to face survey for up to 10 employees per SME YESS Employment Generation Workforce Development KII: Appendix 14 YESS Employment Generation: Job Creation Survey: Appendix 20 The evaluation will select the top two leadership of each SME for a KII or randomly select and randomly select up to 10 new employees from each SME for the face to face survey. Full sampling procedures for the EG component of YESS are found in Appendix 23 IBTCI data collection team (the Yemen Polling Center) in Yemen 20 YESS Employment Generation: Job Job Matching (Workforce Development) Job Matching (Workforce Development YESS Employment Generation The evaluation will select the top two leadership of IBTCI data collection team (the ANNEX V. Inception Report Annex V – 15 # YESS Components/ Sub￾Components Data Source Method/ Number of Collection Instrument Reference Sampling Strategy and Location Data Collector Matching (Workforce Development) Sanaa Partners and new employees (as selected from employment documentation ) Partners): 2 KIIs per partner Job Creation SME new employees: face to face survey for up to 10 employees per SME Workforce Development KII: Appendix 14 YESS Employment Generation: Job Creation Survey: Appendix 20 each SME for a KII or randomly select and randomly select up to 10 new employees from each SME for the face to face survey. Full sampling procedures for the EG component of YESS are found in Appendix 23 Yemen Polling Center) in Yemen 21 YESS Employment Generation: Job Matching (Workforce Development) Hadramout Job Matching (Workforce Development) Partners and new employees (as selected from employment documentation ) Job Matching (Workforce Development Partners): 2 KIIs per partner Job Creation SME new employees: face to face survey for up to 10 employees per SME YESS Employment Generation Workforce Development KII: Appendix 14 YESS Employment Generation: Job Creation Survey: Appendix 20 The evaluation will select the top two leadership of each SME for a KII or randomly select and randomly select up to 10 new employees from each SME for the face to face survey. Full sampling procedures for the EG component of YESS are found in Appendix 23 IBTCI data collection team (the Yemen Polling Center) in Yemen 22 YESS Employment Generation: Access to Finance (Sanaa) Financial Providers (Al￾Kuraimi MF Bank (KIMB), Azal MF Program, Yemen Kuwait Bank (YKB)) Financial Providers: 2 KIIs per provider (3 providers) YESS Employment Generation Financial Provider KII: Appendix 15 KII’s for each partner will be with the two most senior leadership personnel directly involved in the access to finance IBTCI data collection team (the Yemen Polling Center) in Yemen ANNEX V. Inception Report Annex V – 16 # YESS Components/ Sub￾Components Data Source Method/ Number of Collection Instrument Reference Sampling Strategy and Location Data Collector activities (with Pragma input). 23 YESS Employment Generation: Micro-Finance Cluster SME Micro Finance Cluster: Tadhmoun Microfinance (Sanaa) Tawasoul Microfinance Program (Sanaa) National Microfinance Foundation (Sanaa) Hadhramout Microfinance Program (Hadramout) Loan Guarantee Program (Sanaa) Nama Microfinance Program (Sanaa) Cluster Participants: 2 KII’s per participant TBD TBD IBTCI data collection team (the Yemen Polling Center) in Yemen Table 3: Data Collection Geographic Breakdown (Governates with an * have been designated as high security risk areas and consultation is needed on the security situation in all governorates but especially those identified) Location Relevant Activity/Sub￾Activity Data Collection (references to rows in sample table) Amman All activities 1,2,3,12,15 ANNEX V. Inception Report Annex V – 17 Aden Trade facilitation, Employment Generation 4,5,6,7,8,9,10,11,18,19 Sanaa* Livelihoods (Coffee Value Chain Trainees, Demonstration and Farmer Field Schools), Employment Generation 14,16, 20, 22, 23 Amran* Livelihoods (Honey and Potato Value Chain Trainees, Demonstration and Farmer Field Schools) 12 Hadramout Trade Facilitation (Mukallah), Livelihoods (Honey and Livestock Trainees, Demonstration and Farmer Field Schools), Employment Generation (Mukallah) 13,17,21, 23 DATA COLLECTION, MANAGEMENT, QUALITY CONTROL AND ANALYSIS Most of the data collection taking place in Yemen will have technical oversight from Amman, Jordan. The IBTCI team will not be entering Yemen due to the ongoing security situation. As a result, we have developed extensive quality control protocols for all methods of data collection, including instrument development, enumerator training, quality control checks, data management and analysis. Quality control protocols will mitigate the risk involved in managing an evaluation from a distance. FIELD STAFF COMPOSITION The Yemen based data collection field staff will include a Project Manager (PM) who will coordinate and manage all data collection activities. The PM will supervise a Data Manager (DM) and data collection field teams. Field teams will consist of Provincial Field Managers (PFMs) and approximately 4 enumerators per Governorate or city. Data collection in each Governorate or city will be assisted by two Systems Programmers / Data Entry Clerks, an Administrative Assistant, along with Moderators and Note Takers for KIIs and FGDs. Data collection in Jordan and the United States will be completed by IBTCI. FIELD OVERSIGHT All data collection will include sufficient field oversight for all field staff to ensure high quality data collection and adherence to study protocols. The subcontracted firm will include a detailed plan for field oversight as part of the quality assurance plan. This includes: • Provincial Field Managers (PFMs) PFMs will monitor enumerators and ensure adherence to all data collection protocols (e.g. obtaining informed consent, completion of all survey modules, and testing, and adherence to the prescribed random or systematic sampling strategy) • PFMs are responsible for ensuring that follow-up visits to households or businesses where errors were found are completed • Regularly error checks of completed surveys if paper-based data collection methods are used ANNEX V. Inception Report Annex V – 18 • Enumerators and PMM maintenance of hard-copy logs to document household and business visits, interviews, and any issues encountered during fieldwork • PFM management of consent forms and identification of any consent form errors • Daily debriefings between DM, PFMs and enumerators to address any concerns or suggestions • Promptly reporting of issues or challenges that arise during data collection to the evaluation team • Weekly meetings (at minimum) between IBTCI, the PM and PFMs to monitor progress, discuss challenges, and address issues noted in the field ELECTRONIC DATA COLLECTION We intend on using electronic data collection and will include range and logic checks, automatic skip patterns, and other data quality aids according to the specifications in each survey instrument. All sampling units will be assigned a unique ID suitable for linking purposes. It is essential that the proposed platform have the capability to populate a table of respondents at the beginning of the interview, and for each subsequent module question that references a specific household member, ensure accurate linking between the household member-specific question and that household member’s record in the table of household members. Manual linking or hand coding of IDs for the household members’ records in each module shall be considered unacceptable. Data Entry Plan (for paper-based data collection methods) For all paper-based data collection methods to be used (as back up in case of electronic failure), we will provide a comprehensive plan for data entry and the minimization of errors that arise from the process. It will include plans for training data entry staff; double data entry of paper forms or optical scanning and a strategy to resolve any data inconsistencies found therein; a decision process for how to manage questionnaires with inconsistent, incomplete or illegible answers; and a process for flagging and notifying field staff of surveys with inconsistent responses where a follow-up visit or consultation may be needed. A data entry timeline that ensures that entry does not lag far behind data collection should also be included. In either an optical scan approach or a manual data entry approach, the resulting databases must employ an appropriate relational data structure and linking mechanisms for the instrument prior to database development. Data Cleaning Plan The data quality assurance plan will also include a detailed data cleaning plan. The plan will describe how errors during data collection and data entry if applicable will be identified and addressed. The plan will be submitted as part of the data quality assurance plan. This is critical for data integrity, interoperability of datasets and overall success. Key elements of data cleaning are to include: • Data validation during collection or entry: Automated processes such as validation check code and application of validation logic during collection or data entry, as applicable, will be used to ensure that data collected adhere to design parameters of study instrument, sampling protocol and interview methods. • Full logging of field activity outcomes: Attempts to conduct interviews that result in incomplete data, refusals, or otherwise to be excluded are identified and flagged. • Double data entry: If manual data entry of paper forms is required, forms shall be double-entered and systematically compared for discordant entry to reduce error rate. In this scenario, error ANNEX V. Inception Report Annex V – 19 rates and reconciliation shall be fully documented in a regular report. In the event of excessive error rates, a plan should be outlined with the possibility of further training and re-entry of data. • De-duplication: Merges, concatenations and other data management operations will be consistently and reliably performed on appropriate data; redundant information will not be included in product datasets. • Naming consistency: Data table naming schemes, database logging, journaling and similar tracking features are to be consistently applied in according to a written protocol. • Error or inconsistency detection and removal: Detection of errors and inconsistencies will be logged according to a written protocol. Correction efforts are to be fully documented including an explanation of the strategy used to correct the error and the date and time. Undocumented modification of data to attempt to correct inaccuracies shall be considered unacceptable. • Reporting: Automated reporting should be included in workflow to provide information to IBTCI about non-participation rates, completed cleaning operations and reconciliation performed. Descriptions and corrections of errors should be qualitatively and quantitatively summarized in reports. The format, contents, and timing of these reports will be discussed and determined by the evaluation team. However, there is an expectation that reports will be formal in nature, with structured, tabular outputs from study databases. • Secondary verification: IBTCI will compare subsets of raw data with cleaned data sets using independently developed code. • Specific data check operations will include, but are not limited to, identifying, documenting, explaining and correcting the following type of problems: incorrect formatting or field types incorrect data based on questionnaire skip pattern logic incorrect data based on a numeric range incorrect data based on relationship rules invalid/incomplete or ambiguous dates, or illogical date ranges improperly formatted or stored date values excessive outliers in legal range but outside of expected values incorrect use of free-text specifies fields duplicates of records or fields where not appropriate missing data redundant data non-unique values in unique key fields invalid coded values inconsistent variable values/commonality (similar responses or definitions coded with varying values) invalid journaling/timestamps for records ANNEX V. Inception Report Annex V – 20 interview records that are incomplete, refused, or otherwise subject to possible exclusion invalid or inaccurate geolocations errors with linking variables or key relationships referential integrity problems orphaned records (records that should be linked/associated to others but are not) childless parent records (records that should have related/dependent records but do not) data resulting from deviations from sampling protocol KII, FGD DATA COLLECTION/MANAGEMENT AND QUALITY CONTROL For this evaluation, all qualitative data collection will be comprised of an enumerator and an assistant. The enumerator will be in charge of facilitating discussion, asking questions, etc., while the assistant will ensure proper functioning of audio recording (all qualitative collection will include audio recording) and note taking (provides contextual background for use in producing transcripts and providing guidance to the facilitator and eventual analysis). All qualitative collection will produce audio recordings and full transcripts that adhere to PII protection protocols. These transcripts will be send to the evaluation Team Lead (Michael Cooper) within three days of collection along with summary sheets and full notes form the enumerator and assistant for each round of data collection. Analysis Surveys: All surveys will be statistically tabulated with disaggregation by gender, age, location and component/sub-component. It is anticipated that Microsoft Excel will provide the necessary functionality for quantitative analysis and reporting but if need be, IBTCI can utilize other software like STATA. KII’s/FGD’s: At the end of each interview, we will prepare a 1-2 page interview summary sheet reducing information into manageable themes, issues, and recommendations. Coding involves a systematic recording of data. While numeric codes are not appropriate, descriptive codes can help organize responses. All qualitative data collection will be coded and assess using content analysis and given the lower numbers of qualitative data collection per activity/sub-activity, the scope of collection will be fixed, meaning that there will be no point of saturation that alleviates the need for additional collection. The content analysis will utilize a phased approach where additional codes are produced with each round, double checked with other coders before a final code book is produced and used for analysis. The code book will be made available to USAID upon request. It is anticipated that no specialty software like NIVIVO will be used for this content analysis given the scope of the data collection but IBTCI reserves the right to use NIVIVO if the scope of collection necessitates it. Document Review: Document review will utilize the same protocol for content analysis as the other qualitative data analysis outlined above with a phased approach and development of a code book. ANNEX V. Inception Report Annex V – 21 ADMINSTRATIVE DISSEMINATION PLAN Starting with this Inception Report, it is proposed that all deliverables (for this evaluation the only remaining deliverable is the Final Report) be reviewed in draft form by USAID and any other stakeholders, at the direction of USAID, with written feedback submitted with a follow-on feedback session meeting with all relevant stakeholders to discuss their relevant feedback on the draft. This allows for back and forth conversation around questions and comments before USAID and other stakeholders are invited to submit final feedback which can include expressions of disagreement in written form that can be included as annexes to the final version of the Final Report along with IBTCI’s response on why (or why not) and how the feedback was responded to in the content of the report. This will be done in accordance with the timetable outlined in the Gantt chart below. Likewise, IBTCI strongly advocates the sharing of evaluation results with those whose opinions and data were used in the report, namely the data sources for various KIIs, FGDs and surveys. IBTCI would like to discuss with USAID possibilities around sharing evaluation results with these stakeholders for feedback in an attempt to better utilize CLA principles in the evaluation. ANNEX V. Inception Report Annex V – 0 EVALUATION TIMELINE/WORK PLAN The below Gantt chart outlines the timeline for the primary evaluation tasks. YESS Evaluation Work Plan July 2019 August 2019 September 2019 October 2019 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 Evaluation SOW Approved Draft Work Plan/Evaluation Design prepared and submitted USAID reviews/ approves work plan Inception Report Submitted and Approved Document Review Translation/Back Translation of Instruments TL Travel to DC for Interviews with USAID and Pragma TL Travels to Amman Enumerator Training (from DC and Amman) Data Collection Piloting in Yemen ANNEX V. Inception Report Annex V – 1 YESS Evaluation Work Plan July 2019 August 2019 September 2019 October 2019 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 Piloting Completed and Instruments Finalized TL completes KIIs, FGDs in Amman Data Collection in Yemen Daily Quality Control Checks of Data Collection from Amman EG Growth Specialist in Amman Follow-on Data Collection in Yemen Preparation of Data Sets for analysis Data Analysis Codebook finalized Drafting of Reports Weekly briefings for USAID (phone or quick emails) Draft Report Shared for Feedback Final Report Finalized ANNEX V. Inception Report Annex V – 2 YESS Evaluation Work Plan July 2019 August 2019 September 2019 October 2019 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 Briefings for USAID and other Stakeholders Translation of report ANNEX V. Inception Report Annex V – 0 Evaluation Risk and Risk Management There are two primary risks to utilization of YESS Evaluation Results, 1.) The lack of an articulated end use for YESS evaluation findings and 2.) the ongoing conflict in Yemen. Evaluations that take place in conflict environments are always prone to data quality risks stemming from the inability to collect all the data that is necessary due to security. This is applicable in a variety of ways for YESS due to the evaluator (IBTCI) not being able to enter Yemen and thus having to manage the evaluation from Jordan, which includes the inability to train enumerators. The lack of an identified end￾use for evaluation findings (as defined by a set of decisions and/or decision-making process with correlated data requirements for the decision making) means that the investment in the evaluation is at risk as there is not articulated plan for the utilization of evaluation results. These risks and our mitigation strategies are summarized in the below table. Table 4: YESS Evaluation Risk Management Summary YESS Evaluation Risk Risk Mitigation Inability to train enumerators face to face Recruitment of data collection firm (Yemen Polling Center) with extensive experience in all forms of data collection (KII’s, Face to Face Surveys, WhatsApp Surveys) in Yemen Lack of direct access/control over data management Establishing detailed checklists and data management protocols from point of collection to point of transmission of data sets to IBTCI in Jordan Security conditions in Yemen stemming ability to collect data as envisioned (specifically in Amran, Sanaa) Daily updates from data collection staff on conditions and ability to operate Lack of identified end use of evaluation results More open-ended data collection and evaluability assessment that can hopefully identify findings that USAID or other stakeholders find valuable for decision making ANNEX V. Inception Report Annex V – 1 ANNEX V. APPENDIX TABLE OF CONTENTS APPENDIX #1 YESS Evaluation: 2 APPENDIX #2 YESS EVALUATION: OVERALL YESS MONITORING AND EVALUATION FGD 4 APPENDIX #3 YESS EVALUATION: YESS TRADE TEAM FGD 5 APPENDIX #4 TRADE FACILITATION SURVEY STAKEHOLDER PROFILE 7 APPENDIX #5 TRADE FACILITATION KII 10 APPENDIX #6 TRADE FACILITATION BENEFICIARY FGD 12 APPENDIX #7 YESS AGRICULTURAL LIVELIHOODS TEAM KII 13 APPENDIX #8 YESS AGRICULTURE LIVELIHOODS TEAM FGD 15 APPENDIX 9 YESS EVALUATION: LIVELIHOODS SURVEY 17 APPENDIX #10 OVERALL YESS EMPLOYMENT GENERATION TEAM KII 19 APPENDIX #11 OVERALL YESS EMPLOYMENT GENERATION TEAM FGD 21 APPENDIX #12 YESS EMPLOYMENT GENERATION SME (JOB CREATION) KII 23 APPENDIX #13 YESS EMPLOYMENT GENERATION SME (JOB MATCHING) KII 25 APPENDIX #14 YESS EMPLOYMENT GENERATION WORKFORCE DEVELOPMENT PARTNER KII 26 APPENDIX #15 YESS EMPLOYMENT GENERATION: ACCESS TO FINANCE KII 27 APPENDIX #16 DEMONSTRATION SITE AND FARMER FIELD SCHOOL: LEAD FARMER (OR SITE LEADER) KII 28 APPENDIX #17 DEMONSTRATION SITE AND FARMER FIELD SCHOOL PARTICIPANT SURVEY 29 APPENDIX #18 YESS EVALUATION AGRICULTURAL LIVELIHOODS SAMPLING PLAN 32 APPENDIX #19 YESS EMPLOYMENT GENERATION JOB FAIRS ELECTRONIC SURVEY 0 APPENDIX #20 YESS EMPLOYMENT GENERATION: JOB CREATION SURVEY 2 APPENDIX #21 LITERATURE REVIEW 4 APPENDIX #22 EVALUABILITY ASSESSMENT CRITERIA 8 APPENDIX #23 SAMPLING PROCEDURES: YESS EG COMPONENT 0 ANNEX V. Inception Report Annex V – 2 APPENDIX #1 YESS EVALUATION: OVERALL YESS IMPLEMENTATION/MANAGEMENT KII Date: Location: Interviewee: Interviewer: Note Taker: Audio Recording: Yes/No We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1: What do you consider to the main objectives of this YESS assistance? -Follow up: What were some of the main activities involved with these objectives? -Follow up: How do you feel YESS was a pilot? (What attributes of YESS were being piloted and why?) Q2: What do you feel was the two or three biggest successes of YESS? -Follow up: Why do you say that? -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) Q3: If you had the opportunity to start over, what would you do different? -Follow up: What would need to change in order for you to do this differently? Q4: What was the most surprising thing so far with YESS? (prompts: was it something in the context of implementation, was there something in the management and operations? Was there a surprising result that was positive or negative?) Q5: What was the biggest barrier in implementing YESS from a management and/or operations perspective? Follow up: And how did you overcome/address it…. Did you change intervention design/implementation as a result? ANNEX V. Inception Report Annex V – 3 Follow up: Could you share 1-2 examples of management decisions/adaptation to implementation plan / or to a specific intervention Q6: What was the hardest outcome to achieve in YESS? - Follow up: Why do you say that? -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) Q7: From your viewpoint of YESS Theory of Change, did it hold up or prove to be feasible/right? Why or why not? Q8: What do you think are some of the Lessons Learned from YESS? Feel free to be as broad or specific as needed. Q9: Is there anything that you feel is relevant that we have not covered? ANNEX V. Inception Report Annex V – 4 APPENDIX #2 YESS EVALUATION: OVERALL YESS MONITORING AND EVALUATION FGD Date: Location: Focus Group Participants: Focus Group Facilitator: Note Taker: Audio Recording: Yes/No We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1: What do you feel was the biggest success of YESS M&E? -Follow up: Why do you say that? -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) -Follow up: How do you feel YESS was a pilot? (What dimensions of YESS were being tested and why/how? Or what aspects of YESS had high levels of uncertainty) -Follow up: From your view point, how well did the YESS Theory of Change prove to be valid? If you could revise the Theory of Change what would you change and why? -Follow up: Could you share 1-2 examples of management decisions/adaptation to implementation plan / or to a specific intervention Q2: If you had the opportunity to start over, what would you do different in terms of M&E? Q3: What was the most surprising thing so far with YESS? (prompts: was it something in the context of implementation? Was there a surprising result that was positive or negative?) Q4: What was the hardest outcome to achieve in YESS? - Follow up: Why do you say that? -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) Q5: What do you feel are some recommendation and lessons learned/Best Practices? Q6: Is there anything that you feel is relevant that we have not covered? ANNEX V. Inception Report Annex V – 5 APPENDIX #3 YESS EVALUATION: YESS TRADE TEAM FGD Date: Location: Focus Group Participants: Focus Group Facilitator: Note Taker: Audio Recording: Yes/No We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1: What do you consider to the main objectives of this YESS Trade assistance? -Follow up: What were some of the main activities involved with these objectives? Q2: What do you feel was the biggest success with the technical assistance or training you received? -Probe (in case of overly general response) Was there success in increasing the flow of humanitarian assistance or integrating the private sector? -Follow up: Why do you say that? -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) Q3: What are some of the things you learned during implementation? --Follow up: Do you feel YESS Trade was a pilot? (What attributes of YESS were being piloted and why?) Follow up: If you had the opportunity to start over, what would you do different? -Follow up: What would need to change in order for you to do this differently? Q4: What was the most surprising thing so far? (prompts: was it something in the context of implementation, was there something in the management and operations? Was there a surprising result that was positive or negative?) Q5: What was the biggest barrier in implementing these activities? Q6: What was the hardest outcome to achieve for Trade Facilitation? ANNEX V. Inception Report Annex V – 6 - Follow up: Why do you say that? -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) Q7: Is there anything that you feel is relevant that we have not covered? Recommendation and lessons learned ANNEX V. Inception Report Annex V – 7 APPENDIX #4 TRADE FACILITATION SURVEY STAKEHOLDER PROFILE Name: Stakeholder Connection: A. Port Management B. Port User C. International Trade Policy Maker D. Private Sector International Trade Service Questions – Port Management/Operation (Note: Sub-questions will only be used in surveys) How knowledgeable are you about YESS assistance in support of improved port management and operation? a. very knowledgeable b. somewhat knowledgeable c. not very knowledgeable d. Do not know e. Decline to answer What do you feel the goals of this assistance were? Overall, to what extent has the YESS assistance helped to improve port management and operation? a. significantly b. somewhat c. minimally d. Not applicable e. Decline to answer To your knowledge, have port operations been streamlined due to YESS assistance? a. Yes b. No c. Not applicable d. Decline to answer If “YES”, in what ways? To your knowledge, have port capacities increased due to YESS assistance? a. Yes b. No c. Not applicable d. Decline to answer ANNEX V. Inception Report Annex V – 8 If “YES”, in what ways? To your knowledge, do you feel that humanitarian assistance is being delivered more effectively due to YESS assistance? a. significantly b. somewhat c. minimally d. Not at all e. Decline to answer Have you received other donor or GOY assistance to help you improve services? a. Yes b. No c. Not applicable d. Decline to answer If “YES”, in what ways? Can you think of any examples that demonstrate this? Do you feel that the right people were involved in the assistance? Meaning, given the goals of the assistance, do you feel that there needed to be other people involved in deciding what assistance was needed or other people who could have been involved in some way? a. Yes b. No c. Not applicable d. Decline to answer If “Yes” then who do you think should also have been involved and how should they have been involved? Why? How effective do you feel YESS is in helping you use your new training? a. Very effective b. Somewhat effective c. Not very effective d. Not effective at all e. Not applicable f. Decline to answer If a, b, c or d, why or why not? Can you give some examples of why this is the case? How likely are you to use your new training on a regular basis for the for-seeable future? a. Very likely ANNEX V. Inception Report Annex V – 9 b. Somewhat likely c. Not very likely d. Not at all likely e. Not applicable f. Decline to answer If b, c or d, why or why not? Can you give some examples of why this is the case? ANNEX V. Inception Report Annex V – 10 APPENDIX #5 TRADE FACILITATION KII Date: Location: Interviewer: Interviewee: Note Taker: Audio Recording: Yes/No Stakeholder Profile Name: Stakeholder Connection: A. Port Management B. Port User C. International Trade Policy Maker D. Private Sector International Trade Service We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1: In a few words, what was your involvement in the YESS program? Or put another way, how did you work with the YESS program? How would you describe your relation with YESS team members? Q2: What do you feel were the objectives of YESS and your involvement with YESS? (prompts in the case of overly general answer: “how do you feel port operations were streamlined or made more efficient”, “do you feel humanitarian assistance is being delivered more effectively as a result of this assistance?”) Q3: What do you feel was the biggest success of your work with YESS? -Follow up: Why do you say that? Q4: Do you feel that port operations are more effective or efficient now? Why do you say that? Follow up: In the case of increased effectiveness or efficiency, do you feel it is sustainable? Q5: If you had the opportunity to go back in time and start over with YESS, what would you like to see done differently? ANNEX V. Inception Report Annex V – 11 Q6: If YESS were to continue its activities, what problems would you like to see it address? Q7: Is there anything that you feel is relevant that we have not covered? ANNEX V. Inception Report Annex V – 12 APPENDIX #6 TRADE FACILITATION BENEFICIARY FGD Date: Location: Focus Group Participants: Focus Group Facilitator: Note Taker: Audio Recording: Yes/No We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1: What do you feel was the biggest success with the technical assistance or training you received? -Prompt (in case of overly general response) Was there success in increasing the flow of humanitarian assistance or integrating the private sector? What aspects of port operations were made more efficient or effective? -Follow up: Why do you say that? -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) Q2: What are some of the things you learned? -Follow up: If you had the opportunity to start over, what would you do different? Q3: What was the most surprising thing so far? (possible prompts: Did problems emerge? Was there a surprising result that was positive or negative?) Q4: Do you think the results will be sustained in five years? Why or why not? Q5: Is there anything that you feel is relevant that we have not covered? ANNEX V. Inception Report Annex V – 13 APPENDIX #7 YESS AGRICULTURAL LIVELIHOODS TEAM KII Date: Location: Focus Group Participants: Focus Group Facilitator: Note Taker: Audio Recording: Yes/No We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1: What problems do you feel these activities are meant to address? -Follow-up: Do you feel these activities contributed to economic stabilization? Why do you say this? -Follow-up: Do you feel that the assistance helped farmers improve their competitiveness? Why do you say this? Q2: What is the selection criteria for underserved SMEs? How were they selected or how were the criteria applied? Q3: What do you feel was the biggest success within the YESS Livelihood Activities? -Follow up: Why do you say that? -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) Q4: If you had the opportunity to start over, what would you do different? Q5: What was the most surprising thing so far? (prompts: was it something in the context of implementation, was there something in the management and operations? Was there a surprising result that was positive or negative?) Q6: What was the biggest barrier in implementing these activities? How did you respond to this barrier? Q7: What was the hardest outcome to achieve for Livelihoods? - Follow up: Why do you say that? ANNEX V. Inception Report Annex V – 14 -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) Q8: What are some recommendations or lessons learned from the Livelihoods Component of YESS? Q9: Is there anything that you feel is relevant that we have not covered? ANNEX V. Inception Report Annex V – 15 APPENDIX #8 YESS AGRICULTURE LIVELIHOODS TEAM FGD Date: Location: Focus Group Participants: Focus Group Facilitator: Note Taker: Audio Recording: Yes/No We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1: What problems do you feel the YESS Livelihood activities are meant to address? -Follow-up: Do you feel these activities contributed to economic stabilization? Why do you say this? -Follow-up: Do you feel that YESS helped farmers improve their competitiveness? Why do you say this? Q2: What is the selection criteria for farmers that were trained (outside of the Demonstration Sites and Farmer Field Schools)? How were these criteria applied? Q3: What do you feel was the biggest success within the YESS Agricultural Livelihood Activities? -Follow up: Why do you say that? -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) Q4: If you had the opportunity to start over, what would you do different? -Follow up: What would need to change in order for you to do this differently? Q5: What was the most surprising thing so far? (prompts: was it something in the context of implementation, was there something in the management and operations? Was there a surprising result that was positive or negative?) Q6: What was the biggest barrier in implementing these activities? How did you respond? Q7: What was the hardest outcome to achieve for Livelihoods under this activity? - Follow up: Why do you say that? ANNEX V. Inception Report Annex V – 16 -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) Q8: Is there anything that you feel is relevant that we have not covered? ANNEX V. Inception Report Annex V – 17 APPENDIX 9 YESS EVALUATION: LIVELIHOODS SURVEY Date: Location: Interviewer: Interviewee: Note Taker: Audio Recording: Yes/No We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1. How knowledgeable are you about YESS assistance in providing technical training? a. very knowledgeable b. somewhat knowledgeable c. not very knowledgeable Q2. Overall, to what extent has the YESS assistance helped to increase your farm yields? a. significantly b. somewhat c. minimally d. Not at all If “a, b or c” ask follow-on: If you have experienced an increase in farm yields resulting from your interactions with YESS, how has YESS helped increase your yields? Do you feel this increase in yields will be the same in five years? Why or why not? If answer is “d” to #2 ask follow-on: Why do you think there has been no increase to your yields? Q3. Have you experienced new market linkages as a result of YESS? Meaning have you experienced new relationships with suppliers, buyers, cooperatives, etc.? ANNEX V. Inception Report Annex V – 18 1. Yes 2. No If “Yes” have these new relationships been beneficial? 1. Yes 2. No If “Yes” how have these new relationships been beneficial? If “No” how have these new relationships not helped? Q4. Were you trained on any of these agriculture practices through YESS? a. Bi-ridge planting method b. Drip irrigation system c. Fertigation d. Greenhouses If “YES” Follow Up: to any, in what ways? Have you been able to use these practices? Why or why not? If “No” skip to Q5 Q5. Have you received other donor or GOY assistance to help you improve services? 1. Yes 2. No If “YES”, in what ways? Q6. Is there a negative impact of the ongoing conflict on your ability to use this training on a regular basis? If so is the impact: a. Very significant b. Somewhat significant c. Not very significant d. Not significant at all or minimal Q7. How likely are you to use your new training on a weekly basis for the for-seeable future? a. Very likely b. Somewhat likely c. Not very likely d. Not at all likely If b, c or d, what could be done to make it more likely? ANNEX V. Inception Report Annex V – 19 APPENDIX #10 OVERALL YESS EMPLOYMENT GENERATION TEAM KII Date: Location: Interviewee: Interviewer: Note Taker: Audio Recording: Yes/No We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1: What problems do you feel the YESS Employment Generation activities are meant to address? -Follow-up: Do you feel these activities contributed to economic stabilization? Why do you say this? -Follow-up: What aspect of YESS Employment Generation were being piloted? Why were these aspects being tested through a pilot (probe: was there high levels of un certainty or a lack of evidence?)? How were these aspects tested? -Follow-up: Do you feel the YESS Employment Generation Theory of Change is still valid? If you could go back in time and redo the Theory of change how would you change it? Q2: What do you feel was the biggest success of YESS Employment Generation? -Follow up: Why do you say that? -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) Q3: How did you implement a Buyer Led Growth approach in YESS? What do you feel was the biggest success of this approach? What about the biggest barriers? What was the response to these barriers? Q4: What was the biggest thing you learned from implementing YESS? -Follow up: If you had the opportunity to start over, what would you do differently? -Follow up: What would need to change in order for you to do this differently? ANNEX V. Inception Report Annex V – 20 Q5: What was the most surprising thing so far with YESS? (prompts: was it something in the context of implementation, was there something in the management and operations? Was there a surprising result that was positive or negative?) Q6: What was the biggest barrier in implementing YESS from a management and/or operations perspective? Q7: What was the hardest outcome to achieve in YESS? - Follow up: Why do you say that? -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) Q8: How did the conflict affect results? What was the response? Was the response appropriate? Why or why not? Q9: Is there anything that you feel is relevant that we have not covered? ANNEX V. Inception Report Annex V – 21 APPENDIX #11 OVERALL YESS EMPLOYMENT GENERATION TEAM FGD Date: Location: Interviewee: Interviewer: Note Taker: Audio Recording: Yes/No We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1: What problems do you feel the YESS Employment Generation activities are meant to address? -Follow-up: Do you feel these activities contributed to economic stabilization? Why do you say this? Q2: What do you feel was the biggest success of YESS? -Follow up: Why do you say that? -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) - Follow-up: What aspect of YESS Employment Generation were being piloted? Why were these aspects being tested through a pilot (probe: was there high levels of un certainty or a lack of evidence?)? How were these aspects tested? -Follow-up: Do you feel the YESS Employment Generation Theory of Change is still valid? If you could go back in time and redo the Theory of change how would you change it? Q3: How did you implement a Buyer Led Growth approach in YESS? What do you feel was the biggest success of this approach? What about the biggest barriers? What was the response to these barriers? Q4: Has there been follow up with new employees who benefited from YESS after they started their new jobs? What did this follow up consist of? Q5: If you had the opportunity to start over, what would you do differently? -Follow up: What would need to change in order for you to do this differently? Q6: What was the most surprising thing so far with YESS? (prompts: was it something in the context of implementation, was there something in the management and operations? Was there a surprising result that was positive or negative?) ANNEX V. Inception Report Annex V – 22 Q7: What was the biggest barrier in implementing YESS from a management and/or operations perspective? Q8: What was the hardest outcome to achieve in YESS? - Follow up: Why do you say that? -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) Q9: How did the conflict affect results? What was the response? Was the response appropriate? Why or why not? Q10: Do you have any recommendations and lessons learned / best practices? Q11: Is there anything that you feel is relevant that we have not covered? ANNEX V. Inception Report Annex V – 23 APPENDIX #12 YESS EMPLOYMENT GENERATION SME (JOB CREATION) KII Date: Location: Interviewee: Interviewer: Note Taker: Audio Recording: Yes/No We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1: What were the problems that you feel this assistance was supposed to help solve for your business? Q2: What do you feel was the biggest success of your experience with YESS? -Follow up: Why do you say that? -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) Q3: If you had the opportunity to start over with YESS, what would you do differently? -Follow up: What would need to change in order for you to do this differently? Q4: What was the most surprising thing so far with working with YESS? (probe: was there an un￾expected result (positive or negative) or something similar?) Q5: Do you feel the action plan responds to customer preferences for your goods and services? Meaning do you feel the problems that the action plan addressed account for what customers want? Or put another way, how were customer preferences accounted for in developing the Action Plans? Q6: How much do you think YESS affected your sales volume? Did it help, hurt or no impact? Why do you say this? Q7: Were you able to create new jobs as a result of YESS? -Follow up: If not, why not? -Follow up: If so how sustainable do you feel these jobs are? -If they are not sustainable, why not? ANNEX V. Inception Report Annex V – 24 Q8: How did the conflict affect results? Why or why not? Q9: Is there anything that you feel is relevant that we have not covered? ANNEX V. Inception Report Annex V – 25 APPENDIX #13 YESS EMPLOYMENT GENERATION SME (JOB MATCHING) KII Date: Location: Interviewee: Interviewer: Note Taker: Audio Recording: Yes/No We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1: What do you feel was the biggest success of your experience with YESS? -Follow up: Why do you say that? -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) Q2: If you had the opportunity to start over with YESS, what would you do differently? -Follow up: What would need to change in order for you to do this differently? Q3: What do you think the problems are that the YESS technical assistance was meant to address? How well do you think YESS helped to address these problems? Why do you say this? Q4: What was the most surprising thing so far with working with YESS? Q5: How did the conflict affect results? What was the response? Was the response appropriate? Why or why not? Q6: Is there anything that you feel is relevant that we have not covered? ANNEX V. Inception Report Annex V – 26 APPENDIX #14 YESS EMPLOYMENT GENERATION WORKFORCE DEVELOPMENT PARTNER KII Date: Location: Interviewee: Interviewer: Note Taker: Audio Recording: Yes/No We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1: What do you feel was the biggest success of your experience with YESS? -Follow up: Why do you say that? -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) -Follow up: How well do you consider the recruitment assistance to have worked? Why or why not? Follow up: How well do you consider the pre-employment training to have worked? Why or why not? Q2: If you had the opportunity to start over with YESS, what would you do differently? Why do you say this? Q3: How well were you able to find jobs for women and youth? How well were you able to find jobs for those that were vulnerable to being recruited into the conflict? Why do you say this? Q4: How did the conflict affect results? Why or why not? Q5: Is there anything that you feel is relevant that we have not covered? ANNEX V. Inception Report Annex V – 27 APPENDIX #15 YESS EMPLOYMENT GENERATION: ACCESS TO FINANCE KII Date: Location: Interviewee: Interviewer: Note Taker: Audio Recording: Yes/No We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1: What do you feel was the biggest success within the YESS program? -Follow up: Why do you say that? -Follow up: What is the best way to learn more about this (is there material or other sources of evidence?) Q2: If you had the opportunity to start over with a time machine, what would you do different? Q4: What is the biggest barrier to these results being sustained over the next five years? Follow-up: Do you think the results will be sustained in five years? Why or why not? Q5: How did the conflict affect results? What was the response? Was the response appropriate? Why or why not? Q6: Is there anything that you feel is relevant that we have not covered? ANNEX V. Inception Report Annex V – 28 APPENDIX #16 DEMONSTRATION SITE AND FARMER FIELD SCHOOL: LEAD FARMER (OR SITE LEADER) KII Date: Location: Interviewee: Interviewer: Note Taker: Audio Recording: Yes/No We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1: What type of activities and trainings did you perform? Q2: Were these activities and trainings what was originally envisioned? If not what changed and why? Q3: What was the biggest success? Why do you say this? Q4: What was the biggest problem? Why do you say this? How did you address it? Q5: If you could go back with a time machine and start over what would you do different? Why do you say this? Q6: What was the biggest barrier to participants using the training? Why do you say this? Q7: Overall do you feel the training was helpful? Why or why not? Q8: Is there anything else you feel is relevant that we have not covered? ANNEX V. Inception Report Annex V – 29 APPENDIX #17 DEMONSTRATION SITE AND FARMER FIELD SCHOOL PARTICIPANT SURVEY Date: Location: Interviewer: Interviewee: Note Taker: Audio Recording: Yes/No We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1. How knowledgeable are you about YESS assistance at demonstration sites or farmer field schools? a. very knowledgeable b. somewhat knowledgeable c. not very knowledgeable d. Decline to answer Q2. Overall, to what extent has the YESS assistance helped to increase your farm yields? a. significantly b. somewhat c. minimally d. Not at all e. Decline to answer If “a, b or c” ask follow-on: If you have experienced an increase in farm yields resulting from your interactions with YESS, how has YESS helped increase your yields? Do you feel this increase in productivity will be the same in five years? Why or why not? If answer is “d” to #2 ask follow-on: ANNEX V. Inception Report Annex V – 30 Why do you think there has been no increase to your productivity? Q3. Have you experienced new market linkages as a result of YESS? Meaning have you experienced new relationships with suppliers, buyers, cooperatives, etc.? 1. Yes 2. No 3. Decline to answer If “Yes” have these new relationships been beneficial? 1. Yes 2. No 3. Decline to answer If “Yes” how have these new relationships been beneficial? Can you give some examples to demonstrate this? If “No” how have these new relationships not helped? Q5. Q4. Were you trained on any of these agriculture practices through YESS? a. Bi-ridge planting method b. Drip irrigation system c. Fertigation d. Greenhouses If “YES” Follow Up: to any, in what ways? Have you been able to use these practices? Why or why not? If “No” skip to Q6 Have you been able to use these practices? Why or why not? Q6. Have you received other donor or assistance to help you improve services? 1. Yes 2. No 3. Decline to answer If “YES”, in what ways? Q7. Is there an impact of the ongoing conflict on your ability to use this training on a regular basis? If so is the impact: a. Very significant b. Somewhat significant c. Not very significant d. Not significant at all or minimal e. Decline to answer If “a” or “b” follow up: Why do you say this? Can you give some examples to demonstrate this? ANNEX V. Inception Report Annex V – 31 Q8. How likely are you to use your new training on a regular basis for the for-seeable future? a. Very likely b. Somewhat likely c. Not very likely d. Not at all likely e. Decline to answer If b, c or d, what could be done to make it more likely? ANNEX V. Inception Report Annex V – 32 APPENDIX #18 YESS EVALUATION AGRICULTURAL LIVELIHOODS SAMPLING PLAN Given that the YESS livelihoods component was prominently rural based which increases resources needed for enumerators to travel greater distances (logistic and labor costs) and increased security risks as enumerators are further from population centers and more exposed to various threats. Given these barriers, the evaluation prioritized coverage (data collection) on as many value chains, training recipients, governorates, demonstration sites and FFS as possible given these resource and security constraints. Accordingly, we sought to concentrate our data collection into specific governorates so that enumerators could collect data on multiple YESS components and sub-components (i.e. demonstration sites, FFS, etc.) and minimize travel to numerous governorates. The selection of the five governates for data collection allows for collection on all YESS components and within the Livelihood Component allow for data collection on the honey, coffee, livestock and potato value chains in four governates and includes trainees, demonstration sites and FFS. Hence this sampling plan maximizes coverage over geographic areas, value chains, method of assistance delivery (training, demonstration sites and FFS) while not introducing additional governates that unduly increases the resource costs and security risks. Demonstration Site and Farmer Field School Sampling The YESS Demonstration Sites and FFS were selected along a number of criteria (i.e. site leader is willing, geographical location/ease of access, etc.) and generally follow all the commonly used design criteria for a demonstration site and/or FFS. Usually, especially in conflict situations, this type of programming is usually sampled using a snowball method or primarily from site leader referrals with associated levels of potential bias in the data collection. What we propose is to use an exponential discriminative snowballing method by initially requesting a list of referrals of farmers who utilized the demonstration site or FFS from the site leader. This can be considered the first phase. The site leader is solicited for 10 references as they possess a unique capacity as site leader that needs to be taken advantage of (i.e. the ability to have the contact information for ten farmers who used the demonstration site). Of these references, we will survey four farmers selected at random (each is randomly assigned an ID# which are then randomly selected for enumeration). We will then ask those four farmers for two referrals each. Of these two we will randomly choose one through coin flip and survey them as the process repeats itself until we survey 20 farmers. If the farmer that is randomly selected for a survey has already been surveyed or is unavailable/unwilling to complete the survey, then 1.) the other referred farmer is enumerated and if that person has also already been enumerated or incapable or unwilling to take the survey, then the #5 farmer from the original referral list from the Site Leader is chosen for the survey and solicited for two referrals as normal until 20 farmers are surveyed. If this scenario repeats itself, the #6 farmer from the original referral list is chosen and the normal process creates itself. This introduces randomization in an attempt to mitigate bias and increase variation in the population surveyed. This type of snowballing method is commonly used but based on an initial literature/systematic review, it has not been used in an evaluation for demonstration sites and/or FFS. While the demonstration sites were not a significant expenditure of YESS resources, this proposed sampling method creates a unique opportunity for USAID to make a contribution to the evidence base for evaluation methods for agriculture demonstration sites and FFS. This sampling method poses no additional cost to data collection and given the prominent use of demonstration sites/FFS in international development programming at large, there is a lot of learning potential in the application of this sampling method. ANNEX V. Inception Report Annex V – 0 Trainings, Demonstration Sites and Farmer Field Schools included in data collection Value Chain Dates of Training Location Training Total Trainees Male Coffee 9/8-9/18 Bani Matar,Sana'a Harvest & Post-Harvest Techniques for Coffee Crop 1 21 21 Coffee 9/10-11/18 Bani Matar,Sana'a Harvest & Post-Harvest Techniques for Coffee Crop 2 24 24 Coffee 9/12-13/18 Bani Matar,Sana'a Harvest & Post-Harvest Techniques for Coffee Crop 3 31 0 Coffee 9/15-16/18 Bani Matar,Sana'a Harvest & Post-Harvest Techniques for Coffee Crop 4 23 22 Honey 9/3-4/18 Daw'an,Hadramout Honey Cluster Training 1 24 0 Honey 9/5-6/18 Daw'an,Hadramout Honey Cluster Training 1 40 40 Honey 9/10-11/18 Daw'an,Hadramout Honey Cluster Training 1 22 0 Honey 9/24-25/18 Osaymat,Amran Honey Training for Beekeepers 3 36 36 Honey 9/12-13/18 Daw'an,Hadramout Honey Cluster Training 1 34 34 Honey 9/26-27/18 Osaymat,Amran Honey Training for Beekeepers 1 23 23 Honey 9/29-30/18 Osaymat,Amran Honey Training for Beekeepers 2 29 29 Coffee 9/10-11/18 Bani Matar,Sana'a Harvest & Post-Harvest Techniques for Coffee Crop 1 4 4 Coffee 9/8-9/18 Bani Matar,Sana'a Harvest & Post-Harvest Techniques for Coffee Crop 1 7 7 Livestock 10/15-21/2018 Maifa,Hadramout Livestock Training 1 26 18 Livestock 10/22-25/2018 Maifa,Hadramout Livestock Training 1 31 18 Livestock 10/29/2018- 11/01/2018 Maifa,Hadramout Livestock Training 1 32 12 Livestock 11/5-8/2018 Maifa,Hadramout Livestock Training 1 40 5 Livestock 11/26-29/2018 Ghayl Bin Yumain,Hadramout Livestock Training 13 31 Livestock 12/2-5/2018 Ghayl Bin Yumain,Hadramout Livestock Training 11 31 14 Livestock 12/23-26/2018 Ghayl Bin Yumain,Hadramout Livestock Training 19 35 ANNEX V. Inception Report Annex V – 1 Honey 09/16-17/2018 Tareem,hadramout Honey ClusterTraining 3 31 31 Honey 09/19-20/2018 Tareem,hadramout Honey ClusterTraining 3 29 29 Honey 09/23-24/2018 Tareem,hadramout Honey ClusterTraining 3 35 35 Honey 09/26-27/2018 Tareem,hadramout Honey ClusterTraining 3 25 25 Potato 11/24-28/2018 Jabal Eial Yazid,Amran Potato Cluster Training 1 30 24 Potato 12/3-6/2018 Jabal Eial Yazid,Amran Potato Cluster Training 2 30 23 Potato 11/28/2018- 12/2/2018 Jabal Eial Yazid,Amran Potato Cluster Training 3 30 25 Potato 12/8-11/2018 Jabal Eial Yazid,Amran Potato Cluster Training 4 30 24 Potato 12/12-16/2018 Jabal Eial Yazid,Amran Potato Cluster Training 5 40 32 Potato 12/17-20/2018 Jabal Eial Yazid,Amran Potato Cluster Training 6 55 48 Livestock 12/10-13/2018 Ghayl Bin Yumain,Hadramout Livestock Training 26 31 5 Livestock 12/17-20/2018 Ghayl Bin Yumain,Hadramout Livestock Training 27 32 6 Activity Village/District/governorate Start date Completion date Goat fattening Tariba/Sayo’un/Hadramout 1/4/2019 4/3/2019 Boor/Sayo’un/Hadramout 1/4/2019 4/3/2019 Almagad/Qyal/Hadramout 1/13/2019 4/12/2019 Modified beehives for queen inspection and honey extraction Aljahay/Daw’an/Hadramout 1/28/2019 4/28/2019 Substitute bee feeding Aljahay/Daw’an/Hadramout 2/3/2019 3/28/2019 Bi-ridge potato plantation Joub/Eial Yaziz/Amran 1/20/2019 6/13/2019 ANNEX V. Inception Report Annex V – 2 Activity Village/District/governorate Start date Goat fattening/heard management Tariba/Sayo’un/Hadramout 1/4/2019 Almagad/Qyal/Hadramout 1/13/2019 Bi-ridge potato plantation Joub/Eial Yaziz/Amran 1/20/2019 Potato processing Joub/Eial Yazid/Amran 1/20/2018 Dairy processing Tarees/Sayo’un/Hadramout 2/6/2019 Almagad/Qayl/Hadramout 1/15/2019 ANNEX V. Inception Report Annex V – 0 APPENDIX #19 YESS EMPLOYMENT GENERATION JOB FAIRS ELECTRONIC SURVEY Date: Location: Interviewer: Interviewee: Note Taker: Audio Recording: Yes/No We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1. Were you involved with a YESS job fair? a. Yes b. No c. Decline to answer Q2. How were you involved with the job fair? a. Employer looking for employees b. Employee looking for employer c. Decline to answer If “a” skip to Q3, If “b” skip to Q4 Q3. Were you able to find employees at the job fair? a. Yes b. No c. Decline to Answer If “a”, are these employees full time? Are these employees still employed by you? Q4. Were you able to find employment as a result of the job fair? a. Yes b. No ANNEX V. Inception Report Annex V – 1 c. Decline to Answer If “a” was the employment full time? Are you still employed as a result of the job fair? a. Yes b. No c. Decline to Answer If “a” to Q4: Were you offered a contract with your employment? a. Yes b. No c. Decline to Answer ANNEX V. Inception Report Annex V – 2 APPENDIX #20 YESS EMPLOYMENT GENERATION: JOB CREATION SURVEY Date: Location: Interviewer: Interviewee: Note Taker: Audio Recording: Yes/No We appreciate you taking the time to speak with us today. As you may know, we are conducting an evaluation of the Yemen Economic Stabilization and Success Program which is funded by The United States Agency for International Development (USAID) since early 2018. We are interested in your opinions and experience with some of the activities of this program and your participation in speaking with us is completely voluntary. Anything you tell us is strictly confidential. Nothing you say will be personally attributed to you in any way including any reference to your identity or location. There will be no permanent record of our conversation today. Again, your participation in this discussion is totally voluntary. Are you willing to speak with us and answer a few questions? Is there anything else I can answer for you before we begin? “If not then proceed to question #1” Q1: Are you employed by (INSERT NAME OF SME CURRENTLY COLLECTING DATA AT)? a.) Yes b.) No c.) Decline to Answer If “a” proceed to Q2 If “b” ask follow up: How did you know about this interview? If “c” respond with: “Thank you for your time today. We only want to interview people we know are employed here so to make better use of your time we will not need to interview you”. Then terminate the survey. Q2: How long have you been employed here? a.) Less than one month b.) Between one and six months c.) Between six months and a year d.) Between a year and a 1.5 years e.) More than 1.5 years f.) Decline to answer Q3: Have you been employed full time here during this period? a.) Yes ANNEX V. Inception Report Annex V – 3 b.) No c.) Decline to answer If “b” ask the follow on: “If you are not working fulltime, how many hours per week are you working on average?” If “a” follow up: Have you been offered a contract as a result of this employment? a. Yes b. No c. Decline to answer Q4: Were you employed immediately before your current employment? a.) Yes b.) No c.) Decline to answer If “a” then skip to Q5. If “b” skip to Q6. Q5: Was this previous employment fulltime? a.) Yes b.) No c.) Decline to answer Q6: Do you feel you will be employed at your current job for the next year? a.) Yes b.) No c.) Decline to answer If “a” or “b” ask the follow on: “why do you think this?” Q7: Is there anything we have not asked that you feel is important? ANNEX V. Inception Report Annex V – 4 APPENDIX #21 LITERATURE REVIEW This literature review will draw upon the YESS ToC, first covering YESS assumptions and then relevant Component ToC, to organize the selection of relevant evidence available. It also will highlight the potential for this evaluation to inform a larger community of interest beyond specific USAID decision making. YESS Assumptions Given the importance of YESS’s stated assumptions, it is important to assess available evidence from programs similar to YESS that have been implemented in conflict zones. A relevant study is Daniel Karrels’ Aid, Power and Grievances: Lessons for War and Peace from Rural Afghanistan37 which assessed the cascading effects of various USG interventions in rural Afghanistan meant to increase employment as a means of mitigating conflict. There are important differences between this study and YESS., but the lower level economic objectives of the interventions assessed by Karrel resemble the objectives of YESS.). Karrel found that given the lack of information, high levels of uncertainty (generally caused by the “fog of war”) implementers developed partnerships and empowered un-traditional groups and individuals that eventually created higher levels of local disruption. USAID itself has pushed out guidance for conflict sensitivity in agricultural programming in order to: • Understand the conflict dynamics in the context in which the programming is taking place, especially regarding inter-group relationships; • Understand the interaction between the intervention and the conflict dynamics in the context; • Act upon this understanding in order to mitigate negative effects from the conflict context on the programming outcomes or the programing on the conflict dynamics.38 In this same guidance USAID outlines different questions of relevance for fragile contexts, active conflict and post-conflict programming to include: • Which sources of division are most likely to draw more people into fighting or increase levels of violence? • How do people reach across the lines of fighting, even in seemingly insignificant ways? Who is able to do this? Are there ways the project could support those efforts? • How would escalations in violence change risks to project participants? To staff? To others in the community? • When operating environments become inaccessible due to violence, or when armed actors threaten the delivery of food assistance, what adjustments need to be made? How can implementation and monitoring of a project continue? Evidence so far indicates “a relatively secure environment for aid projects to be implemented” is necessary for obtaining even lower level economic growth outcomes. Therefore, we will seek to understand how the ecosystems in which YESS operates in are “secure” enough for outcomes to emerge. YESS success may not be dependent upon stability country wide, but it may require stability within the specific eco-systems in which it is operating. 37 The Economics of Peace and Security Journal, Vol. 10 No. 2. 38 Conflict Sensitivity in Food Security Programming, USAID (DHCA/CMM) 2016. ANNEX V. Inception Report Annex V – 5 Trade Facilitation ToC The YESS trade facilitation activity has a heavy emphasis on improving Customs operations. An effective evaluation of these outcomes requires an evidence driven design based on the best “lessons learned” available. The World Customs Organization has provided valuable insight into best practices around Customs capacity development to include: • Accurate diagnosis of capacity building needs and the development of country-specific responses • Sustained high-level political will and commitment • Enhanced cooperation and coherence • Greater ownership by and participation of Customs personnel • Realistic government and donor expectations • Adequate human and financial resources devoted to capacity building initiatives39 These “best practices” are largely reflected in additional research on the topic to include: • Coordination: Coordination of stakeholders, such as government agencies and businesses, which have an interest in the outcome of the capacity building, is important. It also involves coordination between the international and donor organizations themselves to take advantage of synergies that exist between respective programs. • Commitment: Political commitment to the reform process is vital. Without it, funding the necessary reforms becomes problematic. • Ownership: It is critical that projects do not adopt a ‘one size fits all’ approach. Such an approach to capacity building alienates local stakeholders and undercuts sustainability. • Needs Assessment: It is important to assess the needs of the individual country or countries in question and analyze those needs against the stated objectives of the program. This may involve undertaking a diagnostic check, benchmarking and a gap analysis to establish a baseline against which success will later be measured. • Improving Management Capacity: Improving management capacity is directly related to sustainability in two respects. First, if ‘local’ skills are developed to manage the organizational change inherent in many capacity building programs, local ‘ownership’ will be enhanced. • Use of Common Standards: Any training conducted as part of capacity building should reinforce common standards and best practices rather than simply focus on national or regional variations of them. It is still important, however, to take those variations into account in the context of best practice. • Adopt Holistic Approach: It should be recognized that capacity building cuts across a range of different government departments and has the potential to impact a diverse range of public and private sector interests. • Measurement: It is important to ask how the success of the capacity building program will be measured. What are the desired metrics that will be measured?40 39 Effective Customs Capacity Development, World Customs Organization. 2003 40 Achieving Sustainability in Customs Capacity Building, Stephen Halloway. World Customs Journal, 2007. ANNEX V. Inception Report Annex V – 6 These best practices will be integrated into data collection among the various Trade and Facilitation activity participants, primarily through key informant interviews KII and focus group discussions (FGDs) and through survey questions. These best practices can also be coded terms for content analysis of various materials. Agriculture Livelihoods ToC USAID has commissioned research efforts to identify “lessons learned” regarding the design, implementation and sustainability of agricultural knowledge transfer through activities similar to YESS (training, demonstration sites and FFS) to include: • Utilization of Knowledge, Attitude and Practice principles around uptake of new behaviors (seeing is believing, learning by doing, etc.) • Risk Management (farmers can try a new idea with little cost) • Efficiency (the ability to reach a lot of beneficiary farmers especially if using ToT approaches) • Tradition (communities are familiar with demonstration sites) • Prevalence of weak participatory approaches and demonstration sites being too top lead in terms of design and implementation which leads to poor uptake of desired behavior change and sustainability41. Farm Field Schools (FFS) are meant to be more hands on and participatory than standard demonstrations sites and this description largely holds true for the FSS activities under YESS. A review of applicable impact and qualitative evaluations have identified three primary implication for the design and implementation of FFS to include: • Facilitation- Farmer field schools require skilled facilitators and attempts to target facilitators based on education or literacy levels may be less effective than targeting based on ability to communicate, and appropriate training which enables facilitators to use a bottom-up approach. This may contribute to the lack of results for FFS at scale while smaller, pilot type FFS consistently achieve targeted results. • Diffusion- At a local scale, farmer field schools rely on hands-on experience and demonstration of the benefits of new methods, so diffusion of methods to non-participant neighbors is unlikely. The qualitative studies also showed that strong social networks and concrete, simple methods could enable diffusion, while low levels of social cohesion and socioeconomic differences between participants and non-participants were barriers to diffusion.' • Targeting-The majority of FFS projects end up reaching better-off farmers, which appears to have introduce bias into results. Over half of the projects included in the systematic reviews we have included here used pro-poor targeting but were unable to reach the desired populations. This could be partly due to the more “open” nature of FFS.42 41 Improving the Management of Agricultural Demonstration Sites in Food Security Programs: A Practitioners Guide. USAID, 2017. 42 Farmer Field Schools for Improving Farming Practices and Farmer Outcomes in Low- and Middle-income Countries: A Systematic Review. Hugh Waddington, Birte Snilstveit, Jorge Garcia Hombrados, Martina Vojtkova, Daniel Phillips, Phil Davies, Howard White, 3ie, 2015. ANNEX V. Inception Report Annex V – 7 USAID has prioritized the adoption of Climate Smart Agriculture Practices (CSA) and has put out guidance which outlines several key “lessons learned” that are directly applicable to YESS programming including43: • Nuanced Economics- The standard economic barriers to uptake of new practices/products apply to CSA, namely high up-front cost, dis-economies of scale, low access to financial services/products, but in much different ways and to different degrees. For example, initial costs have been shown to be a much higher barrier for farmer uptake of CSA practices compared to the perceptions of implementers and the shifting nature of these economic incentives as season change. • Cultural Alignment- Informal agricultural practices are deeply integrated into the way of life in rural areas. They drive daily routines and are closely integrated with the social dynamics of communities. Hence the introduction of new practices must align with these dynamics and mitigated disruption and ensure “do no harm”. • Eco-systems are Critical- CSA is a small set of practices in a much large agricultural sector that is itself embedded within a societal eco-system. In order for CSA to achieve adoption, there are basic conditions and agricultural practices that must exist in order for farmer mind sets to be incentivized into adopting CSA practices. Employment Generation ToC The YESS Employment activity, specifically the work with anchor firms, takes a buyer led approach to “sustainably increase sales”. The idea is to stimulate employment opportunities, under the theory that an increase in demand for business services/products will result in an increase in demand for labor to produce those services/products. As stated in the YESS Employment Generation work plan, “the anchor firm’s SME supply chain will create jobs through a buyer-led approach. This starts with a purchase order or similar document from a buyer outlining in detail required product quantities, specifications, and quality standards. “The buyer led approach to job growth is based on the idea that there can be no sustainable creation of jobs or increase in income without accounting for and designing around the continuous and definitive judgment of consumers. In a certain sense, these consumers— external to the project, unmanageable, and largely unpredictable—are the only ‘project monitors’ who really matter. The buyer-led approach looks beyond individual transactions to the sustainability of commercial relationships between buyers and sellers over time.44 In order to evaluate a buyer-led approach, it is helpful to consider the purchase order as the starting point in a theory of change. The purchase order contains all the attributes of a product to which an end user has committed. If the product is produced with those attributes, the end result is a transaction. The problems solved in a buyer-led approach are associated with meeting the requirements of the purchase order. The evaluation must be careful to distinguish between “demand” and “need”. “Demand” is articulated in a purchase order, while “need” is the business practices and capacity to meet “demand”. The evaluation will seek to articulate and assess how YESS captured “demand” and defined “needs” in relation to the “demand”. 43 Adoption of Climate-Smart Agriculture in Africa: Constraints, Incentives and Recommendations. USAID, September 2016. 44 We do Know Now: A Buyer Led Approach to Creating Jobs for the Poor, James Riordan. New Academia Publishing, 2014. ANNEX V. Inception Report Annex V – 8 APPENDIX #22 EVALUABILITY ASSESSMENT CRITERIA Project – Evaluability Assessment Assessment Categories Strengths, Weaknesses, and Next Steps Dimension 1: Is the problem clearly defined and is there sufficient evidence to support the problem diagnostic? Standards and best practices 1. Is the problem(s) clearly defined and understood with sufficient evidence/quantitative (baseline) data available to support claims? 2. Is there evidence to support root causes identified? 3. Will all root causes be addressed by the proposed intervention or other, complementary intervention(s)? 4. Is the institutional context understood, the political economy understood, and does the team clearly understand how the proposed intervention(s) link to other initiatives by the gov’t and/or other partners? 5. Is there a clear understanding on how different social and cultural dynamics (gender, poverty, race, ethnicity, etc.) may be influenced by or influence the problem identified? Strengths: Weaknesses: Next steps: Dimension 2: Are the project objectives and theory of change/logic clearly defined? Standards and best practices 1. Is the objective of the Project clearly stated with a link from the problem diagnostic? 2. Is the project logic plausible and based on existing evidence and literature? If there is limited evidence, is there commitment to generating evidence via an impact evaluation? 3. Is it clear which component(s) of the problem diagnostic will be addressed by the proposed intervention, which will be addressed by complementary activities and which will remain risks? 4. Are the inputs, outputs, outcomes clearly defined? 5. Is the timeline for expected results clear and based on evidence? 6. Is it clear whether or not benefits are expected to be sustained beyond the life of the program? Strengths: Weaknesses: Next steps: Dimension 3: Are the risks and assumptions clearly defined with potential risk mitigation strategies? ANNEX V. Inception Report Annex V – 9 Project – Evaluability Assessment Assessment Categories Strengths, Weaknesses, and Next Steps Standards and best practices 1. Are the risks to achieving expected results clear, with clearly defined risk mitigation strategies? 2. Is it clear how risks will be monitored? 3. Is it clear how design and implementation may be altered as information on new risks/realization of risks occurs? 4. Does the project team make a critical assessment of the degree to which there may be blind spots or unknown unknowns in a project of this nature (e.g. how foreseeable are the potential risks that may arise in new sectors)? Strengths: Weaknesses: Next steps: Dimension 4: Are project participants clearly defined and justified in terms of geographic scope and eligibility criteria? Standards and best practices 1. Is the selection criteria for project participants clearly defined and based on the problem and evidence in the program logic? 2. Is program participants’ selection based on credible, quantifiable selection criteria? 3. Are specific demographics (age, gender, poverty status) defined where necessary? 4. Are the geographic location(s) for the Project defined and based on the problem listed above and evidence in the program logic? 5. Will the Project design and implementation plan vary by different sub-groups and/or geographic locations based on the problem listed above and evidence in the program logic? 6. Can the selection be replicated for the purposes of an evaluation? Strengths: Weaknesses: Next steps: ANNEX V. Inception Report Annex V – 0 APPENDIX #23 SAMPLING PROCEDURES: YESS EG COMPONENT SME Selection for Data Collection (Sanaa, Hadramout, Aden) Job Creation SMEs: Using the Excel sheet labeled “SME List of Partners”, for “Employment Generation SMEs” we used the first tab “Partners-EG) and the “Location” column to select all instances of SME’s in Sanaa and Hadramout (this meant including all references to “Al Mukalla” or just “Mukalla” since it is located in Hadramout). We then assigned each SME location in Sanaa and Hadramout a random number in a new column using the Excel command “= RAND()”. We then sorted these randomly assigned numbers according to sequence and selected the first seven randomly assigned numbers for Sanaa and four for Hadramout (per the draft sampling plan) with their correlated SME for inclusion in data collection. Per the draft sampling plan we will rely on Pragma to identify the two senior most individuals involved with YESS at each SME for a KII. Job Matching SMEs: The same method was used on the "Partners-WD" tab except Aden was included in addition to Sanaa and Hadramout. This was done because of the lower numbers of SMEs involved, compared to Job Creation, and the fact that data collection teams will already be in Aden and can collect this data at a marginal increase in cost but it adds five more SMEs in a pool that would otherwise be only nine. Job Creation SMEs Selected • Al Morooj Hotels • Al Wasat • Al Rayan Hospital (Feedback loop beneficiary that was randomly selected) • Be Beautiful Center • Ytech • Al Jeel Al Jadid • Yemen Egyptian • Rover Bird Company • Suroor • IAESCO • Algazira • SBC Mall (Feedback loop beneficiary that was purposively sampled, meaning not randomly selected) • Al Kuraimi Bank (Feedback loop beneficiary that was purposively sampled, meaning not randomly selected) Job Matching SMEs Selected • Al Faqih • Jalap Bros Company • Cluster of Marketing • Cluster Trading Companies • Cluster Clinics in Sanaa • Rexel Security • Yemen Icon & NDPA • Aden Coral Hotel • Aden German International Hospital • Aden Cuban Hospital • Muala Plaza Hotel • Buraihi Hospital • Aico ANNEX V. Inception Report Annex V – 1 • Al Ameen Soft (Feedback loop beneficiary that was purposively sampled, meaning not randomly selected) SME Employment (Employees) Attributable to YESS For all SMEs selected for data collection, the top two senior most leaders (identified by Pragma) involved with YESS will be interviewed. Pragma will provide lists and other employment documentation for new employees attributable to YESS for each SME to be surveyed. IBTCI will randomly select from these employment lists up to ten new employees for each SME to be surveyed. In the case that there is less than ten new employees identified for an SME, we will survey all the employees identified. Financial Providers There are only three SME Financial Service providers that took part in YESS and all three will be collected on. All three providers are in Sanaa and include Al-Kuraimi MF Bank (KIMB), Azal MF Program and the Yemen Kuwait Bank (YKB). 1 ANNEX VI. Sources of Information Annex VI 1 Annex VI: Sources of Information Desk Review Materials “Activity Approval Memorandum, YESS”, Action Memorandum. USAID, 12/17. Activity Monitoring, Evaluation, & Learning Plan (AMELP). Pragma Corporation, 1/18. Annual Report-Middle East Economic Growth Best Practices Project. The Pragma Corporation, FY 2018. Middle East Economic Growth Yemen Economic Stabilization and Success Program (MEG/YESS) Employment Generation Workplan. The Pragma Corporation, 1/18. Middle East Economic Growth Yemen Economic Stabilization and Success Program (MEG/YESS) Livelihoods Workplan. The Pragma Corporation, 1/18. Middle East Economic Growth Yemen Economic Stabilization and Success Program (MEG/YESS) Trade Facilitation Workplan. The Pragma Corporation, 1/18. Middle East Economic Growth Yemen Economic Stabilization and Success Program (MEG/YESS) Quarterly Report . The Pragma Corporation, FY 2018, Q3. Middle East Economic Growth Yemen Economic Stabilization and Success Program (MEG/YESS) Quarterly Report. The Pragma Corporation, FY 2017-18, Q1-Q4. (all quarterly reports) USAID/Yemen Site Visit Report, (Workforce Development Job Matching, Sanaa). IBTCI , 12/18. USAID/Yemen Site Visit Report, (Livelihoods Component). IBTCI, 1/19. YESS, Livelihood Components Training Sign-in Sheets (various) YESS, Employment Generation Hiring Lists (SME generated) ANNEX VIII. Final Report Feedback and Response Annex VIII 1 Annex VII: Disclosure of any Conflicts of Interest Name Michael Cooper Title Consultant Organization IBTCI Evaluation Position? X Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Yemen Economic Stabilization and Success Program I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1.Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2.Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3.Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design ANNEX VIII. Final Report Feedback and Response Annex VIII 2 or previous iterations of the project. 4.Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5.Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6.Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature s/ Michael Cooper Date 02-11-2020