January 31, 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by DevTech Systems, Inc. USAID/EGYPT TRADE FACILITATION PROJECT END OF PROJECT PERFORMANCE EVALUATION REPORT USAID/EGYPT TRADE FACILITATION PROJECT (TFP) END OF PROJECT PERFORMANCE EVALUATION REPORT Submitted to: USAID/EGYPT January 31, 2016 Evaluation Mechanism Number: AID-263-I-15-00001 Prepared by:  Dr. James Fox, Team Leader  Dr. Arwa Morsy Obaidallah, Local Technical Advisor  Mr. Georges Fadel, U.S.Technical Advisor  Mr. Sherine Mourad, Monitoring and Evaluation Expert Cover photo: Credit to Kemet Food. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. CONTENTS EXECUTIVE SUMMARY .................................................................................................................................1 EVALUATION METHODOLOGY & LIMITATIONS ………………………………………..…...7 1. Purpose of Evaluation ............................................................................................................................7 2. Evaluation Team Composition .............................................................................................................7 3. Data Collection Methods ......................................................................................................................7 4. Data analysis Plan ....................................................................................................................................9 5. Limitations.................................................................................................................................................9 ANALYSIS OF SURVEY RESULTS.............................................................................................................. 17 1. Findings for the Export Development Program ............................................................................ 17 2. Findings for the Energy Efficiency Program……...………………………………………. ...21 3. Findings on Gender.............................................................................................................................. 23 FINDINGS & CONCLUSIONS (QUESTIONS 1-5).............................................................................. 24 RECOMMENDATIONS (QUESTION 6)................................................................................................. 33 ANNEXES ...........................................................................................................................................................I ANNEX I: Desk Study References .........................................................................................................II ANNEX II: Stakeholder Data Collection Instrument ........................................................................ IV ANNEX III: Beneficiary Data Collection Instrument .........................................................................IX ANNEX IV: List of Stakeholders and Beneficiaries Contracted ..................................................XX ANNEX VI: Timeline for the Evaluation ..................................................................................... XXXVI ANNEX VII: TFP Evaluation Design- Questions, Sources, and Analysis ............................. XXXVII List of Tables Table 1 SME’s Distribution by Governorate and Sector……………………………..........8 Table 2 Project Related Events…...………………………………………....…...….….....12 Table 3(A) Progress Under Phase 1, According to Semiannual Reports…..…………….…..13 Table 3(B) Progress Under Phase 1, According to Semiannual Reports…….......…….….…14 Table 4 Activities and Tasks during Phase 2 of TFP……...……………........…..… ..…….15 USAID/Egypt TFP End Of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. ACRONYMS B2B Business to Business DEC Development Experience Clearinghouse DevTech DevTech Systems, Inc. ECA Egyptian Customs Authority ENCPC Egyptian National Cleaner Production Center EEP Energy Efficiency Program EEC Engineering Export Council EDP Export Development Program EDPP Export Development Pilot Project FEC Food Export Council FSU Food Safety Unit GOE Government of Egypt GOEIC General Organization for Import and Export Control GSP Generalized System for Preference HTEC Home Textile Export Council IPR Intellectual Property Rights ITDA Internal Trade Development Authority MTI Ministry of Trade and Industry MITSME Ministry of Industry, Trade and SMEs NCIS National Customs Information System QIZ Qualified Industrial Zone RMG Ready-made Garment RMGEC Ready-made Garment Export Council SOW Scope of Work SMEs Small and Medium Enterprises TFP USAID/Egypt’s Trade Facilitation Project USAID United States Agency for International Development USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 1 EXECUTIVE SUMMARY This is an end of project performance evaluation for the Trade Facilitation Project (TFP), carried out from June 1, 2011, to October 31, 2015, under contract number AID-263-C-11-00003.The TFP was awarded to Nathan Associates in the amount of $10.9 million for the period of June 1, 2011, to May 31, 2014, with an option year having a ceiling of $14.0 million. Several extensions and the exercise of the option year brought the end of project date to October 31, 2015. The development hypothesis formulated at the inception of the TFP activity1 states that assistance in trade facilitation will help improve economic development objectives in several ways. First, the reduction in barriers to trade and the creation of more efficient internal and external trading systems would help streamline and expedite the flow of goods across and within Egypt’s borders. Second, Egyptian firms could improve their competitiveness and, when coupled with improved access to export promotion activities and available trade preferential agreements, increase their exports and contribute to job and wealth creation. The project was designed to “address key constraints to improving Egypt’s trade environment while providing support to Egypt in fulfilling its commitments under the Strategic Economic Partnership.The ultimate objective of this activity is to promote economic growth and job creation and reduce poverty through improvements in the trade environment.” 2 USAID designed and launched TFP in late 2010 when Egypt’s almost 30-year-long Mubarak regime seemed likely to continue. The TFP had two expected results: External trade is made more efficient and internal trade processes are improved and strengthened. The activity design was predicated on five critical assumptions:  The Ministry of Trade and Industry (MTI) is the main counterpart and is responsible for the two expected results.  The MTI coordinates technical assistance with other TFP participating organizations.  The Government of Egypt (GOE) has ownership of the trade-related policy reform agenda.  The GOE is committed to the implementation of the tasks identified in the activity.  The GOE and the implementing partners will jointly produce a plan to ensure sustainability. Exogenous circumstances—notably the January 25, 2011, events that led to the resignation of President Mubarak—introduced a long period of uncertainty and rapid changes in project GOE counterparts, — rendering all of the original assumptions behind the project unfeasible, such that the original design could not be completed successfully. Neither USAID nor Nathan Associates bear responsibility for these outcomes, which could not reasonably have been predicted at the time of the contract award. Because the TFP contract described the initial tasks as illustrative, the contractor and the Mission had the flexibility to adjust tasks to the prevailing conditions in Egypt. USAID therefore largely shifted from assisting the GOE trade ministry on policy issues, to promoting small and medium enterprises (SMEs) 1 USAID currently uses the term “activity” for what previously were called “projects” by the Agency. In this evaluation, the term “activity” is used to refer to the TFP. 2 See Annex 5 of this report, “Evaluation Statement of Work.” USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 2 through the ministry-sponsored Export Councils in four sectors—food, engineering, home textiles and ready-made garments (RMGs). The purpose of this performance evaluation is to assist USAID/Egypt and USAID/Washington in informing decisions regarding: (a) the effectiveness of TFP’s approach to improving the export capacity of Egyptian SMEs and the sustainability of these improvements; (b) the effectiveness of policy-related technical assistance to the GOE; and (c) the nature and scope of possible future export promotion projects based on lessons learned from TFP. The audience of this report is USAID/Egypt—specifically the Office of Economic Growth—its trade and investment team, implementing partners and other international donors. An Executive Summary and recommendations will be provided to the Ministry of Industry, Trade and SMEs (MITSMEs) and shared with the general public via the Development Experience Clearinghouse (DEC). USAID will use the report to inform future assistance strategies and to share lessons learned with other stakeholders. A five-person DevTech Systems, Inc., (DevTech) evaluation team conducted the performance evaluation in Egypt between October 2, 2015, and November 12, 2015. All team members are native Arabic speakers other than the team leader, who has extensive experience in project evaluation, including four previous studies in Egypt. The team used separate questionnaires for targeted beneficiary SMEs, assisted energy efficiency firms, stakeholders, export consortium groups and project trainers. Overall, the team interviewed 31 (out of 140) SMEs, 10 (out of 28) energy efficiency firms, 15 (out of 18) stakeholders, 2 (out of 2) export consortium groups, and 6 (out of 11) project trainers. The team used both primary and secondary data sources in answering the six evaluation questions. Relevant primary data were derived from:  Interviews with 31 SMEs to evaluate the TFP Export Development Program (EDP);  Interviews with 10 energy efficiency firms to evaluate the TFP Energy Efficiency Program (EEP);  Focus groups to evaluate two 'export consortium groups (automotive spare parts and hospitality); and  A focus group and interviews with TFP advisors and trainers. Emphasizing triangulation wherever possible, the evaluation team used secondary data sources that included the TFP performance monitoring plan;semiannual progress reports; internal evaluations of training and trade show results and activity deliverables; events reports; the EEP; and, as well as the academic literature on SME exports from Egypt. Below is a summary of TFP evaluation findings and recommendations: Evaluation Question 1: To what extent were the Export Development Program and the Energy Efficiency Program objectives achieved according to the program design and the development hypothesis, at the planned scale, in a timely manner, and within budgeted costs? Why or why not? Our findings from interviews with 31 SMEs that benefited from EDP show that TFP contributed directly to increasing export sales for the targeted SMEs:With 25.8% of interviewed SMEs showing an increase in their export sales as a result of TFP assistance through the EDP. We also conclude that TFP has improved the competitiveness of targeted SMEs. Moreover, 29% of interviewed SMEs succeeded in entering new markets as a result of TFP assistance. Our conclusions on employment and employment productivity are limited by the number of responses received on these two indicators. More than one-third of SMEs (14 out of 31) interviewed for the EDP did not respond to questions on employment and employee productivity. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 3 Through the TFP implementing partner’s capacity to reach some “export-ready” firms, the activity was able to improve Egypt’s export prospects significantly, with trade shows achieving more than $19 million in direct sales and more than $51 million in anticipated future sales. Further, the promotional techniques and new business practices SMEs learned from TFP advisors in preparation for trade shows appear to have improved SMEs’ abilities to approach new buyers/importers and advanced their business communication and negotiation skills. The specific contribution of training to export promotion and employment is not clear from the data. Our evaluation used SME survey results to get respondents’ perceptions of training quality as a proxy for effectiveness, and the analysis revealed that 88% of the respondents perceived training as the strongest activity of TFP. Our findings from interviews with 10 energy efficiency firms that benefited from the EEP show that 60% reported energy saving in their organizations resulting from the EEP; the remainder stated that investments needed more time to show energy-saving results. The evaluation team concludes that the EEP is a promising mechanism to lower SME production costs and improve competitiveness, even though the team was unable to undertake a benefit-cost analysis of this aspect of the activity. This was not possible due to lack of information on costs and limited information on benefits. Overall, we regard USAID/Egypt’s redirection of TFP resources and activities toward supporting SMEs as having led to substantial positive results. In terms of budget and timeliness, we believe that the activity implementer was flexible and responsive to changing circumstances, reacting quickly to the change in activity direction. In addition, the activity came in under budget. Evaluation Question 2:What were the economic benefits and costs of the Export Development Program and the Energy Efficiency Program? The TFP did not track the allocation of its budget resources among the different activities, so data does not exist to estimate the costs of each of these two activities. Consequently, we cannot establish a cost basis for either the EDP or the EEP. Similarly, information is incomplete on the benefits of either program. Nevertheless, available sales evidence suggests that the EDP was highly effective in producing substantial benefits to the Egyptian economy that significantly exceeded activity costs. For example, the $19 million in trade show export sales and the potential $51 million in follow-on sales far exceeded the goal of the activity, which was to achieve $6 million in sales according to the performance monitoring plan. Although no reliable estimate of the cost of the energy efficiency activity is available, a former chief of party estimated it to be between $500,000 and $1 million. Overall, we see this as a promising activity, and, for which more concrete measurements of expenditures and related energy cost savings will be needed in any future activity. Evaluation Question 3:What were the factors that constrained the implementation of the trade facilitation component/policy reform measures with the GOE? What are the lessons learned and actionable recommendations with respect to this component? External circumstances—the events of 2011 and subsequent frequent changes in senior positions in counterpart agencies— were the primary obstacle to effective GOE policy reform implementation. The most actionable lesson for future projects is to continue the approach used under TFP to provide a flexible design so that activity implementation can adapt to changed circumstances. Although there was considerable success in assisting SMEs in exporting, progress on the Egyptian policy USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 4 framework was a constraining factor. The policy framework continues to be an obstacle to expanding exports, and further improvement in this area is needed.The lesson here is that progress could be made on simplifying import-export regulations, adopting risk-based management and completing a revised customs tracking system. Evaluation Question 4: How has the current implementation approach contributed to the achievement of actual results in terms of export sales, increase in employment, and institutionalization/adoption of sustainable improvements within targeted SMEs? We conclude that TFP was successful in terms of promoting exports, export development and improving the competitiveness of the SMEs. Notwithstanding this success, SMEs achieved only modest increases in employment. Findings from SME questionnaires provided before and after attending trade shows conclude that TFP training and coaching visits to SMEs in preparation for attending trade shows were effective in helping them to set their trade show objectives and targets. Trade show participation helped SMEs to answer direct questions from buyers related to price, shipment, production capacity and maximum order quantity, in addition to questions related to the product value, specifications and their competitive advantages. Further, many SMEs considered TFP’s advice about the best representatives to send to trade shows and how much orientation they should have before attending. TFP also initiated two export consortia, which enabled member SMEs to achieve good results during the TFP assistance period (mainly from trade shows attended). Yet the sustainability of these two export consortia is not guaranteed. Lastly, the evaluation team concludes that EEP is a promising mechanism to lower the production costs, despite the team’s inability to undertake a cost-benefit analysis of this aspect of the activity. Evaluation Question 5: How effective have the capacity-building services been in developing sustainable capacity in target SMEs? Why? The capacity-building services undertaken by TFP produced mixed results. Some SMEs have internalized lessons learned from engagement in activity-funded training and trade shows that they will be able to sustainably incorporate into their business practices. Other SMEs, however, did not adopt the lessons from the training, reflecting in part capacity constraints to internalizing training and businesses and/or management were insufficiently export oriented and therefore not a good fit for this activity assistance. At the other end of the spectrum, some assisted SMEs were already well-positioned in export markets, and found modest value in activity assistance. The evaluation team analyzed data available from activities related to export consortia, trade shows and training. For the two export consortia initiated by TFP, we conclude that although the initiative is very promising and should be continued, the sustainability of these export consortia is not guaranteed at present. The consortia still need assistance in forming their strategic and action plans. Trade shows produced positive results on export sales. Further, capacity building provided through TFP trade shows achieved sustainable results, especially in preparing SMEs for the export experience. However, our findings also show that SMEs still need training assistance on organizing business-to￾business (B2B) meetings in target markets as part of the trade show experience. For the EEP, the evaluation team used survey responses of the 10 interviewed SMEs that received EEP as a proxy for determining training effectiveness. Five of these SMEs utilized the energy audit visits as on- USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 5 the-job-training. “Audit visits” was at the top of the list of strongest TFP activities, with eight frequencies, followed by “training on energy efficiency awareness and measures,” with only two frequencies. Discussions of their future needs suggest that SMEs desire more training on technical topics such as the ISO 50001:2011 Energy Management System, Solar Systems Installation and Green Energy. The evaluation team employed survey responses from 31 SMEs that received EDP. About 88% of respondent SMEs perceive training as the most valuable TFP activity in which they participated. However, the evaluation team also found that some SMEs, medium enterprises, were not happy with the training content, assessing its technical content at a basic level that fell below their knowledge and experience or was irrelevant to their sector or specialties. Other SMEs, especially small enterprises, viewed the training content as very valuable given their lower knowledge and experience at the personal and organizational level. We conclude that training should be customized according to the size of enterprise, export experience and sector-specific needs. Evaluation Question 6: Based on answers for each of the above questions, what are the actionable recommendations that USAID/Egypt can adopt/implement in terms of SME export promotion activities and sustainability in future projects? Overall, we recommend that interventions undertaken by the TFP should be continued and deepened with a follow-on activity that builds on the redirection of TFP activities that now are fostering both SME growth and greater energy efficiency. As part of a new USAID activity initiative, we recommend that USAID consider adopting the following activity design and implementation measures to further increase the effectiveness of its trade capacity building and energy efficiency assistance: 1. Adopt a long-term (at least five-year) period for activity implementation. Given the tentative, resource-constrained and risk-averse nature of most small businesses, a new activity must be capable of providing sustained support for an incubation period of several years before SMEs achieve levels of self-sustaining production, marketing and investment growth. 2. Introduce more rigorous eligibility criteria for selecting SMEs to receive assistance. The Export Councils do not yet have the capacity to guide this process. More stringent criteria are needed, and the activity should consider reaching out to other Egyptian associations for additional promising SME prospects. For example, implementers of a new activity should assess SME firms more carefully with respect to their readiness and capacity to export. SMEs could be categorized by the level of their knowledge about exporting and their previous experience to identify the type and level of assistance required and to tailor training according to recognized deficiencies. Similarly, a new activity should customize the type of technical assistance services to SMEs by size, experience and sector. 3. Develop and implement a systematic and coordinated SME training plan. Training should be periodic, begin far in advance of trade shows and require firms to do their homework between training events. For example, as part of hands-on training, SME managers would be required, on their own, to research markets, seek B2B contacts and tailor product selection to markets. Subsequent training would grade their progress and warn laggards that funding for their participation might be in jeopardy. Logistical factors such as training locations and timing should be part of training plans for SMEs. 4. Set up clear standards and guidelines for trade show selection, SME eligibility for trade show USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 6 participation and trade show follow-up support. Locating companies in visible high-traffic locations at trade shows is important for improving the numbers of visitors, clients, purchase agreements and export transactions. Trade shows should also be complemented by other types of technical assistance; some SMEs recommended assisting on trade missions and trade tours improving the quality of training and trainers and supporting export consortia. 5. Assure that implementing partners undertake more rigorous selection of training staff. As mentioned above, more specialized and advanced trainings should be provided. This requires better trainers and may involve training of trainers by activity implementers. 6. Offer incentives to export consortia for bringing SMEs together. A new activity could include activity-funded advocacy, advisors and services to existing and new export consortia to incentivize membership and participation by SMEs. Such incentives could include online communities of practice, how-to marketing and export guides, mentoring of SMEs by large B2B partner/clients and jointly shared trade fair exposition space with an emphasis providing the unique type of technical and marketing services that SMEs require for competing and growing. 7. Introduce systems for more comprehensive and systematic monitoring and record keeping of activity interventions. Activities with multiple tasks need systematic and sustained performance tracking, comprehensive data collection and sustained record-keeping that generate information to assess the costs and benefits of each activity in order to evaluate their absolute and relative effectiveness. 8. Design and implement a dissemination plan for sharing information on International Best Practices. In the context of a new EEP, a follow-on activity could include a dissemination plan for training, electronic media and other mechanisms for sharing environmental best practices—e.g., clean energy and energy efficiency—that are within the context and capabilities of small-to-large Egyptian firms. Training could be conducted in the industrial zones where the firms are located, out of peak hours and at both the basic and sector/industry-specific Technical reports should be translated into Arabic for greater accessibility and use. 9. Recruit short-term specialized trainers for conducting advanced training workshops that fit SMEs with higher experience, expectations and export capacity. The evaluation findings highlight that SMEs do not share the same levels of management, marketing and production skills; training should be tailored to SME skills and experience and trainers selected accordingly. 10. Adopt phasing-in/phasing-out strategy for SME capacity building. The goal is to ensure that every SME “graduates” to self-sustaining export market participation. This requires a clear SME graduation strategy based on needs assessments that provide training and other support tailored to a firm’s export-readiness, with more advanced training available for firms with greater capabilities. The overall conclusion of the evaluation team is that this activity successfully adapted to changed conditions on the ground to achieve very positive results in exports and learning by Egyptian SMEs and broke potentially promising ground on energy efficiency. Additional USAID-supported activities focusing on these areas are clearly warranted. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 7 EVALUATION METHODOLOGY & LIMITATIONS 1. PURPOSE OF EVALUATION The purpose of this evaluation is to inform Mission decisions regarding: (a) the effectiveness of the USAID/Egypt Trade Facilitation Project’s (TFP) approach to improving the export capacity of Egyptian small and medium enterprises (SMEs) and the sustainability of these improvements; (b) the effectiveness of policy-related technical assistance to the Government of Egypt (GOE); and (c) the nature and scope of possible future export promotion projects based on lessons learned from TFP. The evaluation was conducted between October 2015 and January 2016 at a cost of $205,836. The audience of the evaluation report is USAID/Egypt—specifically the Office of Economic Growth, its Trade and Investment team, implementing partners and other international donors. An Executive Summary and recommendations will be provided to the Ministry of Industry, Trade and SMEs (MITSME) and shared with the general public via the Development Education Clearinghouse (DEC). USAID will use the report to inform future assistance strategies and to share lessons learned with other stakeholders. 2. EVALUATION TEAM COMPOSITION The DevTech evaluation team consisted of a team leader, two technical advisors and one monitoring & evaluation expert. The team was led by Dr. James Fox - a U.S. senior economist with more than 30 years of experience in international development. The technical advisors—both fluent in Arabic—are: Dr. Arwa Morsy, a senior Egyptian economic advisor with extensive experience in economic development and reform, management of development results and monitoring and evaluation; and Mr. Georges Fadel, a U.S. senior economist with the Economic Analysis and Data Services (EADS) team with a strong background in diverse sectorial data analysis and Middle East and North Africa regional expertise. The team’s monitoring and evaluation expert, Mr. Sherine Mourad Awwad, is an Egyptian evaluation specialist with more than 25 years of experience in organizational management, strategic planning, capacity building, technical auditing and monitoring and evaluation. The evaluation team was assisted by two experienced local logistics coordinators, Ms. Laila Kamal and Ms. Wessam Shaban. 3. DATA COLLECTION METHODS The evaluation team has drawn on a variety of sources to support its findings, making strong efforts to collect qualitative data. These sources included:  Document review and secondary statistical data;  Key informant interviews with beneficiaries of TFP’s export development program (EDP) and energy efficiency program (EEP);  Focus groups with beneficiaries of the export consortia initiative;  A focus group with TFP trainers and advisors;  Site visits to Export Councils (food, engineering, home textiles and ready-made garments [RMGs])  MITSME officials and Ministry-affiliated counterpart agencies;  Representatives of SMEs from each of the four sectors. Interviews included owners as well as staff. Overall, the evaluation team interviewed 31 SMEs that received assistance from TFP’s EDP (out of 90 assisted firms), 10 SMEs that received assistance through its EEP (out of 25 assisted firms) and 15 TFP stakeholders. Although the interviews provided numerous insights into the effectiveness of the activity, they did not allow any sort of cost-benefit analysis of the USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 8 EEP or a cost-effectiveness analysis of the EDP. This was the result of a lack of cost information from Nathan Associates, the activity implementer, breaking down activity costs by activity; the inaccessibility of export sales data from SMEs due to confidentiality restrictions; and the unwillingness of interviewees to provide the evaluation team with such information. The evaluation utilized a mixed-methods approach to data collection and analysis, utilizing multiple data sources to provide evidence relative to each evaluation question and drawing on these evidences to support findings through an analytic process of triangulation. In addition to interviews and focus groups with TFP beneficiaries and providers of capacity-building activities, the evaluation relied on the following data sources to respond to the evaluation questions:  The TFP performance monitoring plan;  Semiannual progress reports;  Internal evaluations of training and trade show results;  Activity deliverables, including briefing papers on international property rights (IPR) agreements, food safety website, event reports and the EEP; and  Academic literature on SME exports from Egypt. The evaluation team selected SMEs to interview and sites to visit through a purposive process that reflected the following factors: sectoral coverage; number of times assistance received; gender of the owner; duration of participation in the program; and geographic considerations to be as representative as possible. The interviews focused on perceptions of TFP capacity building, its effectiveness in improving both the sales and export capacity of SMEs and the sustainability of these improvements. For the purpose of assessing the EDP, the evaluation team interviewed 31 SME’s across six Egyptian governorates and in four sectors: engineering, food, RMG and home textiles. (See Table 1 below.) Table 1: SMEs’ Distribution by Governorate and Sector Governorate Engine ering Food RMG Textile Total by Governorate Sharqia Governorate-10th of Ramadan City 1 1 2 4 Giza Governorate-6th of October City 1 2 2 5 Alexandria Governorate 2 2 2 6 Cairo Governorate 1 1 3 2 7 Giza Governorate 1 1 2 Gharbeya Governorate – Mahallah 2 2 Qalyubia Governorate – Obour City 1 1 2 Qalyubia Governorate 1 2 3 Total by Sector 4 5 8 14 31 Evaluation team members also carried out interviews with 10 energy efficiency firms in 10th of Ramadan and 6th of October cities. The team also met with two SME focus groups comprising members of two export consortia initiated by TFP—the hospitality export consortium and the automotive spare parts USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 9 consortium—to investigate their satisfaction with the TFP assistance they received, challenges they faced and their recommendations for future USAID assistance. Further, the evaluation team arranged focus group meetings and individual interviews with the trainers, coaches and facilitators of the activity’s capacity-building activities, trade shows, direct training and coaching. The interviews and focus groups addressed their perceptions of the effectiveness of their capacity-building efforts as well as the SMEs’ internal processes and potential sustainability. Team members also conducted key informant interviews with TFP chief stakeholders at MITSME and its principal ministry-affiliated counterpart agencies at the governorate level. They include Qualified Industrial Zone (QIZ) and Generalized System for Preference (GSP) units, the Egyptian National Cleaner Production Center (ENCPC), Internal Trade Development Authority (ITDA), Food Safety Unit (FSU) and Egyptian Customs Authority (ECA).The team also interviewed stakeholders at the four Export Councils. The stakeholder interviews focused on assessing the support received from TFP, prospects for sustainability, and lessons learned. Key informant interviews were with the USAID activity manager, the two activity chiefs of party and key TFP implementers. Using the methodology described above, the evaluation team has analyzed the findings from the data collection instruments included in an annex to this report, and structured the report around the six evaluation questions specified in the evaluation scope of work (SOW). 4. DATA ANALYSIS PLAN The methodology is based on both primary and secondary data sources in answering the evaluation questions. The team used all relevant data from the interviews and focus groups to answer the questions. The team also used data obtained from secondary sources—including the TFP performance monitoring plan, semiannual progress reports, internal evaluations of training and trade show results and activity deliverables, events reports, the EEP and the academic literature on SME exports from Egypt—in answering the evaluation questions, emphasizing triangulation wherever possible. 5. LIMITATIONS As in all evaluations, compromises between what is ideal and what is possible in the methodology are often necessary,resulting in limitations that may affect the generalizability of the findings and the level of confidence in attribution of observed changes to TFP activities. These limitations might be separated into three categories: accessibility, availability and confidentiality. Accessibility. The SOW identifies data limitations in responding to the evaluation questions. Key informant interviews—proposed as the primary data collection method—are subject to limitations that include sensitivity of the questions asked (e.g., profits, employment), the respondents’ knowledge of the answers to relevant questions and the extent to which respondents’ answers might be influenced simply by the presence of the interviewer.To mitigate this concern, the evaluation team made concerted efforts to approach and interview individuals who were directly involved in theTFP activities and were fully aware of the program, usingArabic speakers whenever appropriate to encourage greater responsiveness. Availability. Often, the information sought by the team was not available to the person interviewed, despite efforts by the team to ensure that the most knowledgeable person is available. In SMEs, the CEO is often that person but even when he/she is scheduled to meet the team, business issues often mean that the CEO is replaced by a subordinate who may know less about the issues for which the team sought answers. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 10 In all cases, data analysis of the evaluation questions includes and explains in detail the percentage of responses received and not received and the conclusion of the evaluation team. For example, to evaluate SMEs’ perception of training quality (as a proxy for effectiveness), the analysis revealed that 22 of the interviewed 31 EDP participant SMEs have received training (other than trade show-related) fromTFP (71%). But only 16 SMEs responded to a question on the strongest and poorestTFP activity. With this small number of responses, caution must be practiced when generalizing results. In this case, the evaluation team uses its expert judgment and results fromTFP evaluation reports to overcome this limitation and feel comfortable with the conclusions from the findings and their related recommendations. Confidentiality. Disclosure of firm results is of particular concern to firms in Egypt, lest competitors learn of their success. The TFP implementer promised confidentiality to the TFP firms regarding export sales and data from energy audits. The evaluation team’s inability to access these data sets prevented a more granular assessment of the extent to which the export success of individual firms could be linked to the efforts of TFP. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 11 PROJECT BACKGROUND Egypt’s trade policies have long been seen as an obstacle to the country’s international competitiveness. TFP was designed in late 2010 to promote improvements in the policy environment and planned under the assumption that the institutional governing frameworks in Egypt, which were in place for nearly 30 years prior, remained throughout the life of the activity. The principal executing agency was to be the Ministry of Trade and Industry (MTI), with smaller roles played by several other organizations, including the Ministry of Finance and the Egyptian Export Councils. The activity was to allocate $10.9 million with a possible option year that could bring total funding to $14.0 million. The contract was awarded in March 2011, and TFP execution began in June 2011 and ended in October 2015, with actual expenditures of $10 million. There were two expected results from TFP: improving trade facilitation measures and strengthening domestic markets. The activity design was predicated on five critical assumptions:  The MTI is the main counterpart and is responsible for the two expected results.  The MTI coordinates technical assistance with other organizations participating in the activity.  The GOE has ownership of the trade-related policy reform agenda.  The GOE is committed to the implementation of the tasks identified in the activity.  The GOE and the implementing partners will jointly produce a plan to ensure sustainability. Between the design of the activity and its award, unrest in Egypt, culminating with the January 2011 events, led to the resignation of President Mubarak and a period of uncertainty in Egypt. This called into question all five of the basic assumptions on which TFP was based. At USAID’s request, the activity start was delayed three months in the hopes that conditions could become more favorable for TFP execution. Subsequently, the activity began with major uncertainties regarding the validity of the five critical assumptions, and further difficulties arose as events in the country continued to unfold. As Table 2 indicates, exogenous events (shown in italics) affected the execution of the activity as originally intended. The arrival of newly minted ministers and other senior officials, often with different views of what the TFP activity should do—or even if the activity was useful — complicated the efforts of the implementation team. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 12 Table 2: TFP-related Events Date Event December 2010 Deadline for submission of TFP proposals February 2011 President Mubarak resigns March 2011 Contract awarded to Nathan Associates; delayed start was requested June 2011 Contract signed; activity starts late in month September 2011 Nathan in place; implementation begins Early 2012 TFP explores cooperation with Export Councils July 2012 President Morsi takes office Early 2013 TFP begins work on energy efficiency in food processing February 2013 TFP begins pilot SME project in textiles, automotive sectors March 2013 TFP emphasizes promotion of SMEs, QIZs and GSP June–July 2013 June 30 events; Nathan expatriate staff evacuated November 2013 Nathan staff return to Egypt November 2013 TFP wind-up plan approved December 2013 TFP Modification 1 issued February 2014 Modification 2 adds incremental funding June 2014 TFP extended two months by Modification 3 July 2014 TFP extended one year by Modification 4 October 2015 TFP ends As originally designed, TFP included 12 illustrative tasks: 1. Improve import-export regulation 2. Establish a risk-based management system 3. Streamline mandatory standards 4. Finalize customs reforms 5. Improve public-private dialogue 6. Establish non-tariff database 7. Enforce intellectual property rights (IPR) at border 8. Promote distribution/retail 9. Address deceptive advertising 10. Improve food safety 11. Improve IPR enforcement in the domestic market 12. Improve environmentally sustainable production. The first seven of these tasks dealt with the trade policy environment and the other five with internal trade processes. By one year of the activity, nearly all activity in three of the seven policy environment tasks had been halted, or had not begun, and no activity was underway in three of the five internal trade process tasks. In light of changes in the public policy environment including lack of support for overall trade policy, interest in supporting SMEs, and special trade opportunities like the QIZs and GSP—USAID amended the Nathan contract to give more emphasis to these activities. The changes to TFP in March 2013 represent a shift in focus from trade policy to SME development and promotion of GSP and QIZs. Thus, the project can be seen as changing the emphasis on policy (Phase 1) to the new emphasis on SMEs (Phase 2). Tables 3 (A) and (B) provide a capsule summary of results and tasks, based on Nathan Semi-annual reports, from the onset of the project until the project shifted to Phase 2 in March 2013. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 13 Table 3(A): Progress Under Phase 1, According to Semiannual Reports Semiannual Report Dec 2011 Jun 2012 Dec 2012 Jun 2013 Dec 2013 Assessment Result 1: External Trade Made More Efficient Improve export￾import regulations Meetings held, progress made Activity halted due to GOE Waiting, no activity Training provided, no action by GOE Activity ended Some training, little result Establish risk management system Risk Management Report delivered, hope for progress Considerable work on principles, training Serious efforts, no results from GOE on decree to permit use of risk management Training provided, some lab progress but no decisions Activity ended Some training, no results to show, although the long￾awaited decree may still happen Streamline mandatory standards Uncertainty about approach, progress made Work underway haltingly Considerable effort and response to request but no action Training provided, some hope for progress Activity ended No results Finalize customs reform Work on NCIS resumed, training, equipment Considerable work, some equipment provided but no results Computer system decisions not made, so TFP efforts produced no implementati on Activity ended No results achieved Better public￾private dialogue Meetings to give importer complaints to GOE Met with business associations Worked with Export Councils on export promotion, produced trade show with good results Considerable effort, work with Export Councils, events, promotion of energy efficiency Activity ended Progress in engaging the private sector through presentations, conferences and Export Councils Non-tariff database No activity No activity No activity No activity Activity ended IPR enforcement at border No activity Initial activity, halted Progress made, but later halted by GOE No progress Activity ended No results USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 14 All five of the critical assumptions were rendered invalid by events. Nevertheless, activities in two sectors—engineering goods and home textiles—began to generate significant results by the time of the phase-out directive. Other activities, such as promotion of a risk-based import inspection system and establishing an FSU in the MTI, made significant progress in convincing Egyptian counterparts of the need for reforms. After a mid-term review of the activity, Phase 2 narrowed the agenda. Most policy activities were eliminated or cut back, and there was increased emphasis on SME development. SME activity was extended to two more areas, RMG and food processing. In addition, a pilot effort to improve energy efficiency in the food processing sector produced promising results and was extended to the textile sector. As indicated by Tables 3A and 3B, TFP began as an effort to improve the export regime in Egypt. Because of exogenous circumstances, this effort did not achieve the expected results due to conditions in the GOE agencies responsible for undertaking the needed reforms. When that became evident, TFP used the flexibility in its design to shift the focus to helping SMEs to export and to improve SMEs’ energy efficiency. In effect, TFP shifted from a policy-focused, top-down approach to increasing Egyptian exports to an SME-focused, bottom-up approach, but with both approaches ultimately focused on the TFP goal of increasing Egyptian competitiveness in world markets Table 3(B): Progress Under Phase 1, According to Semiannual Reports Semiannual Report Dec 2011 Jun 2012 Dec 2012 Jun 2013 Dec 2013 Assessment Result 2: Internal Trade Processes Improved and Strengthened Promote distribution/reta il Meetings held, strategic plan developed Not activated Initial activity, awaiting GOE action Meetings no action by GOE Activity ended No results Address deceptive advertising No activity No activity No activity Several presentatio ns by TFP, stirring interest Activity ended Some increased awareness by GOE staff Improve food safety Progress made, tasks agreed Task halted Start-and￾stop negotiations, little progress No progress on law, support for FSU, website Some support Website operational, FSU still awaits decree on role Improve IPR enforcement in domestic market Issues about merging three offices relevant in this area, training Briefed on Egypt’s obligations, began IPR training Training provided, hope for some simplificatio ns Training of staff, 100 judges, on IPR Support ended Modest awareness increase Improve environmentall y sustainable production No activity Developed action plan Discussion, looking for progress in 2013 Pilot launched in April for food￾processors Delayed by evacuation, underway in December Action underway USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 15 Despite the problems in implementing Phase 1, it is possible that some of TFP’s efforts will lead to meaningful GOE policy and procedural reforms. The implementation of a risk-based inspection system for imports awaits only a ministerial decree, as General Organization for Import and Export Control (GOEIC) is committed to it, and its staff has been trained. Similarly, the FSU in the MIT awaits only a parliamentary decree for its establishment. After a long delay, a new parliament has recently convened. Work on customs law, with training provided by the National Customs Information System (NCIS), may yet lead to improvements in Egyptian trade practices. Other possible outcomes that may have been catalyzed by the TFP could also materialize. Table 4 provides a summary of activities during the January 2014–July 2015 period. Table 4: Activities and Tasks During Phase 2 of TFP Semiannual Report Jun 2014 Dec 2014 July 2015 Result 1: External Trade Made More Efficient Improve export￾import regulations No activity No activity No activity Establish risk management system Equipment delivered, GOEIC encouraged to market testing TA to GOEIC on marketing lab tests No activity Streamline mandatory standards No activity No activity No activity Finalize customs reform No activity No activity No activity Better public-private Dialogue SME support, FSU website, EDPP workshop, EEC directories Considerable activity Considerable activity Non-tariff database No activity No activity No activity IPR enforcement at border Chapter in IPR book IPR book launch No activity Promote GSP and QIZs QIZ equipment delivered, U.S. ended GSP QIZ staff training, trade show prep for QIZ SMEs QIZ SMEs trained, went to MAGIC show. GSP renewed too late for activity Promote SMEs Began expansion of SME promotion to two more sectors Substantial support to export￾ready SMEs through training, trade shows Substantially increased promotion of SMEs through training, coaching and trade shows USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 16 Result 2: Internal Trade Processes Improved and Strengthened Promote distribution/retail No activity No activity No activity Address deceptive advertising No activity No activity No activity Improve food safety FSU website Continued work with FSU FSU website Improve IPR enforcement in domestic market IPR book issued Same Same Improve environmentally sustainable production nine firms received audits on energy efficiency Three energy audits, visits to 41 firms 16 companies received energy audits USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 17 ANALYSIS OF SURVEY RESULTS 1. FINDINGS FOR THE EXPORT DEVELOPMENT PROGRAM 1.1 Findings from Stakeholders 1.1.1 Findings from Export Councils The evaluation team conducted separate interviews with each of the four Export Councils that partnered with TFP: home textiles, RMG, food and engineering. All four Export Councils have received TFP training and trade show and website review assistance. The RMG Export Council has participated in a trade mission to New York. The overall technical assistance received by the Export Councils was perceived positively in general. Of the four, two Councils rated the overall assistance as “extremely good,” one Council rated it as “good,” and the fourth rated it as “neutral.” Trade shows and their relevant preparatory trainings/orientations were perceived by all Export Councils as activities thatTFP performed particularly well. The trade shows gave participating SMEs the opportunity to travel and gain exposure to buyers, new markets and competitors. Some of the training programs were valued highly, including product development and marketing. However, the Councils believed that some training programs were conducted in a very traditional manner, rather than using interactive and hands-on experiences. The director of one Export Council indicated thatTFP-supported training was unnecessary, as she believed that their members’ firms had lengthy experience and were more knowledgeable than the training content reflected.The content was described by this director as very basic in some cases. Two Councils regarded the export consortium as a very good idea, but felt it did not deliver sustainable results mainly because of time limitations. (Further specifics on export consortia are found under Question 6.) The Souk events carried out in March andApril 2015 were perceived as an excellent idea, as they provided member SMEs the opportunity to connect/collaborate and exchange ideas with other beneficiaries, as well as with export service providers. The overall context after the various political events that occurred since January 2011 in Egypt—as well as the processes used by the government that did not allow exporters the opportunity to provide feedback on export procedures and regulations—were seen as factors that limited trade facilitation reform by the GOE, and moreover, affected exporting SMEs. In the future, all Export Councils, with the exception of the Food Export Council (FEC), plan to advocate for changes in trade policy,strategy and/or procedures. Export Councils also plan to work with member SMEs to learn about the problems in their sector and to raise their voices about their demands in the markets. Further, Export Councils plan to help Egyptian exporting companies better understand theAfrican markets and the export procedures requested. The percentage of female participation in different capacity-building activities by the Export Councils was estimated at 30% by two of them;the other two Councils were unable to respond because of lack of recordkeeping on female participation. All Export Councils expressed the need for future technical and capacity-building assistance from USAID to improve trade-related matters, as well as to improve public-private dialogue,risk-based management, food safety organization, customs reform, financial support, trade missions, in-depth market analysis, SMEs’ website development and on-the-job practical training (vocational, leadership, teamwork, young designer program, quality standards and certification and entrepreneurship).The Export Councils seek a capacity-building component for their work as well as for relevant GOE organizations in any successor USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 18 toTFP.They also believe that future training should be provided by trainers with hands-on experience and with very specialized knowledge of exporting and international markets. One Export Council suggested that the export consortium mechanism could be done better by using a company that specializes in gathering the different products of Egyptian companies participating in the consortium; standardizing packaging of the different products; and assuring quality by applying unified systems, measures and standards defined in international quality references. 1.1.2 - Findings from Government Organizations The evaluation team met with 10 GOE organizations (see Annex IV for a complete list) that received assistance related to export promotion, including training, market awareness, specific topics (such as tariffs and food markets), software orientation, IPR and new/existing website development (E-learning). The overall rate of satisfaction rating for TFP assistance received by GOE organizations ranges between good and neutral. The chart below illustrates the distribution of responses from GOE of the overall rate of the assistance received from TFP. Six out of the ten GOE organizations were satisfied withTFP work on SMEs because, according to GOEs, it exposed these SMEs to trade shows and new markets and buyers and increased their ability to travel and meet international competitors. Moreover, when asked whatTFP does particularly well, 5 out of 10 GOE organizations praised theTFP consultants for being very collaborative and responsive. The most common complaint aboutTFP was that its objectives were overly broad with too little time provided to accomplish them. Some GOE organizations also complained about not being updated on TFP’s shift in approach from Phase 1 to Phase 2. Below are some of the main complaints that the team gathered from the interviews:  Low quality of training courses. Four of the ten GOE officials complained that the courses offered were weak and basic and were not the ones most needed by the SMEs. Additionally some of the courses were already available through variety of other sources.  Lack of availability of export figures pre- and post-TFP assistance. TFP did not report on actual exports after SMEs received trainings. GOE counterparts stressed the importance of receiving data and reports on the impact of technical assistance.  Low quality of consultants. Two GOE organizations did not receive the consultants’ CVs before they were hired, or even the SOW beforehand for review.They were not fully satisfied by the quality of the reports shared by these consultants. In addition to the above complaints, one GOE organization did not receive one of the servers they needed for unknown reasons. Furthermore, one GOE organization did not appreciateTFP working USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 19 directly with the private sector and thought communication with the SMEs should have gone through them.As a result, that organization did not know the level of support thatTFP provided. Lastly, TFP reports were not shared with two GOE organizations, including documents related to trade shows and the final reports ofTFP and its related activities. We conclude thatTFP had too little communication with some of the government organizations and did not build as much of a strong partnership with those organizations/stakeholders in Phase 2 of the activity as in Phase 1. Moreover, much of the unhappiness by some of the GOE officials who complained about TFP reflects their unwillingness to become involved with the SME export activity. For future capacity-building assistance, GOE organizations asked for more equipment;risk management training on other commodities such as food; courses on the U.S.risk-based inspection system; in-depth market analysis on subsectors that are competitive in the U.S. market (such as textiles, leather, and furniture); and more training on the technical aspect of IPR. 1.2. Findings from SMEs According to interviews with SMEs, 50% of the respondents reported that TFP did particularly well in training courses, specifically when it came to training on pricing, trade maps, packaging and marketing. About 35% of the SMEs said that TFP did particularly well on trade shows—particularly on de-briefing, assisting the SMEs in showcasing their products and helping SMEs to network during trade shows. Also, 17% of SMEs stated that TFP staff was very passionate about TFP’s goal and its collaborative approach. Finally, only 10% of SMEs said they were impressed by the quality of the consultants. Survey questions on what went well with TFP in terms of assistance were open-ended so as to allow the respondents to reply with more than one answer. Thus answers do not sum to 100%. Most SMEs raised concerns about the short duration of the EDP, which did not allow them enough time to fully benefit from different activities to improve their companies’ exports. Some complained that TFP assistance was not tailored to their firm size, export experience, sector or specialty. Ten percent of SMEs also complained about their location at the trade show, blaming it on lack of proper preparation by TFP. Locating companies in high-visibility places at trade shows is necessary to improve the number of visitors, clients, deals and exports. Our findings from the interviews with SMEs assisted through the EDP also show that: 1.Of the SMEs interviewed by the evaluation team, 48.4% reported increases in export sales over the past two years, 45% did not report an increase in exports and 6.6% did USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 20 not know if their exports have changed over the past two years. 3 2.Of those SMEs that reported increases in export sales, 40% increased by 5–15%, 26.7% increased by 15–30%, 20% increased by more than 30%, 6.6% increased by 5% or less and 6.6% did not know if their exports have increased. 3.Among those companies that did not see an increase in export sales, 32.3% have introduced new products over the past two years and have good potential for exports.4 4.Of the total number of SMEs that reported increases in export sales over the past two years, 53.3% reported that this increase came as a result of TFP technical assistance, either from training, trade shows or support in marketing. 5 5.The results also indicate that 25.8% of interviewed SMEs showed increased export sales as a result of TFP assistance. 6 6.Of 31 SMEs interviewed, 14 reported entering new markets over the past two years; 78.6% of these SMEs reported that this was as a result of TFP assistance. 7.Of 31 SMEs interviewed, 14 reported new buyers over the past two years. Seven out of these SMEs (58.3%) reported that this was a result of TFP assistance. 8.Of the SMEs interviewed, 54.8% reported seeing increased sales over the past two years. 7 Of these, 23.5% reported increases in sales of 5– 15%, 17.7% reported increases of 15–30%, 11.8% reported increases of more than 30%, 23.5% reported increases of less than 5% and 23.5% did not know the percentage of increases. 3 15 out of 31 SMEs interviewed to evaluate the EDP reported increases in export sales, 14 out of 31 did not notice increases in export sales and 2 did not know. 4 10 out of 31 interviewed SMEs to evaluate the 'Export Development Program (EDP)' 5 8 of the 15 SMEs that showed increases in export sales over the past two years. 6 8 of 31 SMEs interviewed to evaluate EDP. 7 17 of 31 SMEs interviewed. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 21 9.Of the 17 SMEs (54.8%) that reported an increase in sales, 10 SMEs (32.2%) have increased their sales because of TFP assistance and 7 SMEs (22.6%) have increased their sales not because of TFP assistance. 10. Of the 10 SMEs that reported sales increases as a result of TFP assistance, six reported that this was a result of TFP assistance. Three SMEs also reported increases in employee productivity. Our results on employment and productivity are limited by the number of responses received. About one-third of firms did not respond to these two questions, either through lack of knowledge or unwillingness to share information. 2. FINDINGS FOR THE ENERGY EFFICIENCY PROGRAM The evaluation team interviewed 10 SMEs in the food and textile industries that received technical assistance from TFP through the EEP. All received assistance from TFP in the form of (a) audit reports that include recommendations on possible energy-efficiency measures, (b) classroom training or (b) on￾the-job training during audit visits. The responses reveal that six SMEs perceive EEP assistance and work as “very good,” while four perceive it as “good.” “Audit visit” appears at the top of the list of best types of assistance received (with eight frequencies), followed by “establishing the concept of and guidance to energy efficiency” (with four frequencies).“Training on energy efficiency awareness and measures” came last on the list with two frequencies. Three of the 10 SMEs cited limitations to the assistance, respectively mentioning “absence of follow-up procedures,” “remote training site,” and “unfriendly English-language technical report.” which this evaluation assesses as non-critical issues. The government agency ENCPC perceived TFP assistance as “very good,” especially with regard to the training programs and awareness sessions that they have participated in that were delivered by TFP international experts in collaboration with ENCPC’s local staff members. With regard to energy-saving measures, the evaluation team found one case where the firm stated that the cost of applying the recommendations was higher than the expected savings from the measures. The vast majority of the energy efficiency works done by the firms were no- or low-cost activities. Some EEP participants found that low- to medium-cost investment recommendations for energy efficiency measures were too basic, and they already knew about them prior to TFP EEP assistance. However, the recommendations on medium- and high-cost investment projects were considered very useful, but the Cost category Investment cost level No cost 0 EGP Low cost 1 EGP – 10,000 EGP Medium cost 10,001 EGP – 50,000 EGP High cost 50,001 EGP and above USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 22 projects could not be implemented due to lack of funding. Some of the no-cost investment projects were related to raising awareness for SMEs’ employees on basic energy efficiency measures, such as limiting the use of air conditioning, reducing the consumption of running water and installing energy-efficient light bulbs. Some of the low-cost investment projects were related to isolation and fixing the boiler to reduce the consumption of natural gas. In addition to installing new lines, recommendations included changing control meters for some machines and replacing old equipment. Overall, 7 of the 10 interviewed SMEs have reported energy saving in their organizations as a result of the EEP. The rest of the SMEs stated that investments needed more time to show results. None of the SMEs mentioned savings in solar or in carbon emissions because they did not implement any recommendations on these EEP activities. However, savings were reported (unverified by the team) in natural gas (21%), electricity (14%) and water (12%). One SME mentioned savings of 70% in the consumption of diesel by wrapping the thermal pipes to isolate the boilers. Based on interviews conducted by the evaluation team with the 10 SMEs, 4 have reported savings resulted from having clean production, 3 saved by cutting production costs, 2 said that saving provided more opportunities to export, and 1 offered competitive prices. One of the SMEs was able to offer competitive prices that better positioned their products in the market place and lead to an increase in exports. Four of the 10 interviewed SMEs plan to use the money saved to invest in medium- to high-cost investment projects, such as buying energy-friendly machines or solar heating systems. One SME is using the savings to cover the money lost due to current economic and political circumstances in the country. Three SMEs are not planning to use the saved amount in any other areas of investment, and the remaining two were not willing to share the answer to this question. The evaluation team received responses on major barriers to implementation of energy efficiency measures from 80% of the SMEs interviewed. Half of the SMEs stated that the high cost of the energy efficiency implementation was the largest barrier. Moreover, one SME wanted more assurance that investment will actually lead to sustainable savings. The evaluation team also asked SMEs about their awareness of national programs on energy efficiency and international best practices. Only two SMEs were familiar with the existence of energy efficiency promotion programs in Egypt, such as clean energy loans for solar energy systems and the Electricity Reward System,which might be an indicator of SMEs’ need for such information. Meanwhile, none of the interviewed SMEs are aware of the international best practices to promote energy efficiency in their relevant sectors.Thus, the knowledge and awareness of relevant sources and resources that address energy efficiency remain core needs for SMEs. Finally, SMEs were asked an open-ended question about their recommendations in order to encourage them to provide a full, meaningful answer using their own knowledge.The following—ordered by frequency of responses—are suggestions by SME EEP participants for future program improvements:  Follow up after the audit (3)  Replicate on larger scale (other factories) (2)  Increase capacity building activities (2)  Work with manufacturing sector (1)  Provide more sector-oriented training (1) USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 23  Use closer training sites (1)  Provide more focus on solar energy (1) SMEs cited several future needs concerning EEP.The needs reveal that orientation on international best practices and financing for new machines are both main concerns for SMEs as they were both mentioned twice out of five responses. Training on the ISO 50001:2011 Energy Management System, training on Solar Systems Installation and research studies and conferences on Green Energy were mentioned one time each. The evaluation team concludes that EEP addressed a significant need to lower the cost of production for SMEs. This evaluation also recommends continuing to collaborate with the ENCPC. 3. FINDINGS ON GENDER The project had explicit goals for gender:  20% of assisted SMEs should be women-owned or women-managed.  30% of trainees should be female. The second objective was achieved throughout the life of the project and the first objective was achieved in the final year of the project. Overall, 18.6% of the assisted firms were owned or managed by women.The results for training and workshops also fell slightly short of the target, with 29.5% of women trained.The only skewed result was IPR training for judges, which included 64 men and only 1 woman. Excluding this training, for which participants were beyond project control, the 30% target was exceeded. Overall, we consider this result satisfactory, as the short time available for project execution, and the necessary reliance on the Export Councils, made it difficult to target women more effectively. More generally, we do not see major obstacles to the economic role of women in Egypt.The presence of women among the stakeholders we interviewed was dominant.All four of the Export Councils assisted byTFP were headed by women, and the majority of other stakeholders in the government and other organizations were women. Given the centrality of women in these coordinating and policy-implementing positions, we see no reason to expect that government and the Export Councils will be an obstacle to the advancement of women in the economy. In summary, we do not believe that Egypt offers major obstacles to women-owned enterprises present in some of the other \countries in the region. As the head of the EgyptianWomen’s BusinessAssociation commented, women-owned firms tend to be newer players in the economy, so they tend to have less capital than male-owned businesses and find more difficulty in acquiring financing. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 24 FINDINGS & CONCLUSIONS (QUESTIONS 1-5) Question 1: To what extent were the Export Development Program and the Energy Efficiency Program objectives achieved according to the program design and the development hypothesis, at the planned scale, in a timely manner, and within budgeted costs? Why or why not? TFP was mainly designed to address important trade policy challenges in Egypt, improve trade policy environment and expand trade through the provision of technical assistance, training, coaching and limited IT solutions. The development hypothesis formulated at the inception of the activity states that assistance in trade facilitation will help improve economic development objectives in several ways. First, the reduction in barriers to trade, and the creation of more efficient internal and external trading systems, would help streamline and expedite the flow of goods across and within Egypt’s borders. Second, Egyptian firms could also improve their competitiveness and, when coupled with improved access to export promotion activities and available trade preferential agreements, could increase their exports and contribute to job and wealth creation. This hypothesis constituted the base for the design of the second phase of TFP when the first phase of the activity did not achieve its outcomes due to the country’s shifting political circumstances. The second phase of TFP introduced the EDP and the EEP with two illustrative tasks: (a) Improve awareness and use of special export opportunities and preference programs available to Egyptian exporters and (b) Support government and private sector efforts to develop successful SME exporters. The EDP included various types of technical assistance—training, mentoring, coaching and follow-up—on a number of export-related subjects in order to help targeted SMEs improve their knowledge, readiness and capacity to export. EDP targeted SMEs in the home textile sector through the Home Textile Export Council (HTEC) and the automotive and metal form sector through the EEP. The EDP also partnered with two additional Export Councils, the FEC for food and the RMGEC for ready-made garments, and some of their SMEs. For its technical and capacity building assistances, TFP targeted exporting SMEs or those that have high potentiality for export. Our findings from the interviews with SMEs that participated in EDP show that TFP achieved success in increasing export sales for the targeted SME. Our findings, presented above, from the interviews of 31 SMEs that received assistance through EDP indicate that, of the total number of SMEs that reported increases in export sales over the past two years, 53.3% reported that this increase came as a result of TFP technical assistance, either from training, trade shows or marketing support. 8We conclude that 25.8% of total number of interviewed SMEs showed increased their export sales as a result of TFP assistance through the EDP. 9 We also conclude that TFP has improved the competitiveness of targeted SMEs to a reasonable extent. Our findings indicated that 29% of all total interviewed SMEs successfully entered new markets as a result of TFP assistance, 10 and 22.6% of them had new buyers over the past two years as a result of TFP 8 8 out the 15 SMEs that showed increase in export sales over the past two years. 9 8 out of 31 SMEs interviewed to evaluate EDP. 10 14 of total 31 SMEs interviewed reported that they entered new markets over the past two years. 78.6% of these SMEs reported that this was as a result of TFP assistance. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 25 assistance.11 Further, 32% successfully increased sales over the past two years as a result of TFP assistance. 12 For employment and employee productivity, the impact was less than that observed for exports, sales, new markets and new buyers. Our findings indicate that 19.4% of SMEs interviewed for the EDP had new employment over the past two years as a result of increases in sales arising from TFP assistance. 13 The impact on employee productivity is very modest. Only three companies reported improvement in employee productivity as a result of TFP assistance (9.7% of SMEs interviewed). Our conclusions on employment and employment productivity are limited by the number of responses received for these two indicators. About one-third of SMEs interviewed for the EDP did not respond to questions on employment and employee productivity. Data on the contribution of specific EDP activities to competitiveness, export promotion and job and wealth creation are not available for a number of the activities (like training, export consortia, assistance in marketing and IT). Export sales figures are only available for the trade show activity, which targeted most of the SMEs assisted by TFP. SMEs reported initial sales of more than $17 million and projected sales of $50 million in two years of TFP assistance. 14 TFP performance monitoring plan reports did not provide a baseline for SMEs’ export sales in years before 2011. Our discussion with SMEs indicated that for many of them, trade shows had enabled them to keep their sales levels either stable or to improve them slightly above 2011 and earlier years. For them this is a significant achievement, especially for the period of Egypt’s political unrest. Further, the promotional techniques at trade shows/missions and new business practices SMEs learned from TFP advisors appear to have improved SMEs’ abilities to approach new buyers/importers and advanced their business communication and negotiation skills. Assistance provided for the two export consortia was mainly in the form of trade shows and preparation trainings. Our discussions with consortia member SMEs in two focus groups with the two export consortia initiated by TFP, indicate that member SMEs were satisfied with the results of the trade shows attended. The specific contribution of training to export promotion and employment is not clear from the data. However, our evaluation used SMEs’ perception of training quality as a proxy for effectiveness. The analysis revealed that 22 out of the 31 EDP participant SMEs have received training (other than trade show-related) from TFP (71%). Only 16 SMEs responded to a question on the strongest and poorest TFP activity. Detailed analysis finds that 14 out of the 16 respondent SMEs perceive training received as the strongest activity of TFP (88%). However, 3 of the 14 have some concerns about the training, citing old￾fashioned training delivery by lecturing, absence of English terminology and holding the training event in a five-star hotel. Only 2 of the 16 respondent SMEs perceive training received as the poorest activity of TFP (12%). The EEP was introduced in 2013. As energy prices have risen and are expected to continue to do so the value of energy efficiency became more important to participants. 11 12 of the total 31 SMEs interviewed reported new buyers over the past two years. 58.3% of these SMEs reported that this was as a result of TFP assistance. 12 54.8 % of the SMEs have seen increases in sales (17 out of 31 SMEs interviewed); 58.8% of those companies that reported increases in sales over the past two years indicated that this increase resulted from TFP assistance (10 of the 17 SMEs). Of the 31 total SMEs interviewed, 10 increased sales as a result of TFP assistance through EDP (32%); 17.7% indicated that the increase was not from TFP, and the remainder did not respond to this question. 13 6 of 31 SMEs interviewed for the EDP program. 14 This figure does not include number of trade shows. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 26 Overall, 60% of SMEs have reported energy saving in their organizations as a result of the EEP; the remainder of the SMEs stated that investments needed more time to show savings results. None of the SMEs mentioned savings in solar or in carbon emissions. However, savings were reported by the activity implementers in natural gas (21%), electricity (14%) and water (12%), although no estimates of savings in dollars or Egyptian pounds were provided. The evaluation team concludes that EEP is a promising mechanism to lower the cost of production. This is despite the inability of the team to undertake a benefit-cost analysis of this aspect of the activity. This was not possible for two reasons: lack of information on costs and limited information on benefits. The team has queried Nathan Associates several times for activity costs by activity. In order to provide useful information to track energy savings it is recommended that in future activities should reduce the confidentiality requirement by including monthly reports on usage by assisted companies, along with prices actually paid for these services, followed up by a repeat visit, six months after any energy audit, to confirm the value of energy or water savings both in quantity and in cost. Aggregated, to preserve confidentiality, the latter result may be of considerable value in advocating to other firms in Egypt the potential for cost reduction from addressing energy issues. Nevertheless, despite only modest results so far, we recommend further work in this area. New activities should include follow-up visits, perhaps six months after energy audits to track actual energy savings, and the information-dissemination activities discussed in Recommendation 8. Overall, we regard USAID/Egypt’s redirection of TFP resources and activities toward supporting SMEs as one that led to substantial positive results. In terms of budget and timeliness, we believe that the TFP implementer was very flexible and responsive to changing circumstances, reacting quickly to the change in direction of the activity. The activity came in under budget, of the initial budgeted $14 million in project expenditures, $10.9 million was obligated, and about $10.5 million was expended, based on data available to the evaluation team. The project’s slow start, due to exogenous circumstances, as well as activity disruptions during implementation, slowed expenditures. Question 2: What were the economic benefits and costs of the Export Development Program and the Energy Efficiency Program? The TFP did not allocate its budget resources among the different activities, so it is not possible to make estimates of the costs of each of these two activities. The implementer has provided a breakdown of expenditures, but this provides little help in allocating costs among the various TFP activities. Consequently, we cannot establish a cost basis for either the EDP or the EEP. Similarly, we have very incomplete information on the benefits of either program. Some information was gathered on each activity, but confidentiality considerations made them unavailable to the evaluation team. Nevertheless, available evidence suggests that the EDP was highly effective in producing substantial benefits to the Egyptian economy that significantly exceed activity costs. In particular, the $19 million in trade show exports and the potential $51 million in follow-on sales far exceeded TFP targets. We also regard these estimates as probable understatements, as some firms failed to meet their commitments to disclose their sales to TFP staff, and exports not related to trade shows were not reported. Of course, export sales do not represent the complete economic benefits to Egypt of the export activity. Firms do incur substantial production costs in order to have products to sell. Calculating the economic benefit to Egypt of $1 of export earnings from an individual firm would be a Herculean undertaking, requiring details on the costs involved in production and raising conceptual problems as well. Is the benefit only the additional profit from the sale, or is there some benefit to having hired workers to produce it? Moreover, the benefits from exporting typically differ from production for the domestic USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 27 market, as the firm encounters much greater competition and needs continually to adapt to changing conditions in international markets. This requires firms to research markets and innovate. When asking, “is this a cost or a benefit of exporting?” the answer is likely to be both, as exporting firms have to be more productive in order to compete successfully in international markets. A careful study of small Egyptian carpet producers (Atkin, Khandelwal and Osman, 2015) carried out over several years showed that exporting firms learned to produce higher-quality products, were more productive and were generally more efficient than control firms that produced only for the domestic market. The energy efficiency activity is more problematic. Although no reliable estimate of the cost of this activity is available, a former chief of party estimated it to range between $500,000 and $1 million. Based on interviews with the 10 recipients of energy efficiency assistance, we regard the benefits so far as lower than those hoped for by the energy auditors and so far are unlikely to match the cost of the activity (based on very fragmentary data). Most firms interviewed implemented only the no-cost and low-cost recommendations. Still, some changes in energy use may be forthcoming. Three firms indicated that they were considering larger investments but had so far been constrained by financial limitations. Overall, we see this as a promising activity for which more concrete measurements would be needed in any future activity.This would include providing logs of monthly energy use over the period of implementation of the activity to compare with pre-activity usage. Confidentiality will be an issue in obtaining energy cost savings data in any future project: Many firms are likely to be unwilling to disclose information on exports or energy savings except to activity staff with whom a substantial degree of trust has been established. Consequently, assessment of the effectiveness of future assistance will require a system where aggregated results can be provided to monitoring and evaluation staff without disclosing the results for individual firms in order to maintain confidentiality. Data in such aggregated form would preserve confidentiality, but it would limit the ability of evaluators to assess progress at the firm level. Question 3:What were the factors that constrained the implementation of the trade facilitation component/policy reform measures with the GOE? What are the lessons learned and actionable recommendations with respect to this component? None of the actions undertaken by the activity appear to have been an obstacle to effective GOE policy reform implementation. Rather, it was external circumstances, particularly the 2011 revolution, with its frequent changes in senior positions in counterpart agencies. The MIT’s name was changed twice, and it was led by a succession of six different ministers between the activity design and its completion. The pace of the initial set of tasks for the activity (shown above in Table 2) is based on TFP semiannual reports. Clearly, very little was achieved. The early days of the activity are murky, as most of the GOE officials from that period have long since departed. But we see no evidence that the TFP implementer (a USAID contractor) failed to attempt to carry out the activity goals effectively and efficiently. We conclude that external circumstances prevented implementation of the planned activities. The most actionable recommendation for future projects is what was characteristic of TFP: Provide a flexible design so that activity implementation can adapt to change circumstances—even abrupt ones—as did TFP. Selection of the implementer is also critical. A flexible design will not work unless the implementing organization is also willing to be flexible, changing staff as needed to respond to challenges and able to access a wide range of technical specialists even in areas not originally contemplated in the design. The trade policy reforms proposed in the initial design proved overly ambitious for conditions USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 28 during implementation, but might be achievable under more favorable conditions. Policy is important, but TFP demonstrates that exports can grow even within the limitations of the current policy framework. Question 4: How has the current implementation approach contributed to the achievement of actual results in terms of export sales, increase in employment and institutionalization/adoption of sustainable improvements within targeted SMEs? TFP performance monitoring plan reports show that exports by assisted SMEs exceeded the export targets by a wide margin for the trade show activity. This finding is not clear for other types of technical assistance provided by TFP, as the activity did not track their exports in performance monitoring plans. As noted earlier, TFP (constrained by firms’ insistence on confidentiality) was unable to keep track of total value of new export sales realized by SMEs as a result of all TFP export development assistance. Activities like training in export-related topics, study tours and missions are not tracked for their contribution to progress in export sales. The reported figures in the TFP performance monitoring plan only indicate reported sales achieved from trade shows. Actual export sales achieved as a result of trade shows far exceed the targeted sales, indicating the importance of trade shows to export development— a finding that was also verified from our interviews with SMEs. Yet, trade shows are not the only approach to increase exports. Some SMEs also recommended seeking other areas of intervention, like assistance on trade missions, trade tours, improving the quality of training and trainers and assisting export consortia. Despite the lack of actual export values for all types of TFP assistance, our findings from the interviews, described above,show that more than one-quarter of interviewed SMEs had increased export sales as a result of TFP assistance. Our findings also show that TFP has improved the competitiveness of targeted SMEs. About one-third of the SMEs interviewed succeeded in entering new markets as a result of TFP assistance. Moreover, 22.6% of SMEs interviewed (7 of 31) also had new buyers over the past two years as a result of TFP EDP. This evidence is also supported by TFP monitoring reports after the trade shows, highlighting that the promotional techniques at trade shows/missions and new business practices SMEs learned appear to have improved their abilities to approach clients and initiate business communication and negotiations. SMEs also benefited from advice on more suitable samples, on reported sales information and on becoming more aware of their competition. Moreover, SMEs’ capabilities also have been developed in approaching new markets and exploring new business opportunities in these markets in ways that can have a positive effect on their export sales in the medium and longer term. Findings on TFP contributions to employment and employee productivity are modest: 19.4% of total SMEs interviewed for the EDP had new employment over the past two years as a result of increases in sales resulting from TFP assistance. 15 Only three companies reported improvement in employee productivity as a result of TFP assistance (only 9.7% of total SMEs interviewed). 16 Modest results were achieved in employment. We support this with findings from TFP questionnaires to SMEs before and after attending the trade shows which conclude that TFP training and coaching visits to SMEs in preparation for attending the trade show were an effective way to help them to set their trade show objectives and targets. This activity also helped SMEs to directly answer direct questions from buyers related to price, shipment, production capacity and maximum order quantity as well as questions related to the product value, specifications and their competitive advantages. Further, the awareness of 15 6 out of 31 SMEs interviewed for the EDP. 16 Our conclusions on employment and employment productivity are limited by the number of responses we got on these two indicators. About one-third of SMEs interviewed for the EDP program didn't respond to questions on employment and employee productivity. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 29 who should represent the SME at the trade show and how much orientation he/she should have before attending the trade show is considered by many SMEs as a significant contribution by TFP. SMEs more experienced in attending trade shows highlighted their need for more organized business-to￾business (B2B) opportunities and for the organization of trade show activities that promote their participation in the trade show and the products they display. Other results from TFP evaluation questionnaires show that many SMEs have become more aware of a number of trade‐related concepts concerning export readiness and ability to compete in the international market. They have also gained awareness of concepts related to market research trade map procedures and export development planning as a result of TFP capacity-building activities. TFP also initiated two export consortia, which enabled member SMEs to achieve good results during the TFP assistance period (mainly from trade shows attended). Yet, as detailed below in question 5, the sustainability of these two export consortia is not guaranteed. Although members of the consortia are willing to continue, and they share general goals for future international activities, it is clear that members of the two export consortia did not receive assistance to formulate a clear strategy and action plan for their work. There are no written plans/guidelines on how each of the export consortia can improve its profitability through various types of strategies and joint activities. The results on EEP are based on our interviews with 10 SMEs that received this assistance. Sixty percent of SMEs have reported energy saving in their organizations as a result of the EEP (mainly with zero- and low-cost investments). The remainder of the SMEs stated that investments needed more time to show savings results. None of the SMEs mentioned savings in solar or in carbon emissions. However, savings were reported in natural gas (21%) by three SMEs, electricity (14%) by three SMEs, and water (12%) by two SMEs. One SME mentioned savings of 70% in the consumption of diesel. The evaluation team concludes that EEP is a promising mechanism to lower the cost of production. This is despite the inability of the team to undertake a cost-benefit analysis of this aspect of the activity. Question 5: How effective have the capacity-building services been in developing sustainable capacity in target SMEs? Why? TFP capacity-building services manifest mixed results. Export Councils have generally improved their capacity to serve their members, with or without future USAID assistance. Some SMEs have internalized lessons learned from engagement in TFP-funded training and trade show visits that they will be able to apply in a sustainable way in the future. Others have failed to do so, due to lack of capacity by the SME to internalize training, or because firms were not good candidates for TFP assistance. At the other end of the spectrum, some assisted SMEs were already well-positioned in export markets, and found assistance of only modest value. The analysis below explains in detail how effective the capacity-building services have been in developing sustainable capacity in target SMEs. EXPORT CONSORTIA In partnership with UNIDO, TFP approached the export consortia initiative with the aim of promoting collaborative efforts among SMEs to enter specific export markets. In coordination with their relevant Export Councils, TFP helped to establish two export consortia; one for the hospitality sector and other for the automotive parts sector. The selection of consortia member SMEs was guided by the relevant Councils, and TFP provided members with marketing and promotion training and coaching as well as USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 30 support to participate in two trade shows; one in Saudi Arabia for the hospitality group, and the other in Kenya for the automotive parts manufacturers group. Our findings from two focus groups with member SMEs of the export consortia show that they seem to realize the importance of collective efforts to target potential markets. Initial efforts that were supported by the TFP focused on providing members with capacity-building activities, more than enabling them to work together as a team and formulate their export strategies. Member SMEs are, in general, homogenous in terms of sector and foreign market competencies. Their similar profile and interests provide favorable conditions for developing shared consortium strategies. The number and composition of members of each of the export consortia is different. The hospitality export consortia currently have a small number of SME members and seem to be unwilling to add more members to the group, as they find it difficult to manage and coordinate a larger number. The automotive parts consortium has a larger number of SME members; although some of them provide substitute products, they seem willing to cooperate with each other toward achieving a shared vision for exporting. The sustainability of these two export consortia is not guaranteed however. Although members of the two export consortia are willing to continue, and they share general goals for future international activities, it is clear that members did not receive assistance to formulate a clear strategy and action plan for their work. Although good ideas have been suggested, there isn't yet a clear strategy on how members of the two export consortia can pool their resources, for example, for the joint acquisition of equipment, supplies and services to promote exports among members of the group. Another factor that affects the sustainability of the two export consortia is the lack of effective organization structures for each one of them. Further to the lack of clear activities that need to be carried out at the consortium level, there is no structure for the role of members of the consortia. Even the coordinator role that they have assigned appears to be ineffective in both export consortia. Effective communications between members of the firms is not demonstrated. The two export consortia still need assistance to bring members of each together and help them form strategic objectives and develop action plans for their activities. This was not carried out by TFP, partially due to the short time of implementation, but needs to done in future. The number of SMEs members in both export consortia is still small. Export consortia still need a mechanism to attract more members who provide complementary products. Member SMEs in both export consortia asked for assessments of potential markets in the sector; trainings and capacity building on export related matters, leadership and teamwork; assistance to carry out business to business meetings; and facilitation to attend more trade shows for members. Members of both consortia suggested to the evaluation team that a business plan would be a useful way to move forward in the future TRADE SHOWS Our results from the interviews show that trade show assistance was largely received by most SMEs. TFP trade show assistance involved in most cases preparation training and coaching for SMEs to attend the shows. Our findings also indicate that these preparations were effective and of particular importance to SMEs. It enabled them in pricing their products for the export market, better shipping and displaying for their products, and preparing to answer expected questions from buyers. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 31 TFP assisted SMEs to attend trade shows 178 times. SMEs reported initial sales of over $17 million and projected sales of $50 million in two years of TFP assistance. TFP performance monitoring plan reports did not provide a baseline for SMEs’ export sales in years before 2011. Our discussions with SMEs indicated that for many of them, trade shows had enabled them to keep their sales levels either stable or to improve them slightly. Further, the promotional techniques and new business practices SMEs learned from TFP advisors in preparation for the trade shows appear to have improved their abilities to approach new buyers/importers and advanced their business communication and negotiation skills. SMEs had positive comments on the support they received from the TFP advisors during the trade shows. Advisors helped SMEs in many ways, including booth arrangement, client communication, bringing interested clients to SME booths, escorting SMEs around the show, meeting with clients outside the trade show, assisting SMEs in studying competitors’ packaging, labeling, pricing and product presentation at the trade show. TFP advisors were perceived by many SMEs as reliable coaches who provided guidance to, and helped build the confidence of, SMEs, particularly those attending the trade show for the first time—to the extent that many of those SMEs reported sales during their first trade show visit. In many aspects, the capacity building provided through TFP achieved sustainable results; especially in terms of preparing SMEs for the export experience through trade shows. Yet our findings also show that SMEs still need assistance to be trained on organizing B2B meetings in target markets as part of the trade show experience. Many SMEs also pointed out that better trade transactions could have been achieved if the commercial representation is more engaged in business networking and matchmaking. We propose that future activities provide capacity building activities for the commercial representation. Some SMEs (4 out of 31 interviewed) complained about the location of their booth in two trade shows. They see this as a very important factor that in many cases kept buyers away from their booths. Having an impact on export sales requires attending more than one trade show. Future technical assistance should review the selection criteria to choose SMEs that really need financial and technical support at the trade shows. Some SMEs only need preparation trainings and support in the matchmaking process. Although trade shows are seen as an effective tool for export development for SMEs, it is not the only approach. Many SMEs highlighted the higher importance of trade missions to the EDP. This can take place by sending potential exporter SMEs to target markets or by bringing potential importers to Egypt to visit SMEs and their factories. TRAINING Training through the Energy Efficiency Program The evaluation team interviewed 10 EEP participant SMEs to assess their perceptions of EEP-related training in term of effectives. It was not possible to attribute EEP training alone to direct effectiveness indicators such as realized savings in the production cost element of energy. Thus, the evaluation used SMEs’ perception of training quality as a proxy for effectiveness. Responses reveal that five SMEs have utilized the audit visits as on-the-job training.“Audit visit” appears at the top of the list of strongest TFP activities, with eight frequencies, followed by “training on energy efficiency awareness and measures,” with only two frequencies. Audit visits were very effective and USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 32 impressive to the SMEs; however, 2 of the 10 SMEs defined EEP training as the best TFP activity. Training sites being located away from SME industrial zones was a concern for one of the SMEs. Responses to discussions over their future needs suggest the need for SMEs to be trained on technical topics such as the ISO 50001:2011 Energy Management System, Solar Systems Installation, and Green Energy. Moreover, responses to discussions over their knowledge of current international best practices and the national energy efficiency measures revealed their great need for these two specific topics. Nine SMEs were not familiar with the international best practices, and eight were not familiar with the national energy efficiency measures. Training through the Export Development Program Because it was not possible to attribute EDP training alone to direct effectiveness indicators,such as realized export or sales increase, the evaluation used SMEs’ perception of training quality as a proxy for effectiveness. The analysis revealed that 22 of the interviewed 31 EDP participant SMEs have received training (other than trade show-related) from TFP (71%). However, only 16 SMEs responded to a question on the strongest and poorest TFP activity. In some cases, the interviewee was not familiar enough with TFP activities to be willing to answer the question.With this small number of responses, caution must be practiced when generalizing. However, the evaluation team uses here its expert judgment and results from TFP evaluation reports to overcome this limitation and feels comfortable with the conclusions from the findings and their related recommendations. Detailed analysis finds that 14 out of the 16 respondent SMEs perceive training received as the strongest activity of TFP. However, 3 of those 14 SMEs had some concerns in regard to training, including old￾fashioned training delivery by lecturing, absence of English terminology and accommodating the training event in five-star hotel.Two out of the 16 respondent SMEs perceive training received as the poorest activity ofTFP–this (12%) of the total respondents. Some SMEs expressed their need to accommodate training events in their local zone instead of traveling across Cairo with its heavy traffic. We believe that conducting training site within the industrial zone of the beneficiaries could make it easier for top management/owners to attend. However, proximity to the workplace could prove a distraction that prevents full attendance and concentration. Thus, a case-by-case decision would be advisable for each training event, with a clear “gentlemen’s agreement” on complete full-time attendance and participation in the training. In addition, training in non-peak hours will be also a good idea. Interviews with some SMEs, especially enterprises with more than 100 workers, revealed that some SMEs were not happy with the training content, determining it to be at a basic level that fell below their knowledge and experience or is irrelevant to their sector or specialty. For other SMEs, especially small enterprises, the training content was viewed as very valuable, given their lower knowledge and experience at the personal and organizational level. In this sense, some SMEs were not happy with the level of trainers while others were. We believe this resulted from the capacity-assessment process applied by TFP that did not rely on appropriate judgments of the export-readiness of firms. Training needs to be customized according to; the size of the enterprise, export experience and sector-specific needs. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 33 RECOMMENDATIONS (QUESTION 6) Question 6: Based on answers for each of the above questions, what are the actionable recommendations that USAID/Egypt can adopt/implement in terms of SME export promotion activities and sustainability in future projects? Overall, we recommend that interventions undertaken by TFP should be continued and deepened with a follow-on activity that builds on the redirection of TFP activities that now are fostering both SME growth and greater energy efficiency. As part of a new USAID activity initiative, we also recommend that USAID consider adopting the following activity design and implementation measures to further increase the effectiveness of its trade capacity building and energy efficiency assistance: 1. Adopt a long-term (at least five-year) period for activity implementation. Given the tentative, resource-constrained and risk-averse nature of most small businesses, a new activity must be capable of providing sustained support for an incubation period of several years before SMEs achieve levels of self-sustaining production, marketing and investment growth. 2. Introduce more rigorous eligibility criteria for selecting SMEs to receive activity assistance. The Export Councils do not yet have the capacity to guide in this process. More stringent criteria are needed, and the activity should consider reaching out to other Egyptian associations for additional promising SME prospects. For example, implementers of a new activity should assess SME firms more carefully with respect to their readiness and capacity to export. SMEs could be categorized in terms of the level of their knowledge about exporting and their previous experience to identify the type and level of assistance required and to tailor training according to recognized deficiencies. Similarly, a new activity should customize the type of technical assistance services to SMEs by size, experience and sector of selected firms. 3. Develop and implement a systematic and coordinated SME training plan. Training should be periodic, begin far in advance of trade shows and require firms to do their homework between training events. For example, as part of hands-on training, SME managers would be required, on their own, to research markets, seek B2B contacts and tailor product selection to markets. Subsequent training would grade their progress and warn laggards that funding for their participation might be in jeopardy. Logistical factors such as training locations and timing should be part of training plans for SMEs. 4. Set up clear standards and guidelines for trade show selection, SME eligibility for trade show participation and trade show follow-up support. Locating companies in visible, high-traffic locations at trade shows is important for improving the numbers of visitors, clients, purchase agreements and export transactions. Trade shows should also be complemented by other types of technical assistance;some SMEs recommended seeking other areas of intervention, like assistance on trade missions, trade tours improving the quality of training and trainers, and assisting export consortia. 5. Assure that implementing partners undertake more rigorous selection of training staff. As mentioned above, more specialized and advanced trainings need to be provided. That requires USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. 34 better and more advanced and experienced trainers, or it may involve training of trainers by activity implementers. 6. Offer incentives to export consortia for bringing SMEs together. A new activity could include advocacy, advisors and services to existing and new export consortia to offer incentives for membership and participation by SMEs. Such incentives could include online communities of practice, how-to marketing and export guides, mentoring of SMEs by larger B2B partners/clients and jointly shared trade fair exposition space with an emphasis providing the unique type of technical and marketing services that SMEs require for competing and growing. 7. Introduce systems for more comprehensive and systematic monitoring and recordkeeping of activity interventions. Activities with multiple tasks need systematic and sustained performance tracking, comprehensive data collection and sustained recordkeeping that generate information to assess the costs and benefits of each activity in order to evaluate their absolute and relative effectiveness. 8. Design and implement a dissemination plan for sharing information on international best practices. In the context of a new EEP, a follow-on activity could include a dissemination plan for training, electronic media and other mechanisms suitable to the Egyptian context for sharing environmental best practices—e.g., clean energy and energy efficiency—that are within the capabilities of small to medium Egyptian firms to adopt. For example, SMEs may benefit from support for solar thermal energy installations for hot water heating as an alternative to current traditional use of fossil fuels. Also, future activities could arrange to conduct training in the industrial zones where the firms are located, out of peak hours and at two levels: basics and sector/industry-specific. Finally, technical reports should be translated into Arabic for greater accessibility and use. 9. Recruit short-term specialized trainers for conducting advanced training workshops that fit SMEs with higher experience, expectations and export capacity. The evaluation findings highlight that not all SMEs have the same level of management, marketing and production skills; training should be tailored to SMEs’ levels of skill and experience and trainers selected accordingly. 10. Adopt phasing-in/phasing-out strategy for SME capacity building. The goal should be for every SME to “graduate” to self-sustaining export market participation. This requires a clear SME graduation strategy based on needs assessments that provide training and other support reflecting a firm’s export-readiness and more advanced training for firms with greater capabilities. The overall conclusion of the evaluation team is that this activity successfully adapted to changed conditions on the ground to achieve very positive results in exports and learning by Egyptian SMEs and broke potentially promising ground on energy efficiency. Additional USAID-supported activities focusing on these areas are clearly warranted. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. I ANNEXES USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. II ANNEX I: DESK STUDY REFERENCES  Atkin, D., Amit Khandelwal, and Adam Osman, “Exporting and Firm Performance: Evidence from a Randomized Trial,” Forthcoming, Quarterly Journal of Economics, 2015  USAID/Egypt Trade Facilitation Project Wind-up Plan, December, 2013  USAID/Egypt Trade Facilitation Project SME Export Toolkit  USAID/Egypt Trade Facilitation Project Option Year Work Plan, August 1, 2014–July 31,2015  USAID/Egypt Trade Facilitation Project Monitoring and Evaluation Report Summary of TFP Activity Results and Lessons Learned  USAID/Egypt Trade Facilitation Project Assignment Report: English Language Editing for SME Websites and Promotional Materials, November 2013  USAID/Egypt Trade Facilitation Project Assignment Report: Assisting SME Export Consortia through Training and Coaching in Market Research and Export Planning  USAID/Egypt Trade Facilitation Project Assignment Report on TFP’s Exporters’ Souk Events: Monitoring & Evaluation Results  USAID/Egypt Trade Facilitation Project Assignment Report: Assisting Selected SMEs in The Metal Form Sector with Training and Coaching in Export Markets  USAID/Egypt Trade Facilitation Project Assignment Report: Assisting Egypt’s Food Export Council (FEC) with Buyer Identification  TFP’s Export Development Pilot Project (EDPP) (May 2013–May 2014) Evaluating Performance Changes at TFP Assisted SMEs  Statement of Work for an End of Performance Evaluation of USAID/Egypt’s Trade Facilitation Project (TFP)  SME Export Pilot Project Design  Semiannual Progress Reports, Contractor’s Narrative  QIZ Unit Study Tour to Turkey June 29–July 6, 2013 Trip Report  Monitoring and Evaluation Report on: New York Now Trade Show, New York, USA  Monitoring and Evaluation Report on: SIALl Trade Show, Paris, France 22–24 October, 2014  Monitoring and Evaluation Report on: RMG Roadshow New York, USA 14–15 January, 2014  Monitoring and Evaluation Report on: RMG Magic Trade Show, Las Vegas, USA ,17–20 February, 2014  Monitoring and Evaluation Report on: PLMA Private Label Trade Show, Chicago, USA  Monitoring and Evaluation Report on: Index Jedda Trade Show, Saudi Arabia, 12–14 November, 2013  Monitoring and Evaluation Report on: Heimtextile Trade Show, Frankfurt, Germany, 8–10 January, 2014  Monitoring and Evaluation Report on: Gulfood, Dubai, UAE, 8–12 February, 2015  Monitoring and Evaluation Report on: Fancy Food Trade Show, New York, USA, 29 June–1 July, 2014  Monitoring and Evaluation Report on: Auto Guangzhou Trade Show, China, 21–23 November, 2013  GSP Unit – Architectural Design  FTTC Evaluation SWOT Assessment and Recommendations  Export Consortia Report, Rania Habib, June 30  Energy Efficiency Progress Review  Egypt Trade Facilitation Project Second Annual Work Plan  Egypt Trade Facilitation Project Performance Monitoring Plan (PMP) Reports  Egypt Trade Facilitation Project First Annual Work Plan USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. III  Directory of Small and Medium Enterprises Assisted by TFP  DevTech Technical Proposal RFTOP No. Sol-263-15-000020 End of Period Performance Evaluation USAID/Egypt TFP  Assisting Selected SMEs by Providing Training and Coaching to Succeed In Export Markets  3-2-1—Export Project Design USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. IV ANNEX II: STAKEHOLDER DATA COLLECTION INSTRUMENT Questionnaire 3: Government and councils/ associations Includes government agencies responsible for the implementation of trade related policies, and Export Councils/associations that worked with TFP. Introduction Thank you for your participation. To help make its programs more relevant and effective, USAID/Egypt has contracted DevTech Systems Inc. to evaluate the Trade Facilitation Project. Your responses will be anonymously combined with the information from other respondents, and in accordance with U.S. government regulations on privacy, cannot be tracked, and individual responses will not be given to USAID. Thank you for your help. Section One: Background Information / Respondent's Information 1. Name: 2. Name of Counterpart/partner: 3. Title: 4. Telephone Number: 5. Preferred Address for Email Contact: 6. Date: 7. Interviewer (s): 8. Counterpart/partner's main task/specialization 10. Size of the organization (Number of employees, …) Section 2: General Questions on TFP's assistance 11. Have your organization received assistance from USAID/TFP? o Yes o No 12. If yes, what is the type of assistance received by your organization from TFP? o Training o Awareness Session USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. V o Software Orientation o Intellectual Property Rights o Others: Pls. specify 13. What is your overall rate of the assistance received from TFP? o 1. Extremely Good o 2. Good o 3. Neutral o 4. Poor o 5. Extremely poor 14. What did the Trade Facilitation Project do particularly well? 15. What did the Trade Facilitation Project do particularly poorly? 16. In your opinion, what could be the factors that constrained or could constrain the trade facilitation reform measures with GOE? 17. How your organization helped in achieving TFP objectives? 18. Is your organization planning any changes in the trade policy, strategy or procedures? o Yes o No 19. If yes, what are these changes? 20. Have you participated in a training / capacity building program for TFP? o Yes; pls. specify  Training  Awareness Session  Software Orientation  Intellectual Property Rights  Others: Pls. specify USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. VI o No 21. If yes, do you describe the percentage of male and female participation? Please list by type of assistance:  Training  Awareness Session  Software Orientation  Intellectual Property Rights  Others: Pls. specify 22. For GOE organizations: Is the private sector involved in your dialogue? o Yes o No 23. Would you request future capacity building assistance from USAID to improve trade￾related assistance?? o Yes; pls. specify  Public-Private dialogue  Risk based management  Food Safety unit  IPR  Customs Reform Unit  Others; pls. specify o No Section 3: Recommendations 24. What areas/components of assistance would you recommend USAID to add in its future trade related assistance 25. What areas of assistance would you recommend USAID to adjust/amend in its future trade-related assistance? ___________________________________________ USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. VII INTERVIEW PROTOCOL FOR CONSULTANTS/TRAINERS/COACHES/ADVISORS This protocol will be the basis for the interviews with TFP consultants/trainers/coaches/advisors at the project management level. In the group (or individual) interviews we will tailor the questions to the specific group (i.e., EDP, EEP, Food Safety, etc.) The focus is on the capacity development of the SMEs, the cascade model of training, and sustainability of the training structure. Background Information Date _________ Trainer Name Age Sex Type & area of Training Years of experience as a trainer 1 2 3 4 5 Were you working as a trainer for the MTI prior to the TFP? Did you receive training/orientation under the TFP? If yes, who provided the training/orientation? What was the best part of the training? How does it compare to other training you had in the past while you are prepared as a trainer? Did the training involve any practical experience? Did your trainers/supervisors/M&E specialist observe you working as a trainer and make suggestions? Did you have a role in planning and designing the training materials of the training you provided? If yes, in what way did you participate? If trained, were the training materials you received useful to you during the training? After the training? Do you have any suggestions for improving the materials? Have you ever visited any of other stakeholders of TFP? If yes – for what purpose and did it meet your need? [If no – does he/she know who the stakeholders are?] USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. VIII Have you been certified as a trainer by any certifying body? [If no – ask if he/she plans to become certified, etc.] Did you provide any follow-up with the trainers/trainees you trained – observation or evaluation? What does this follow-up include? Does it affect their performance or promotion? How? In your opinion, was the training you received and/or provided sufficient for you to train and help others? What was missing? What changes would you recommend in the training model of TFP? Did you do any testing or evaluation to assess whether the trainees understand the practices you are teaching them? Do you think they get enough information in the training to apply the practices in their SMEs? How frequently did you visit trainees in their SMEs? What are some examples of successes you have had with the SMEs? What have been the major factors? What have been the major problems you have encountered with SMEs? What have been the major factors? Do you want to continue to work as a trainer with the MTI-related projects and activities? What are the most important factors that will affect whether you continue to work as a trainer? In general, what are/were there problems that you have encountered? In your opinion, how you evaluate consortiums/tradeshows/trade missions? How do you evaluate the current export, customs, and banking policy and regulations? What is exactly needed in this regard to boost exports? How these could be enhanced in the future? Overall comments/recommendations? USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. IX Annex III: Beneficiary Data Collection Instrument TFP BENEFICIARY DATA COLLECTION FORM Questionnaire 1 1. Individuals: Owners, staff and participants, Export Development Program Includes: Both owners and staff who received training and capacity building. Therefore the survey instruments will focus on individual level data from these two beneficiary/target groups. Section One: Background Information/Respondent's Information 1. Name: 2. Name of company/organization: 3. Title: 4. Telephone Number: 5. Preferred Address for Email Contact: 6. Date: 7. Interviewer: 8. Market Sector: 9. Market Subsector: 10. Size of your company/organization (Number of employees …) Section 2: General Questions on TFP's assistance 11. What is the type of assistance received by your company from TFP? 12. What is your overall rating of the assistance received from TFP? o 1. Extremely good o 2. Good o 3. Neutral o 4. Poor o 5. Extremely Poor 13. What does the Trade Facilitation Project do particularly well? USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. X 14. What does the Trade Facilitation Project do particularly poorly? Section 3: Change in Export Performance (Export Increase and Diversification) 15. Have exports in your company increased over the last two years? o Yes o No 16. If yes, how much increase in export would you attribute to TFP? o 0 % - 5 % o 5% - 15% o 15% - 30% o More than 30% 17. If yes, what has led to these increases through TFP assistance, and through other factors a. Through TFP: b. Other Factors: 18. If no, what has led to these decreases in exports? 19. Have you started to produce new product(s) over the last two years? o Yes o No 20. If yes, was that attributed to TFP? o Yes o No 21. Over the two years, have you started to sell your products to new markets? o Yes o No 22. If yes, was that as a result of TFP's assistance? o Yes o No USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XI o Do not know 23. Over the past two years, have you noted a change in the number of buyers for your product(s) o The number buyers has increased o The number of buyers provided has decreased o No change o Do not know 24. If increased, was that as a result of TFP's capacity building assistance? 25. Please have a look at the following type of assistance and check which of the following has mostly or least impacted your company. Multiple answers are allowed. A1: Assistance that had the most impact on your company A2: Assistance that had the least impact on your company Type of Capacity Building Assistance A1 A2 1.Trade Shows and Related Assistance 2.Souq events and / or related trainings 3.Sector specific Training 4. Coaching 5. Trade Shows 6. Intellectual Property rights 7. Awareness sessions 8. Study Tours 9. Others, please specify Section 4: Employment and productivity 26. Have sales in your company increased over the last two years? o Yes o No 27. If yes, how much increase in sales would you attribute to TFP? o 0 % - 5 % o 5% - 15% o 15% - 30% USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XII o More than 30% 28. If yes, what has led to these increases through TFP assistance, and through other factors o Through TFP: o Other Factors: 29. If your sales and / or exports increased over the past two years as a result of TFP, have you hired more workers to meet the increased demand? o Yes o No 30. If yes, what proportion of the new employment has gone to women? 31. Would you say that employee capacity / productivity has increased as a result of the trade facilitation project? o Yes o No 32. If yes, describe in what form and sense? Section 5: SMEs Recommendations 33. What could the Trade Facilitation Project improve upon? 34. How would you recommend USAID to add, adjust / amend in its future assistance? 35. How, if at all, does the Trade Facilitation project address gender in its programming? Questionnaire 2 Individuals: Owners, staff and participants, Energy Efficiency Program Includes: Both owners and staff who received technical assistance and capacity building for the energy efficiency program. Therefore the survey instruments will focus on individual level data from these two beneficiary/target groups. Section 1: Background Information / Respondent's Information 1. Name: USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XIII 2. Name of company/organization: 3. Title: 4. Telephone Number: 5. Preferred Address for Email Contact: 6. Date: 7. Interviewer: 8. Market Sector: 9. Market Subsector: 10. Size of your company / organization (Number of employees, …) Section 2: General Questions on TFP's assistance 11. What is the type of assistance received by your company from TFP? A. Training: o Yes; Please Specify: o No B. On the job training o Yes; Please Specify o No 12. What is your overall rate of satisfaction for the assistance received from TFP with regard to the energy efficiency program? o 1. Extremely good o 2. Good o 3. Neutral o 4. Bad o 5. Extremely Bad 13. What does the Trade Facilitation Project do particularly well with regard to EEP in your organization? 14. What does the trade facilitation Project do particularly weak with regard to EEP in your organization? 15. Have your organization undertaken any energy efficiency work? USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XIV o Yes; Please Specify: o No 16. If yes, what type of energy efficiency work was performed in your business? 17. If yes, please specify the percentage of energy saving in your organization as result of the 'energy efficiency program' o 5-10% o 10-20% o More than 20% 18. If yes, please put the percentage saving that have been incurred or will be incurred as a result of the energy efficiency program with regard to the following (if applies) o Percentage saving in Electricity = ….% o Percentage saving in Natural Gas = ….% o Percentage saving in Solar = ….% o Percentage saving in Water = …% o Percentage saving in Carbon Emissions = …% 19. What has/have the above reduction in saving resulted in? Multiple answers allowed. o Cutting the production cost o Offering competitive prices o Having clean production o Providing more opportunities to export 20. How much Egyptian pounds have been saved? 21. Do you have plans to utilize the saved amount in other areas of investment? o Yes, please specify o No 22. Are there any major barriers to energy efficiency investment at your organization? o Yes; please specify o No USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XV o Do not know 23. Do you have any suggestion(s) to improve future 'energy efficiency Programs' in the USAID? o Yes; please specify: o No 24. Please list here below the success/failures your organization made/faced with regard to energy efficiency efforts. 25. Please list and explain below if there are future energy efficiency needs by your organization. 26. Are there any policy measure(s) that were taken to promote energy efficiency program in Egypt? o Yes; please specify o No o Do not know 27. Are you aware of the international best practices to promote energy efficiency in your sector? o Yes; Please specify / explain o No o Do not know USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XVI Questionnaire 5: Focus Group Questions: TFP Export Consortia Initiative Section 1: Background Information/Respondents' Information 1. Participating Members & Titles: 2. Name of Companies & Contact Information: 3. Date: Section 2: Questions on TFP's assistance in the Export Consortia Initiative 4. What is the type of assistance you received from TFP in the export consortia initiative? 5. What does the Trade Facilitation Project do particularly well in this assistance? 6. What does the Trade Facilitation Project do particularly weak in this assistance? 7. How do you see the TFP's involvement in the export consortium? (Useful/Not useful) Section 3: General questions on the consortium 8. What was the original number of the members/participants in the consortium? 9. Does the member of the consortium have a unified vision? Please describe. 10.What is the current target market(s) of the consortium? Section 4: Questions on companies' involvement in the export consortium 11. What are the joint activities that were undertaken to promote the goods of the members abroad and /or facilitate access to foreign markets? 12. How do you describe the strategic cooperation among the firms? (Sustainable / non￾sustainable), and why? 13. What are the benefits you gained from joining the consortium? o Lower risk of entering new markets (Y/N)? Please explain. o Lower costs (admin., marketing, promotional, freight, warehouse) (Y/N) Please explain. o Increased Profits (Y/N) Please explain. o Reduced time (Y/N) Please explain. o Increased bargaining power over buyers (Y/N) Please explain. o One stop shop for buyers (Y/N) Please explain. o Sharing financial, human resources and knowledge (Y/N) Please explain. o Others, please specify (Y/N) Please explain. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XVII Section 5: Questions on the market related to the export consortium 14. Who are the largest players in this market? 15. What is the product mix they are offering? 16. How easy is it to supply directly to the end user? 17. Did the export consortium provide you with a different/new/innovative strategy for opening new markets? (Y/N) Please explain. 18. Who are the responsible parties on the management of the export consortium initiative, and what are their roles? 19. Which market segments did you target and plan to target in future for supply? Which market segments would require further product development? 20. Are there any local suppliers for those product lines? 21. What is the expected market share, and how can you grow it? 22. Who are the main supplying countries/countries for these products, and what is their market share? 23. What are the main distribution channels, prices, and promotional methods? 24. How about the local supply? Which products at what prices? 25. What are the typical promotional channels for our products (specialized magazines, tradeshows, online ...etc.). 26. What is the cost of each? 27. What are the prices of similar products, and for which quality? 28. What are the agreed upon payment terms? Section 6: Change in export performance (export increase and diversification) 29. Have exports in your companies increased as a result of the export consortium? (Response for each of the participating companies) USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XVIII o Yes o No 30. If yes, how much increase in export would you attribute to the export consortium? (Response for each of the participating companies) o 0 % - 5 % o 5% - 15% o 15% - 30% o More than 30% 31. If yes, what has led to these increases through TFP assistance, and through other factors? (Response for each of the participating companies) o Through TFP: o Other Factors: 32. If no, what has led to these decreases in exports? (Response for each of the participating companies) 33. Have you started to produce new product(s) over the last two years? (Response for each of the participating companies) o Yes o No 34. If yes, was that attributed to TFP? (Response for each of the participating companies) o Yes o No 35. Over the two years, have you started to sell your products to new markets? (Response for each of the participating companies) o Yes o No 36. If yes, was that as a result of TFP's assistance? o Yes USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XIX o No 37. Over the past two years, have you noted a change in the number of companies that supply input/service to you, or that have helped you in your exporting process? o The number of input/service provided has increased o The number of input /service provided has decreased o No change o Do not know 38. If increased, was that as a result of TFP's capacity building assistance? 39. Over the past two years, have you noted a change in the number of buyers for your product(s)? o The number of buyers has increased o The number of buyers has decreased o No change o Do not know 40. If increased, was that as a result of TFP's capacity building assistance? o Yes o No Section 7: Recommendations of the members of the consortium 41. What are your recommendations for the TFP? 42. What are your recommendations for the export council? 43.What are your recommendations for the participating members? USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XX ANNEX IV: LIST OF STAKEHOLDERS AND BENEFICIARIES CONTACTED Small and Medium Enterprises # SME Name Sector Governorate Contact 1 Kozman Weaving Textile 10th of Ramadan Karim Kozman 2 Magdi El Showeikh & Co. RMG 10th of Ramadan Sherif Magdi 3 Samueltex Textile 10th of Ramadan Rania Raafat 4 United Batteries Engineering 10th of Ramadan Ahmed Hamed 5 Bird cage Food 6th Of October Amgad Zahran 6 Comex Industrial Co. Engineering 6th Of October Abdel Maguid Mohamed 7 Malaika Textile 6th Of October laila Helaly 8 Kemet Food 6th of October Nihal Said 9 Malex Textile 6th Of October Ahmed Malek 10 Baby Coca RMG Alexandria Tamer Negm 11 Cold Alex Food Alexandria Mohamed Kamal 12 Deebtex Textile Alexandria Mohamed Deeb 13 Orient Company Food Alexandria Hisham El Masry 14 Sahinlar RMG Alexandria Ahmed El Deeb 15 Soft Linen Textile Alexandria Khaled Kassab 16 Bishara Textile & Garment Co. RMG Cairo Amir Wissa 17 El Kattan Carpets Textile Cairo Dalia El Kattan 18 El Shamy Textile Factory RMG Cairo Eng. Tamer El Shamy 19 Family Pharmacia Food Cairo Eng.Tarek Farid 20 Helcometals Engineering Cairo Nancy Awad & Mohamed Hassan 21 Pyramid Carpet Textile Cairo Eng. Ahmad Khattab 22 Riad Group RMG Cairo Eng. Hany Nassif 23 HGM Fashions Textile Giza Hossm Hassan 24 International Textile House of Egypt RMG Giza Sayed El Bouhy 25 Al Shark Egyptian textile industries Textile Mahallah Mohamed Fathy 26 El Ghanam Company Textile Mahallah Mohamed El Azazy 27 Bed Janssen Textile Obour Mohamed Tarek 28 Egypt Aluminium (ERAMCO) Engineering Obour Ahmed El Roubi 29 Colour mix Textile Qalyubia Amr El Tabbakh 30 Cotton Textile RMG Qalyubia Mohamed Fekry USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XXI 31 Grand Bed Textile Qalyubia El Shafeei Hussien Energy Efficiency Program # SME Name Sector Governorate Contact 1 Awael Co For Food Industries Food 6th of October Mahmoud El Shazly 2 El Horreya 2000 Food 10th of Ramadan Eng. Sabry El Maleh 3 El Shehab MerDye co. Textile 10th of Ramadan Tarek Abdel Aziz 4 Kemet Food 6th of October Nihal Said 5 La Pomme Food 6th of October Amro El Husseiny 6 Nassertex for weaving and Dyeing Textile Obour Eng. Alaa El Gammal 7 Orion Food Industries Food 6th of October Mohamed Tayel 8 PHATRADE - Pharonic Essential Oils Food Obour Ghada Ibrahim 9 Wardatex Textile 10th of Ramadan Eng. Kamal Kilany 10 Candy Makers - Sima Food 10th of Ramadan Eng. Mohamed Farag Organization Contact Food Export Council (FEC) Dr. Manal Karim/Ahmed Samy/Dina Salem Engineering Export Council (EEC) Eng. Laila El Maghraby and Mahah Saleh RMG Export Council (RMGEC) Sherine Hosny Home Textile Export Council (HTEC) Iman Hosny Egyptian National Cleaner Production Center (ENCPC) Mohamed Mahmoud and Ali General Organization for Import and Export Control (GOIEC) Eng. Osama Salem/ Eng. Mamdouh Taha Food Safety Unit (FSU) Dr. Hussain Mansour /Dr. Ehab Morad/Dr. Iman Helmy Ministry of Trade and Industry (MTI) Rania Ahmed Ministry of Industry and Foreign Trade (MFTI) Hanan El Hadary Internal Trade Development Authority (ITDA) Mona Zaki Foreign Trade Training Center (FTTC) Hala Gidamy Expolink (Egyptian Exporters Association) (EEA) Amr Abdellatif Business Women Association (BWA) Yumna El Shereidy Qualified Industrial Zones (QIZ) Ashraf Ezzeldin Foreign Trade Sector (FTS) Ali El Kebeer / Yousr Moneeb USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XXII ANNEX V: STATEMENT OF WORK (SOW) Statement of Work for an End of Performance Evaluation of USAID/Egypt’s Trade Facilitation Project (TFP) Project Title: Trade Facilitation Project Award Number: 263-C-11-00003 Award Dates: May 2011- July 2015 Funding: $13,988,000 Implementing Organization: Nathan Associates Inc. Contracting Officer’s Representative: Manal El Samadony Evaluation Program Manager: Seba Auda, USAID/Egypt Geographic coverage Main locations are; Cairo Alexandria, 10th of Ramadan, Mahala, 6 th of October and Obour PURPOSE The purpose of this statement of work is to conduct an end-of-project performance evaluation of theTrade Facilitation Project (TFP) contract,Award #: 263-C-11-00003. Nathan Associates Inc. has been implementingTFP from May 2011 through July 2015. This evaluation will assist the Mission and USAID/Washington in informing decisions regarding: (1) the effectiveness ofTFP’s approach to improving the export capacity of Egyptian small and medium enterprise (SMEs) and the sustainability of these improvements; (2) the effectiveness of policy-related technical assistance to the Government of Egypt (GOE); and (3) the nature and scope of possible future export promotion projects based on lessons learned from TFP. BACKGROUND 1.Development Context Trade expansion is particularly critical for many of the smallest and poorest developing and transition economies, where local demand is too weak to support rapid expansion of production, employment, and incomes. Participation in the rules-based trading system reinforces good governance and strengthens legal and institutional reforms, creating a more predictable and conducive environment for mobilizing private finance and generating economic growth. Fostering developing countries’ participation in international trade is a key ingredient for economic growth. In May 2009, the United StatesTrade Representative and the Egyptian Minister of Trade and Industry signed a U.S. Government and GOE strategic economic partnership. The two partners agreed at that time that USAID would design a new project to contribute to USAID/Egypt’s assistance objective of improving conditions for trade and investment, and USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XXIII promote economic growth, create jobs, and reduce poverty through improvements in the country’s comparatively strong position as an international trade conduit and hub. 2. Project background To achieve the strategic partnership objectives, USAID started to design theTFP in the summer of 2010, mainly to address important trade policy challenges facing the Egyptian economy, improve the trade policy environment, and expand trade through the provision of technical assistance, training, coaching, and limited IT solutions. The agenda was very ambitious and was to tackle major policies like the customs law, export and import regulations, and Intellectual Property Rights (IPR) regulations. The development hypothesis asserts that improvements in trade facilitation will contribute to Egypt’s economic development objectives in several ways. First, the reduction in barriers to trade and the creation of more efficient internal and external trading systems will help streamline and expedite the flow of goods across and within Egypt’s borders. These improvements will reduce the transaction costs of trade, thereby lowering overall cost of production and the potential for pass through in wholesale and retail prices. With lower costs, Egyptian firms can increase their margins and improve their competitive position and market share in global markets. The expanded economic opportunities brought about by improved trade, in turn, will lead to more and better jobs and wealth creation, the cornerstone of sustainable and broad-based economic growth. Second, Egyptian firms can also improve their competitiveness, and when coupled with improved access to export promotion activities and available trade preferential agreements, can increase their exports and contribute to job and wealth creation.This hypothesis formed as basis for the second phase of the project as well as the re-direction of activities in this phase. TFP was designed to be a three-year technical assistance project with one option year with a total estimated budget of US $13.9 million. The original aim of its technical assistance was policy reform and capacity building with GOE agencies charged with: (1) border management and (2) development and regulation of domestic wholesale and retail markets. The policy reform measures planned under the project were designed in close cooperation with the Ministry ofTrade and Industry (MTI) (at that time internal trade and external trade portfolios were under one Ministry)1 . (Note: The MTI was renamed in August 2014 to be the Ministry of Industry, Trade and SMEs (MITSMEs). The internal trade portfolio was moved to the Ministry of Supply and Internal trade after 2011.) The project had two main expected results: 1) trade facilitation measures improved; and 2) domestic market strengthened. The first result pertained to reform measures that take place at the border, while the second focused on improvements that take place in the domestic market. The TFP contract was awarded to Nathan Associates Inc. in June 2011, right after the 25th of January events that installed a new government in Egypt. These events resulted in drastic changes in all fronts of the Egyptian society-economic, social, and political. Implementation of the TFP was adversely affected, especially with the complete change in high ranking officials in the MTI, including the change in Ministers four times in almost a year. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XXIV From June 2011 onwards, USAID exerted its best efforts to achieve the intended results, but the successes (reported under the actual results section) were very modest. The implementer began work according to an approved work plan, and implemented all tasks with the appropriate government agencies. Specific steps were taken to achieve results, but the project encountered many obstacles, mainly due to the uncertainty in the GOE’s commitment. Because of the unstable political environment following the events of January 2011, progress towards meeting the established objectives of the project was severely impacted and delayed during the first 12 months of the contract. Therefore, in summer 2012, the Mission started to explore other approaches to achieve project results. While USAID and the contractor were in the process of reorientingTFP’s scope of work to reflect the altered political environment, the Regional Inspector General’s office conducted a performance audit of theTFP contract. The USAID/Regional Inspector General recommended modifying theTFP contract to include new illustrative tasks that reflect the new technical assistance approach to focus more on SME development rather than trade policy related reforms. Following the change in government, the GOE prioritized the development of small and medium-sized enterprises (SMEs) as well as export promotion. Government officials sought to increase exports by taking full advantage of all preferential agreements like the Qualified Industrial Zone (QIZ), without undertaking major policy reforms. In light of the GOE’s new policy preferences, USAID/Egypt changedTFP’s approach on technical assistance to focus more on building export capacity for SMEs, as a means of expanding external trade. To implement this change, USAID/Egypt amended theTFP contract in March 2013 to be more directly focused on building private sector export capacity. Based on this modification,TFP can be divided into two phases: the first phase from June 2011 until March 2013, and the second from March 2013 till the end of the project in July 2015. Upon instructions from USAID, the implementer, Nathan Associates Inc. (Nathan) developed and implemented a new approach to work directly with SMEs. Nathan conducted a study in 2012 to choose potential exporting sectors and initiated a pilot program called Export Development Program (EDP) to improve SMEs’ exporting capacities. In the last two years of TFP, the project largely shifted away from focusing on domestic trade, with the exception of the development of food safety website and the energy efficiency task. 3.Intended Results a) Phase 1: 1) Trade facilitation measures improved:  Improve the current export / import executive regulations by working with the Foreign Trade Sector under MTI.  Establish a coordinated risk management system for imported commodities by USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XXV working with General Organization for Import and Export Control (GOIEC) under MTI.  Streamline Egyptian mandatory standards by working with the Egyptian Organization for Standardization (EOS) under MTI.  Finalize customs reform measures by working with Customs Authority under the Ministry of Finance (MoF).  Strengthen the role of public-private dialogue relating to trade across borders working with /through the GOE and Businessmen Associations.  Support the non-tariff measures database initiated by the MTI and theWorld Bank.  Improve enforcement of Intellectual Property Rights (IPR) border measures by working with Customs and GOEIC. 2) Domestic trade strengthened:  Promote distribution/retail sectors by working with InternalTrade Development Authority (ITDA) under MTI.  Overcome deceptive advertising by working with MTI.  Support the food safety system by working with the Food Safety Management Unit (FSMU) under MTI.  Improve enforcement and application of IPR systems in the domestic market working with different offices under the MTI.  Improve environmentally sustainable production through improvement in energy efficiency of private sector companies by working with The Egypt National Cleaner Production Centre (ENCPC) under MTI. The implementation of some of above-mentioned illustrative tasks would have contributed to the following results:  Improved regulatory control of trade across borders;  Improved application of export standards;  Reduced time of clearance in port;  Improved risk management system in the importation of goods;  Eliminated trade of counterfeit products circulating in the Egyptian economy;  Improved IPR application and enforcement;  Improved internal trade environment; and  Improved food safety system. b) Phase 2 As described above, changes in Egypt’s political environment resulted in significant changes to the tasks as originally planned. Given the shift in focus to building private sector export capacity, USAID amended the contract to add two additional illustrative tasks under the first expected result:  Improve awareness and use of special export opportunities and preference programs available to Egyptian exporters; and  Support government and private sector efforts to develop successful small and USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XXVI medium enterprise exporters. 4.Actual results: 4.a)Phase 1: Due to political events in Egypt,TFP was slow on moving into the direction of achieving the intended results. However, the following are examples of deliverables that were attained under the different illustrative tasks of results 1 and 2:  Creation of Pilot Risk Management Unit (PRMU) in the GOIEC.  A study for the possibility of a risk management-based inspection regime in GOEIC was conducted.  A report on institutional development for the General Department for Laboratory Preparation for GOEIC, to reflect the new needs of the department, developed job descriptions for new positions and recommended procurement of IT equipment for the department.  Creation of a special Egyptian Customs Authority task force to assess the problem of the National Customs Information System (NCIS), and how to resume work in this area. An action plan was produced, containing recommended steps for re-starting user-acceptance testing that was underway in 2010, and completing testing to obtain final acceptance by Customs.  Re-Equipping the National Customs Training Institute.  Preparation of three briefing papers on specific issues related to Intellectual Property Rights (IPR) agreements, to which Egypt is a party. A briefing paper related to The Hague Agreement, the Madrid Agreement and theTrademark LawTreaty submitted to Trademark Office.  Resolution of problem between GOEIC and Customs Authority regarding IPR border measures.  Developing Food safety website at Food Safety Unit at the MITSMEs.  Training 363 government employees, across all GOE entities/partners in different areas of trade and customs.  The energy efficiency task under Result 2 was proceeded very well according to the plan with acceptable delays. The objective of this task was to enhance and sustain trade facilitation gains through promoting energy efficiency measures that impact business competitiveness, profitability and growth. The project partnered with ENCPC, under the MTI and other agencies, to implement the program with the participation of nine enterprises from the food processor sector. The program included the following components: energy consumption and costs analysis, instrumentation of energy auditing, and identification of energy management opportunities. For each of the nine companies, the project produced a full audit report that included findings related to: current consumption, potential energy savings,set of performance indicators, investment required, and payback periods. b. Phase 2: USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XXVII The EDP was initiated with the selection of 19 SMEs from Home textile Export Council and Engineering Export Council, and targeted to attain a total of U.S. $2 million of additional export proceeds. The project exceeded this target by almost U.S. $3 million. Upon the success of the pilot program, the project selected another group of SMEs to be part of the second EDP in different sectors. To date, 100 SMEs have been assisted by the project, and a total of about U.S. $17 million of export sales were realized, with around $45 million in expected sales. In addition, a number of activities were initiated with QIZ and GSP units with the Ministry of Industry,Trade, and SMEs (MITSMEs) to provide training programs for employees, procure essential IT equipment, develop outreach strategy, and conduct technical studies. In the option year, the second phase of energy efficiency program was launched with the selection of 13 companies. The program followed the methodology adopted in phase 1. 5.Critical Assumptions The following are the critical assumptions of the original project design:  The MTI is the main counterpart, and is responsible for the two expected results.  The MTI is coordinating technical assistance with other organizations participating in the project.  The GOE has ownership of the trade related policy reform agenda.  The GOE is committed to the implementation of tasks identified in the project.  The GOE and the implementing partner will jointly produce a plan to ensure sustainability. 6.Target groups/areas There are two sets of target groups for this project:  Government agencies responsible for the implementation of trade related policies.  Private associations and their affiliated SMEs. 7. Project Modifications First modification:The purpose of the first modification was to revise the statement of work to include two illustrative tasks reflecting the new focus on private sector development, and incrementally fund the project with $600,000. It was issued in December 2013. This action was also responding to the recommendation stipulated in the audit report dated March 11, 2013, to revise the SOW. Second modification: The purpose of the second modification was to incrementally fund the project with $960,000, which was issued in February 2014. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XXVIII Third modification:The purpose of the third modification was to incrementally fund the project with $236,000, and to extend the period of performance from May 2014 to July 2014. The decision was issued on June 1, 2014. Fourth modification: The purpose of the fourth modification was to exercise the option year of the contract from August 1, 2014 through July 31,2015, and it was issued in July 2014 Fifth modification: The purpose of the fifth modification was to incrementally fund the project with $2,927,449, to implement the activities of the option year. And it was issued in September 2014. All these modifications amounted to a total of $10,894,589 sub-obligated funds, with the ceiling remaining at $13,988,000. 8.Existing Data Upon arrival in Cairo, the USAID/Egypt Office of Economic Growth will make the following documents available to the evaluation team:  Original contract and modifications  Work plans  PMPs and modifications  Progress reports  Audit findings report  “Evaluation of SMEs performance changes” conducted by the project.  USAID Evaluation Report Checklist  ApprovedWind up plan  Any other documents that may be requested EVALUATION RATIONALE 7. Evaluation Purpose This evaluation will assist the Mission and USAID/Washington in informing decisions regarding: (1) the effectiveness ofTFP’s approach to improving the export capacity of Egyptian SMEs and the sustainability of these improvements; (2) the types of approaches the Mission should consider in any future assistance to improve export sectors; (3) the effectiveness of implementing policy-related technical assistance to the GOE; and (4) the nature and scope of possible future export promotion projects based on lessons learned fromTFP. 8. Audience and Intended Users The audience of the evaluation report will be the USAID/Egypt Mission, specifically the Office USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XXIX of Economic Growth, its Trade and Investment team, implementing partners and other international donors. An Executive Summary and recommendations will be provided to the MITSMEs and shared with the general public via the Development Education Clearinghouse (DEC). USAID will use the report to inform future assistance strategies and to share lessons learned with other stakeholders. Stakeholders that were assisted by the project will learn about their strengths and weaknesses from a third party objective observer; and MITSMEs will learn how to increase the benefits of receiving technical assistance from donors. Similarly, assisted SMEs will have the opportunity to discuss how this type of project could better assist them in the future. Evaluation Questions: 1. To what extent were the Export Development Program and Energy Efficiency Program objectives achieved according to the program design and the development hypothesis, at the planned scale, in a timely manner, and within budgeted cost? Why or why not? 2. What were the economic benefits and costs of the Export Development Program and the Energy Efficiency Program? 3. What are the factors that enabled or constrained the implementation of the trade facilitation component/policy reform measures with the GOE? What are the lessons learned and actionable recommendations with respect to this component? 4. How has the current implementation approach (in terms of the SME selection criteria utilized, the selected partner associations to assist in identifying target SMEs, provision of trade show finance, direct training, and coaching) contributed to the achievement of actual results in terms of export sales, increase in employment, institutionalization /adoption of sustainable improvements within the targeted SME? 5. How effective have the capacity building services (e.g., trade show finance, direct training, and coaching) been in developing sustainable capacity in target SMEs? Why? 6. Based on answers for each of the above questions, what are the actionable recommendations that USAID/Egypt can adopt/implement in terms of SME export promotion activities and sustainability in future projects? EVALUATION DESIGN AND METHODOLOGY Evaluation Design Evaluators will use a mix of quantitative and qualitative data collection and analysis methods to answer the questions of interest in this evaluation. The evaluation must follow the USAID Evaluation Policy of January 2011 (http://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf) USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XXX The performance evaluation will use both quantitative and qualitative methods. Data will be based on perceptions of stakeholders (Ministry of Industry,Trade, and SMEs (MITSMEs), and different government agencies under the Ministry, Export Councils directors, SME staff, SME owners, Qualified Industrial Zone (QIZ) staff, Generalized System for Preference (GSP) staff, Customs Authority staff, and others) on what the project has achieved and how well the activities were implemented. A. Data collection methods The EvaluationTeam will propose data collection and analysis methods that are most appropriate to meet the evaluation objectives. The evaluation design matrix will be reviewed and approved by USAID before the data collection and analysis methods are applied in the field. Any instruments developed should be rich enough to draw both subjective perceptions and objective observations of relevant project stakeholders. The data collection methodology will include a mix of tools appropriate to answer the evaluation questions and can include document review, in-depth interview with the key stakeholders, focus group discussions with beneficiaries, and site visits to at least 30% of the total number of SMEs assisted by the project (around 30 SMEs). Sample selection of SME respondents should include sectoral coverage, type of assistance, type and size of enterprise, participation duration in the program and geographic considerations to as representative as possible. Document Review: USAID/Egypt will provide the Evaluation Team with electronic access to key program-related documents prior to the start of the in-country work. All team members shall review these documents in preparation for the initial team planning meeting. Interviews and SiteVisits: The Evaluation Team will conduct in-depth interviews and focus group discussions with the following organizations/staff:  Export Councils (Home textile, Engineering, Food, and Ready Made Garments)  MITSMEs officials and Ministry affiliated counterpart agencies such as QIZ and GSP units), Energy Cleaner Production Center (ECPC), Internal Trade Development Authority (ITDA), Food Safety Unit (FSU), and Egyptian Customs Authority (ECA).  Representative SMEs for each sector (owners as well as the staff that benefited from the training programs). Number Of Companies Location 5 Mahalla 1 Toukh 7 Alexandria 1 Kafr ElSheikh 1 Sohag 28 Cairo USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XXXI 2 Giza 12 10th of Ramadan 10 6th of October 3 AlSharkia 5 Obour 1 Sadat City 1 Demietta B. Data analysis methods Prior to the start of data collection, the evaluation team will develop and present, for USAID/Egypt review and approval, a data analysis plan that details how focus groups and key informant interviews will be transcribed and analyzed; what methods will be used to analyze qualitative and quantitative data from key informant and other stakeholder interviews; and how the evaluation method will integrate qualitative data with quantitative data such as indicators from project performance monitoring records, to reach conclusions about the effectiveness and efficiency of the trade facilitation and export promotion activities conducted byTFP. The data collected will be analyzed by the Evaluation Team to uncover correlations and detect major trends in the variables that are relevant to answer the questions of interest in this evaluation. Data will be disaggregated by gender when applicable in terms of gender of owner of the SME as well as of training and coaching participants, to identify how the project benefitted women. In questions 4 and 5, gender disaggregated data are expected. The EvaluationTeam will conduct an ex-post comparison of financial benefits and costs related to the Export Development Program and the Energy Efficiency Program to determine the net financial benefit to the MSMEs.The EvaluationTeam is expected to exert its best effort to use this analysis in assessing any recommendations it will make to improve the financial merit of the project. C. Methodological Strengths and Limitations Key informant interviews are suggested as primary data collection method for this evaluation. It is noted that these methods can be subject to standard limitations associated with the survey method, including sensitivity of questions asked (e.g., on income), influence of the presence of USAID staff on the respondents’ answers, and language translation limitations. USAID expects that all threats to data validity be discussed and documented in the evaluation planning stage – including detailed mitigating plans to minimize threats to validity. This plan will be included in the final report. The comparison of financial benefit and cost associated to the Energy Efficiency Program is straightforward, but the Export Development Program can be challenging since it largely involves human capacity building with longer-term impacts. The EvaluationTeam is expected to use widely accepted assumptions based on literature review of longevity of the benefits of USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XXXII human capacity building through training, compared to others who did not get the training but who are gaining capacity and skills through on the job experience, and common practice in conducting this type of analysis. EVALUATION PRODUCTS 1. Pre-stakeholder meetings:  Work plan:The team planning agenda will include discussion of a detailed work plan which will include the methodologies to be used in the evaluation and timeline in a detailed Gantt chart. The work plan will be submitted for approval to the Evaluation Program Manager at USAID/Egypt no later than the 3 rd day from starting date.  Methodology plan:A data collection and analysis plan (evaluation design and data collection and analysis) will be discussed during the team planning meeting and discussed with and approved by USAID prior to implementation.  List of interviewees and schedule: USAID will provide the EvaluationTeam with a list of project stakeholders. Prior to data collection, the EvaluationTeam will provide USAID with a list of interviewees and a schedule for conducting the interviews. The Evaluation Team will provide USAID/Egypt with an updated list of interviewees and schedules as meetings/interviews are in progress, especially when stakeholders are added to/deleted from the list.  Data collection instruments: Prior to start of fieldwork, the EvaluationTeam will present the data collection instruments with USAID’s Evaluation Program Manager for review, feedback and/or discussion, and approval. 2. Post-stakeholder meetings:  Discussion of preliminary draft evaluation report:The EvaluationTeam will submit a preliminary draft evaluation report to the USAID/Egypt Evaluation Program Manager for review, prior to final Mission debriefing.  Debriefing with USAID:The EvaluationTeam will present the major findings of the evaluation to the USAID/Egypt Mission. This will be scheduled prior to approval of the draft report and before the team’s departure from country. The debriefing will center on the evaluation questions and include recommendations for future design and implementation of activities.The EvaluationTeam will incorporate in the final report comments from the debriefing, where appropriate.  Debriefing with partners:The EvaluationTeam will present the major findings of the evaluation to GOE project counterparts and other relevant stakeholders. This presentation will be scheduled prior to the team’s departure from country. The debriefing will include a discussion of activities and their associated achievements only, with no recommendations for USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XXXIII future programs. The team will incorporate in the final report partners’ comments, where appropriate.  Draft evaluation report:A draft evaluation report of the findings and recommendations should be submitted to the USAID/Egypt’s Evaluation Program Manager prior to the Team’s departure from Egypt. The written report should clearly describe findings, conclusions, and recommendations. USAID/Egypt will provide written comments on the draft report within 10 working days of receiving the document.  Final report:The EvaluationTeam will submit a final report no later than five working days after receipt of USAID/Egypt’s written comments. If USAID/Egypt determines that there are still content issues to be addressed or that previous feedback has not been satisfactorily addressed, the submitted unedited report will be considered as second draft and further feedback will be given to the team no later than 5 days of receipt of the second draft. If USAID/Egypt determines that there is no need for further changes, the report will be accepted as final. 3. Data sets All data collection instruments, datasets, presentations, meeting notes, and final report for this evaluation will be submitted to USAID/Egypt’s Evaluation Program Manager on three (3) flash drives. All data on the flash drive will be in an unlocked and editable format (preferably cvs or excel). Any quantitative data sets should comply with the requirements in ADS 579. 4. Final report The proposed format for the report should be organized as follows:  Executive Summary  Table of Contents  Introduction  Background  Methodology  Limitation  Findings/Conclusions/Recommendations  Lessons learned/ Future Directions  References  Annexes The final evaluation report should be submitted to USAID/Egypt’s Evaluation Program Manager in electronic format (MicrosoftWord) as well as printed and bound copies (five copies in English and five in Arabic) not later than 2 working days after the receipt of the acceptance decision. The entire report shall be no longer than 30 pages,single-spaced in a Times New Roman font,size 12 type font. All data are to be surrendered to and will remain the property of USAID. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XXXIV Reporting Guidelines  The evaluation report must represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why.  The report must include the evaluation Scope of Work as an annex. All modifications, whether in technical requirements, evaluation questions, evaluation team composition, methodology, budget, or timeline must be agreed upon in writing by the USAID/Egypt Evaluation Program Manager.  Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides must be included as an Annex in the final report.  Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias,recall bias, unobservable differences between comparator groups, etc. ) and what is being done to mitigate the threats to data validity.  Evaluation findings must be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings must be specific, concise and supported by strong quantitative or qualitative evidence.  Sources of information must be properly identified and listed in an annex.  Recommendations must be supported by a specific set of findings.  Recommendations must be action-oriented – organized according to whether recommendations are short-term or long-term, practical, and specific, with defined responsibility for any required action. The final report will be reviewed using the Checklist for Assessing USAID Evaluation Reports (http://www.usaid.gov/results-and-data/information-resources/program-evaluations) D. TEAM COMPOSITION The team must include the following personnel, and all efforts should be made for the team to be comprised of a balanced number of male and female members.  Team Leader: The team leader must be a U.S. citizen and have a minimum of a Master’s degree in trade and international business economics discipline or related field, and 10 years of experience in monitoring, evaluation and analysis of trade related development projects. Experience in designing development assistance program monitoring systems (including survey design) is required.  (2) Technical Advisors – SME experts: The EvaluationTeam must include a U.S. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XXXV and local SME expert. It is strongly recommended that the following characteristics be reflected in the SME Technical Advisors: fluency in Arabic language (applies to the local expert); past experience in export promotion, market access, SMEs technical assistance, monitoring and evaluation of development projects; extensive field experience in Egypt or the MENA region;strong written and verbal communication skills.  Monitoring and Evaluation Expert: The Evaluation Team must include a local monitoring and evaluation expert. The following characteristics must be reflected in the monitoring and evaluation expert: fluency in Arabic and English language; past experience in monitoring and evaluation of economic growth development projects focusing on SME development; past experience in evaluating SME training for capacity building and has the skills to identify and compare the associated costs and benefits, past experience in conducting economic analysis of training programs, extensive field experience in Egypt; and strong written and verbal communication skills.  The local logistics coordinator:A local consultant must serve as local logistics coordinator. The person must be fluent in written and spoken Arabic. He/she will provide logistical, administrative, and clerical support to the team. He/she must have at least four years of experience in an administrative support role E. EVALUATION MANAGEMENT 1. Logistical Support USAID/Egypt will provide overall direction to the evaluation team, identify key documents, and assist in developing a work plan. USAID/Egypt will assist in arranging meetings with key stakeholders identified by USAID prior to the initiation of field-work. The evaluation team is responsible for arranging vehicle rental and drivers, as needed for their site visits around Cairo and in the designated governorates. They will also need to arrange their own hotel accommodations if necessary and procure their own work/office space, computers, internet access, printing and photocopying services. Evaluation team members will be required to make their own payments. USAID/Egypt personnel will be made available to the team for consultations before and during the evaluation process, on information and issues relevant to the evaluation work. 2. Period of Performance Evaluation work is to be carried out over a period of eight weeks. Within three months of issuing the final report, the final report must be submitted by the evaluation team to the USAID Development Experience Clearinghouse (DEC). USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XXXVI Annex VI: Timeline for the Evaluation  Week 1: Document Review prior to team travel  Week 2: Team assembles in Cairo, develops work plan, evaluation methodology, preliminary list of interviewees, interview protocols, and carries out entrance meeting with USAID.  Week 3: Team begins interviews in Cairo with government agencies, export councils, and other institutions, with team splitting into two person sub-teams in order to carry out more KIIs.  Week 4: Using the 2-person sub-team methodology, the team interviews numerous recipients of assistance, mainly in the Cairo area and in nearby industrial zones.  Week 5: Team continues work with assistance recipients, reaching out to other areas. Some team members will travel to Alexandria, while others travel to Upper Egypt.  Week 6: Complete interviews, prepare initial draft of the evaluation report, meeting with USAID on Thursday, November 12 for an exit interview, after which expatriate team members depart for USA.  Week 7: Preparation of draft final report, based on USAID comments, and additional analysis of findings from analysis of quantitative and qualitative information from data collection in Egypt, and submission to USAID of the report  Week 8: Inactive, awaiting USAID comments  Week 9: After receipt of USAID comments on the draft, prepare revisions to the draft report, and submit to USAID. USAID/Egypt TFP End of Project Performance Evaluation Report – FINAL DevTech Systems, Inc. XXXVII ANNEX VII: TFP EVALUATION DESIGN - QUESTIONS, SOURCES, AND ANALYSIS Question Source Analysis Question 1. To what extent were the Export Development Program and Energy Efficiency Program objectives achieved according to the program design and the development hypothesis, at the planned scale, in a timely manner, and within budgeted costs? Why or why not? Review PMP and baseline indicator data. Project documentation, in depth interviews and focus groups with stakeholder organizations and staff. Analyze progress in indicator data, contextualize progress with broader economic developments, tradeable sector developments, and contextual changes. Question 2. What were the economic benefits and costs of the Export Development Program and the Energy Efficiency Program? Project data (budgets) on the EDP and the EEP, with assumptions noted for the EDP. Surveys and indicator data on local, regional, national benefits. Descriptive statistics and cost-benefit analysis, incorporating where possible evidence of long-term improvements beyond the project life. Question 3. What are the factors that enabled or constrained the implementation of the trade facilitation component/ policy reform measures with the GOE? What are the lessons learned and actionable recommendations with respect to this component? Project documentation, including RIG evaluation, internal evaluation, planning documents, stakeholder interviews, focus groups. Coding of key themes from interviews and focus groups, contextual analysis from in depth interviews and focus groups with stakeholder organizations and staff. Question 4. How has the current implementation approach contributed to the achievement of actual results in terms of export sales, increase in employment, institutionalization/ adoption of sustainable improvements within targeted SME? PMP and Progress Reports, indicator data, interviews with SMEs and beneficiaries, survey data. Descriptive statistics and tabular analysis, contextual evidence based on external indicators, coded interview results, survey analysis. Question 5. How effective have the capacity building services been in developing sustainable capacity in target SMEs? Why? Surveys, in depth interviews and focus groups with SMEs, stakeholders and staff. Descriptive statistics of survey data and coding themes for qualitative data. Question 6. Based on answers for each of the above questions, what are the actionable recommendations that USAID/ Egypt can adopt/ implement in terms of SME export promotion activities and sustainability in future projects? Evaluation results, mid-term evaluation results and RIG assessment. Review of findings and conclusions from indicator analysis, stakeholder and focus group analysis, progress reports, contextual information. USAID/Egypt 1A Nady El Etisalat Street New Maadi, Cairo, Egypt 11435 Tel: (2-02) 2522-7000 Fax: (2-02) 2516-4628 cairomedia@usaid.gov www.usaid.gov/egypt