USAID 2017 TRADE CAPACITY BUILDING EVALUATION APRIL 2017 This report was produced for the United States Agency for International Development. It was prepared by Dexis Consulting Group under AID-OAA-C-13-00075, Improving Business Environments for Agile Markets (iBEAM). This contract is held by Dexis Consulting Group, and led by Chief of Party Erin M. Endean (Palladium International, LLC). The report was submitted to Contracting Officer’s Representative, Paul J. Fekete. USAID 2017 TRADE CAPACITY BUILDING EVALUATION APRIL 2017 Authors: Greta Boye, Stephanie Monschein, Rachel Downs, and Jillian Neuberger (Dexis Consulting Group), and Erin Endean (Palladium International LLC) i | USAID 2017 TCB Evaluation CONTENTS Acronyms ..................................................................................................................................................... iii Executive Summary .................................................................................................................................. iv 1. Introduction.......................................................................................................................................... 1 1.1 Evaluation Purpose and Scope ....................................................................................................................... 2 1.2 Evaluation Methodology and Limitations..................................................................................................... 2 1.2.1 Methodology, Data Collection, and Analysis ..................................................................................................... 2 1.2.2 Data Limitations ....................................................................................................................................................... 5 2. Trade Capacity Building Portfolio Overview .............................................................................. 7 2.1 Quantitative Description ................................................................................................................................ 7 2.2 Qualitative Description ................................................................................................................................... 8 3. Findings and Conclusions: Impact on Capacity in Assisted Countries ................................. 9 3.1 Secondary Data and Document Review ...................................................................................................... 9 3.2 Perspectives from USAID Mission Staff .....................................................................................................13 4. Findings and Conclusions: Success in Accomplishing Program Objectives ...................... 15 4.1 Perspectives From USAID HeadquarterS Staff ........................................................................................15 4.2 Perspectives from USAID Mission Staff .....................................................................................................16 4.3 Document Review Using Indicators ...........................................................................................................16 4.4 Document Review Using TCB Results Framework ...............................................................................17 5. Findings and Conclusions: Characteristics of Successful TCB Activities........................... 19 5.1 Perspectives From USAID Headquarters Staff ........................................................................................19 5.2 Perspectives from USAID Mission Staff .....................................................................................................21 5.3 Document Review ..........................................................................................................................................22 6. Findings and Conclusions: Integrating M&E into TCB Activities......................................... 25 6.1 Document Review and Secondary Sources ..............................................................................................25 6.2 Perspectives from USAID Staff ....................................................................................................................27 7. Recommendations ............................................................................................................................ 30 7.1 Activity Design .................................................................................................................................................30 7.2 Implementation ................................................................................................................................................30 7.3 Monitoring and Evaluation, and Capacity Building ..................................................................................31 7.4 Knowledge Management and Learning ......................................................................................................31 Annex A. Summary of TCB Investment Dashboard and Sample ...........................................32 Annex B. Stakeholder E-Survey ..................................................................................................40 Annex C. Interview Questionnaire .............................................................................................49 Annex D. Document Review Guide ............................................................................................51 Annex E. Most Successful TCB Activities included in TCB Evaluation Sample ....................64 Annex F. Compiled Evaluation Findings and Recommendations ............................................66 Annex G. List of Key Interviewees ........................................................................................... 119 Annex H. TCB Results Framework: Success Stories .............................................................. 120 Bibliography ................................................................................................................................ 121 USAID 2017 TCB Evaluation | ii FIGURES AND TABLES Figure 1. Map of TCB Activities in the Evaluation Document Review .................................................................4 Figure 2. Success of TCB Activities by Trade Focus ...............................................................................................11 Figure 3. Mission Perspectives on TCB Impact ........................................................................................................13 Figure 4. USAID Mission Perspectives on TCB Activity Impact ...........................................................................16 Figure 5. Indicator Performance and Success ...........................................................................................................17 Figure 6. Overall Factors of Success, by Information Source ...............................................................................20 Figure 7. Custom Indicator Categories ......................................................................................................................27 Table 1. Percent of Aggregate TCB Funding by Super Category ........................................................................... 7 Table 2. Super Category Concentration per Activity ............................................................................................... 7 Table 3. Distribution of TCB Activities Across Geographic Regions ................................................................... 8 Table 4. Life of Program Indicator Performance (Cumulative, Standard and Custom) ..................................11 Table 5. Details of Keys to Success: Themes Presented in Narratives of TCB Document Sample ............23 Table 6. Successful Trade-Related Activities: Details of Keys to Success Presented in TCB Documents 24 Table 7. Sample of Frequently Used Standard Indicators ......................................................................................26 iii | USAID 2017 TCB Evaluation ACRONYMS ACTE African Competitiveness and Trade Expansion ADS Automated Directives System CAFTA The Central America Free Trade Agreement CDCS Country Development Cooperation Strategy DDA Doha Development Agenda DEC Development Experience Clearinghouse EATH East Africa Trade Hub EG Economic Growth FTE Full-Time Equivalent GAO U.S. Government Accountability Office IP Implementing Partners JBCs Joint Border Crossings KM Knowledge Management LOP Life of Project M&E Monitoring and Evaluation MEL Monitoring, Evaluation, and Learning PIRS Performance Indicator Reference Sheet PMP Performance Management Plan RF Results Framework RVCP Rural Value Chain Project SIECA Secretaría de Integración Económica Centroamericana SOW Scope of Work TCB Trade Capacity Building TRADE Trade-related Assistance for Development USAID/E3/TRR USAID Bureau for Economic Growth, Education, and Environment, Office of Trade and Regulatory Reform USAID United States Agency for International Development USG United States Government WATH West Africa Trade Hub WTO World Trade Organization USAID 2017 TCB Evaluation | iv EXECUTIVE SUMMARY This evaluation responds to USAID’s Bureau for Economic Growth, Education, and Environment/Office of Trade and Regulatory Reform’s (USAID/E3/TRR) interest in better understanding the performance of the Agency’s trade capacity building (TCB) portfolio in priority areas and countries. Using a mixed methods approach to gather quantitative and qualitative data on activities funded and implemented by USAID that began between 2009 and 2015, the evaluation team arrived at the following findings, conclusions, and recommendations for four research questions that aim to inform future activity design and implementation. FINDINGS AND CONCLUSIONS Question 1: To what extent have USAID programs contributed in a measurable way to improved trade capacity in the target countries? The majority of USAID TCB activities have met or exceeded their stated objectives and indicator targets, contributing to improved trade capacity in target countries. Activities contributed to growth in export sales, investment, market linkages; reductions in time and cost to trade; and improved capacity of government institutions tasked with supporting trade and investment. Based on the survey responses and information gathered from interviews, USAID staff both in missions and at headquarters viewed these activities as having positive contributions to trade capacity; however, we cannot quantify these contributions with certainty in a context of systemic change. Issues were cited with TCB activities’ measurement of outcomes, progress, and contribution to trade outcomes due to the absence of an explicit trade-focused theory of change and corresponding TCB monitoring and evaluation (M&E) system. Question 2: To what extent have the interventions funded by USAID since 2009 succeeded in accomplishing the program’s objectives? Although the responses from USAID staff varied according to different degrees of success and the reasons for that success, all responses were positive. However, participants from USAID missions were more positive about USAID’s success in meeting TCB program objectives than were evaluation participants from USAID headquarters, for the same activities. The evaluators’ analysis of the document review also reflected the overall positive findings of USAID assistance. Analysis of performance indicator data confirmed the team’s hypothesis that the combination of private sector and public sector/policy activities would do better than private sector-focused activities that lacked a policy dimension. Question 3: Which activities have been more successful in achieving their objectives, and what were the primary factors responsible for their relative success? According to USAID mission staff and the evaluators’ analysis of TCB documents, the primary factor accounting for an activity’s success was effective management. In particular, activities that achieved stakeholder buy-in, were adaptable, or had some form of flexible-response mechanism, and which maintained a roster of high-quality activity staff and consultants tended to be the most successful. In contrast, the most important factor for success according to USAID headquarters staff was sound design. That said, USAID overseas staff, USAID headquarters staff, and evaluators observed that relationships with all types of stakeholders were the second most important determinant of success. v | USAID 2017 TCB Evaluation Question 4: How can USAID integrate monitoring and evaluation into the design and implementation of TCB programs more systematically? Progress has been made in M&E for trade with the development of the ProjectStarter online Monitoring, Evaluation, and Learning (MEL) Toolkit and the Trade Indicator Handbook, in addition to a more substantive set of standard indicators in the most recent revision. However, more collaborative work needs to be done between USAID mission and headquarters staff to standardize M&E approaches for monitoring trade outcomes, especially in projects that are deemed standalone-trade activities. The Trade Indicator Handbook, while useful, does not recommend standard indicators at each expected result level. TCB projects should focus on using both appropriate output indicators and more trade-specific outcome indicators. These should be standardized where appropriate to allow for aggregation at the TCB level, but nuanced where a standardized approach to a set of custom indicators in not feasible given the activities. RECOMMENDATIONS Activity Design Implement guidance under ADS 201 and 205. Articulate explicit theories of change (pursuant to guidance under Automated Directive System [ADS] 201), develop trade-related logic frameworks, and identify monitoring indicators that focus on measuring progress toward desired outcomes, as well as on measuring outputs, giving attention to the Agency’s broad mandate to consider gender with the advent of ADS 205. Employ adaptive management techniques. During the planning stage of an activity, consider the possibility of things going wrong and plan for contingencies or unexpected developments (adaptive management). Examples include considering how the activity might need to change in the event of a natural disaster or political-security threat. Integrate alternative plans into the design and contracting process or have these conversations with the Implementing Partner (IP) at the beginning of implementation. Use M&E learning tools. Emphasize M&E in new trade Scopes of Work (SOWs) as a management and learning tool, rather than merely as a reporting requirement. Include adequate funds for M&E in contracts. Ensure that activity and contract budgets reflect an appropriate level of M&E, as well as activities for learning. Implementation Develop stakeholder relationships. The time and energy required to develop relationships with stakeholders, which often includes targeted technical assistance to create formal partnerships, are worth the effort to achieve buy-in and foster sustainability. Define and agree on roles and responsibilities within the U.S. Government, as well as between USAID and various host country stakeholders. A best practice is to ensure that everyone working on a given TCB activity understands the role and the value of all interagency players involved in trade, even though different U.S. agencies may see trade through different lenses. USAID 2017 TCB Evaluation | vi Coordinate bilateral and regional activities to magnify impact. Coordination and alignment between bilateral activities and regional ones by USAID staff and IPs is important; most trade issues cross borders and have regional linkages, opportunities, and challenges. Tap the experience of regional institutions. Regional TCB activities benefit from collaborating with regional institutions (such as SIECA in Central America), where one is available and effective; a regional institution may be the focal point for support from multiple donors, and it may coordinate trade capacity building efforts within the private sectors or governments of the region – especially on issues such as standards and border clearance. Monitoring and Evaluation, and Capacity Building Integrate knowledge management and learning resources into USAID platforms. Update the Trade Indicator Handbook to include a more selective list of outcome-focused harmonized custom indicators specific to trade capacity building, and provide training and improve access to and awareness among USAID staff of this document on USAID Knowledge Management (KM) and learning platforms including Learning Lab, ProgramNet, and ProjectStarter. As USAID/E3/TRR’s KM and learning platform for trade￾related M&E, ProjectStarter could be more widely disseminated and utilized. Knowledge management and learning resources (including best practices on trade-related M&E) should also be integrated into and shared on those platforms. Improve TCB indicators by incorporating those from Feed the Future and regional trade and investment hub programs. Integrate and align TCB indicators and data-collection methodologies with the Feed the Future Monitoring System and/or the model performance monitoring plan (PMP) developed for USAID’s three trade and investment hubs in Africa. Expand outcome indicators to TCB activities. For activities heavily focused on capacity building, include outcome-level, knowledge-based training indicators. Knowledge Management and Learning (MEL) Improve MEL resources. Integrate more learning resources into Learning Lab, ProgramNet, ProjectStarter, or another USAID centralized KM platform (e.g., evaluations, research, best practices). When discussing ProjectStarter, emphasize its use as a project design, performance management, and learning tool instead of just a monitoring, evaluation, and learning toolkit. 1 | USAID 2017 TCB Evaluation 1. INTRODUCTION The United States Agency for International Development (USAID) has been involved in trade capacity building (TCB) activities since 2001, when the World Trade Organization (WTO) launched negotiations on the Doha Development Agenda (DDA). (For definitions of key terms, see Box 1). Many development and technical organizations–multilateral, regional, and bilateral—have supported TCB since the launch of the DDA negotiations. The Agency published its first trade capacity building strategy, Building Trade Capacity in the Developing World, in 2003.1 In 2010, USAID commissioned an independent impact evaluation of its TCB activities (Hageboeck, 2010a and 2010b), which concluded that the Agency contributed substantively to expanding the number of people benefiting from global trade.2 In addition, the 2010 evaluation highlighted key opportunities USAID could take to improve the monitoring and evaluation of its TCB efforts. That study comprehensively reviewed TCB activities implemented from 2006-2008, and determined, among other findings, that $1 invested in TCB activities generated, on average, $42 in exports by the assisted countries. USAID released a policy document on the Agency’s priorities and approach to TCB in September 2016 entitled Policy for Trade Capacity Building, updating the 2003 strategy document to reflect the evolution in global trade, and in USAID’s role within the U.S. Government (USG) with respect to TCB. 3 Although in recent years USAID’s funding obligations for TCB assistance have declined, the Agency contributes the largest amount to these activities compared with other agencies such as the Millennium Challenge Corporation and U.S. Department of State (GAO, 2014). Undoubtedly, USAID remains a focal point for the organization of TCB assistance at the country level due to its in-country presence and its mandate to lift the world’s poorest out of poverty through improved trade. In September 2016, USAID commissioned this second evaluation of its TCB activities initiated from 2009 to 2015. USAID’s goal for TCB assistance is to increase the capacity of developing countries to harness global economic forces to accelerate broad-based inclusive growth, increase incomes, and reduce poverty. USAID Policy for Trade Capacity Building Strategy, 2016 1 See http://pdf.usaid.gov/pdf_docs/pdabx241.pdf. 2 See http://pdf.usaid.gov/pdf_docs/Pdacr202.pdf. 3 See https://www.usaid.gov/sites/default/files/documents/1865/20161216%20USAID-TCB%20Policy-digital.pdf. BOX 1. DEFINITIONS  USAID defines TCB as “assistance to developing countries and their citizens to enable them to engage more effectively in trade.” Other donors and technical assistance agencies also use the terms “trade-related technical assistance” and “aid for trade.”  The term “trade facilitation” is associated with activities involved in streamlining the physical movement of goods across borders. USAID 2017 TCB Evaluation | 2 1.1 EVALUATION PURPOSE AND SCOPE This evaluation responds to USAID/E3/TRR’s interest in better understanding the performance of the Agency’s TCB portfolio in priority areas and countries. To provide this information, the evaluation team analyzed the outcomes and structure of USAID-funded and implemented TCB activities based on a document review sample of TCB activities that began between 2009 and 2015, as well as information gleaned from in-person interviews, an online survey, and two case studies. The team sought to uncover the aggregate change brought about by those TCB activities, focusing on four key research questions (see Box 2), which were also included in the 2010 evaluation. While doing so, the team also gathered information on lessons learned, best practices, mini success stories, and ways to integrate M&E systems. These different sources of data were used to arrive at recommendations that aim to inform future project and activity design and implementation, and to improve M&E systems with respect to TCB activities. 1.2 EVALUATION METHODOLOGY AND LIMITATIONS 1.2.1 METHODOLOGY, DATA COLLECTION, AND ANALYSIS The research for this evaluation was undertaken over the August 2016-March 2017 period by an external evaluation team from Dexis Consulting Group (Dexis) commissioned by USAID under the Improving Business Environments for Agile Markets contract. The evaluation process consisted of three phases:  Phase 1: Defined the universe of TCB activities to determine a study sample, created a methodology, developed tools, and undertook a desk review of project documents for 75 activities broadly representative of USAID-supported TCB.  Phase II: Conducted interviews and an online survey; traveled to the field to gather case study information.  Phase III: Analyzed all the information collected, wrote this report, and vetted findings with USAID. The team adopted a mixed methods approach to gather quantitative and qualitative data that included a meta-evaluation, online survey, questionnaire, and comparative case studies (see Box 3). In addition to BOX 2. EVALUATION QUESTIONS 1. To what extent have USAID programs of this type contributed in a measurable way to improved trade capacity in the target countries? 2. To what extent have the interventions funded by USAID since 2009 succeeded in accomplishing the program’s objectives? 3. Which activities have been more successful in achieving their objectives, and what were the primary factors responsible for their relative success? 4. How can USAID integrate monitoring and evaluation into the design and implementation of TCB programs more systematically? 3 | USAID 2017 TCB Evaluation the six-member study team, many experts and stakeholders from USAID and implementing partners directly contributed to this evaluation. Given this broad range of data and information sources, the team created summary documents to facilitate analysis after completing the collection of data by source. As a result, this document presents a synthesis of evidence that goes beyond a summary of findings. In other words, the synthesis required judgment and interpretation on behalf of the evaluation team, individually and as a group. Where possible, quantitative findings are incorporated into the mostly narrative analysis, which was systematically undertaken around the stated research questions. It should be noted that this evaluation does not intend to update the extensive literature review and empirical research presented in the 2010 evaluation. While parts of the methodology of the two evaluations are similar, the current evaluation focuses more on synthesizing qualitative information than the 2010 study had done. Nonetheless, where possible, the evaluation team draws similarities in findings between the two, with the objective of highlighting changes in USAID TCB assistance in the context of a changing global trading system. We also highlight recommendations for improvement in TCB programming and implementation, including recommendations relating to M&E that build on those made in the 2010 evaluation. BOX 3. SUMMARY OF TCB EVALUATION METHODOLOGY DATA SOURCE TYPE OF DATA COLLECTED EVALUATION METHODOLOGY SAMPLE SIZE & HOW IDENTIFIED TCB Investment Dashboard Quantitative and qualitative data on TCB activities Review of universe of TCB activities with study team and USAID 75 activities chosen in collaboration with USAID based on representative funding and categories of TCB assistance; When reviewing documents, the evaluation team condensed some of the TCB Investment Dashboard activities and removed one to have a final sample of 71 activities. Development Clearinghouse (DEC) Documents: Evaluations (mid-term, final, Office of Inspector General), contractually required reports (quarterly, mid-term, annual, final), and non-required reports (briefings, fact sheets) Meta-evaluation and systematic document review using standardized tool Over 150 documents related to activities identified in TCB Investment Dashboard. E-survey Qualitative and quantitative Questionnaire using close-ended (Likert scale) and open-ended queries 32 records created from staff in overseas missions; USAID identified sample. Interviews (in-person or telephone) Qualitative Questionnaire using open-ended questions 19 persons from USAID Headquarters; USAID identified sample. Field visits Qualitative and quantitative Comparative case studies 2 case studies - Guatemala Rural Value Chains Project (RVCP) and Philippines Trade-related Assistance for Development (TRADE) project - identified with USAID. USAID 2017 TCB Evaluation | 4 TCB Investment Dashboard—Evaluators initially reviewed data from the TCB Investment Dashboard,4 with the objective of choosing a sample of activities to be reviewed that USAID funded and implemented during 2009-2014. We supplemented this information with a list of other TCB activities provided by USAID from 2015 (and ongoing). The team considered the resulting dataset to be the universe for this evaluation. In all, we identified 563 TCB activities over the 2009-2015 period; this represented more than $3.2 billion in obligated funding. Activities in this sample averaged $5.9 million, ranging from $2,500 to $113 million. Evaluators removed activities that were not representative of USAID TCB programming (e.g., funding to other institutions) and arrived at a sample frame of 466 activities comprised of 2,750 records. The team then further narrowed the number of activities to 202 by focusing on the largest activities, in terms of the total funding amount reported in the TCB Investment Dashboard. Those 202 activities represented 74 percent of total TCB funding. Since the Investment Dashboard is not linked to activity evaluations, the team turned to the DEC, Internet, and IPs to locate relevant activity documentation. Figure 1. Map of TCB Activities in the Evaluation Document Review DEC—For these 202 activities, the team searched various databases for documentation sufficient to undertake the desk review. Evaluators found sufficient documentation for 75 activities; these activities represented a little less than $1.1 billion in funding. Overall, this sample was reasonably reflective of the sample frame characteristics, especially in terms of geographic distribution and the overall proportion of 4 The TCB Investment Dashboard can be found here: https://tcb. usaid.gov/dashboard.html. It is a USAID website that collects and displays data on TCB activities gathered annually. More information on the website’s history and methodology can be found here: https://tcb.usaid.gov/about.html#tab-methodology. 5 | USAID 2017 TCB Evaluation TCB funding dedicated to each TCB “super category.”5 During the document review, the team discovered duplicates in activities, which narrowed the number of activities in the final sample to 71 activities. See Annex A for details on the final sample, and Figure 1 for a map of TCB activities in our document review. E-survey: USAID overseas staff—The evaluation team designed an e-survey containing a mix of 39 open-ended and closed-ended questions. (For details, see Annex B). The survey was sent to more than 90 persons located in more than 20 missions outside USAID’s Washington headquarters who were identified by the team from USAID/E3/TRR. The evaluation team considered the response rate of about 34 percent adequate, and comparable with the response rate for the 2010 evaluation. For questions in the survey whose answers are quantitative data (e.g., question 7 in Annex B that asked the participant to score his or her answer), the evaluation team was able to calculate descriptive statistics. For answers to the questions that evoked qualitative data (e.g., question 8 in Annex B), the evaluation team coded the answers by theme and summarized those themes. The team created an Excel database to sort and analyze information gathered by both types of questions. Face-to-face and telephone interviews: USAID headquarters staff—Most of the 19 interviews conducted were by telephone; a few were conducted in person. The objective of the interviews was to gather qualitative data from a wide range of USAID staff working at USAID headquarters in Washington, DC. Interviewers used a standardized questionnaire (see Annex C) of 16 open-ended questions related to four broad areas: (1) evolution of TCB portfolio over the years, (2) success or failure of TCB activities, (3) using M&E to measure performance of TCB activities, and (4) priority needs for the TCB program in the future. A two-person team participated in the interviews to facilitate team discussion. To analyze the answers to the questions, the evaluation team created an Excel database, and following the same process to analyze qualitative information as in the e-survey, coded answers by theme and then further analyzed those themes to arrive at findings and conclusions. Field visits—A technical advisor traveled to the field twice to prepare case studies that provided important input to this evaluation. The first trip was to the Philippines during October 2016 to better understand the USAID-funded Trade-related Assistance for Development (TRADE) project; the second was to Guatemala during November 2016 to discuss the Rural Value Chains Project (RVCP). 1.2.2 DATA LIMITATIONS In any evaluation using a mixed methods approach, there are opportunities for error. While this evaluation faced a number of challenges in this regard, none appear to be significant enough to introduce major distortions. Many of the areas that may have introduced possible errors also appeared in the 2010 evaluation. TCB Investment Dashboard errors. Since TCB funding is not a line item in a budget, USAID staff in the field may not know how to properly input data on TCB funding in to the Investment Dashboard. 5 To manage more effectively the chosen sample of activities, the team created six super categories based on 14 labels (or sub-categories) assigned to each activity in the TCB Investment Dashboard: (1) trade-related infrastructure, (2) other TCB, (3) trade facilitation, (4) trade-related agriculture, (5) private sector TCB, and (6) public sector TCB. For details of allocation of sample activities to super-categories and sub-categories, see Annex A. USAID 2017 TCB Evaluation | 6 Thus, the study team found that the TCB Investment Dashboard contained activity sub-components that were entered as standalone activities, and removed them to avoid double counting.6 Missing project documents. The process of locating documents for USAID TCB activities was long and complicated, just as it was with the 2010 evaluation. Although the 71 TCB activities for which the study team eventually located sufficient documentation to conduct an evaluation represent nearly all activities in the original sample frame of 75, they only represent 35 percent of the 202 activities for which we searched for documentation.7 During this effort, the study team discovered that DEC or Internet searches did not always turn up documents that one would expect to have been submitted (e.g., annual and final reports). In those cases, the study team worked with DEC administrators and requested documentation from IPs. Nonetheless, the document search uncovered important limitations of the DEC, viz., inconsistent reporting/submission of regular reports and evaluations by IPs that would have informed this evaluation or which could help with others in the future, as well as with project design and implementation. Inconsistency in information presented in USAID project reports. The evaluation team relied heavily on project documents for its desk review, many of which were prepared by the U.S. firms and private voluntary organizations that implemented those activities. The team reviewed more than 150 documents that included evaluations carried out at USAID’s request by both internal and external evaluators and the Office of Inspector General. For a list of all documents and evaluations reviewed as part of this study, see Bibliography and Annex F. The percentage of activities for which evaluations of this sort were found, however, was relatively low (69 percent). With respect to the content of project documents, evaluators used a document review guide to populate about 50 fields of a database, for the guide, see Annex D. The quantity and nature of the information gained in the document review was very much dependent on the type of activity documents found. The documentation available per activity varied in amount and comprehensiveness; in some cases, this made aggregating or comparing data a challenge. Use of subjective evidence. As described in the “Methodology, Data Collection, and Analysis” section, one important source of data for this evaluation was a voluntary survey of USAID mission staff who have direct responsibilities for managing TCB activities. The team recognizes that many of these survey respondents may have an interest in the perception of those activities’ results, and that the voluntary nature of the survey also raises the possibility of selection bias. When using the survey data, the team kept these limitations in mind and triangulated data as much as possible, while acknowledging the value of the data on perception that the survey provides. 6 Three activities were found to be the same as other activities in the sample. In all cases, it appears that activity sub-components were entered as stand-alone activities in the TCB Investment Dashboard during sampling; this fact was unknown to the evaluation team at the time the sample was compiled. Where sub-components were encountered, the evaluation team conducted a single, comprehensive review of the associated activity/program. Two of these sub-component activities were associated with the Colombia MIDAS program; the other was part of the Dominican Republic Environmental Policy Compliance Program. 7 In addition to the three activities removed from the sample due to duplication, as described above, an additional activity was removed because its documentation was for a long-closed predecessor activity that was outside of the scope of this evaluation. As a result, the sample frame was reduced to 71 TCB activities. 7 | USAID 2017 TCB Evaluation 2. TRADE CAPACITY BUILDING PORTFOLIO OVERVIEW To properly evaluate TCB activities and the success of USAID TCB interventions, the evaluators examined the broad characteristics of the TCB portfolio. We accomplished this review by collecting both qualitative and quantitative data on current TCB activities and their characteristics, development, and implementation approach. 2.1 QUANTITATIVE DESCRIPTION Looking first at the condensed sample frame referenced in the methodology section, certain key characteristics of the TCB portfolio between 2009 and 2015 are evident. This frame contains 466 activities, comprising $2.8 billion in obligated funding. Individually, these activities averaged $6.2 million in size; they ranged from $2,500 to $87 million. The TCB Investment Dashboard disaggregates data into 18 categories (excluding the General Trade Facilitation category only used between 1999 and 2001). For ease in analyzing the data, we clustered these activities into six “super categories.” Table 1 shows the breakdown of activities according to these super categories; some activities had aspects of more than one super category and are counted in multiple categories. Overall, the most funding appeared in the Private Sector, Trade-Related Agriculture, and Public Sector super categories. In terms of the concentration per activity (the number of super categories to which each activity was attributed), the majority of activities were in a single super category. For greater detail, see Table 2. Geographically, the sample spans 13 “regions” per the TCB Investment Dashboard (see Table 3). The most TABLE 1. PERCENT OF AGGREGATE TCB FUNDING BY SUPER CATEGORY TCB SUPER CATEGORY FUNDING ACTIVITIES $ % Number % Private Sector 921,095,992 33% 260 56% Trade-Related Agriculture 736,115,646 26% 180 38% Public Sector 650,087,314 23% 228 49% Trade Facilitation 198,427,412 7% 99 21% Trade-Related Infrastructure 287,824,548 10% 68 15% Other TCB 94,982,980 3% 64 14% Note: Sample frame equals 466 activities. TABLE 2. SUPER CATEGORY CONCENTRATION PER ACTIVITY TCB Super￾Category Concentration # Activities Percent of Sample 1 Category 248 53% 2 Categories 97 21% 3 Categories 62 13% 4 Categories 30 6% 5 Categories 21 4% 6 Categories 7 1% Note: Sample frame equals 466 activities. USAID 2017 TCB Evaluation | 8 represented regions were (1) Central America and Caribbean, (2) Central and Eastern Europe, and (3) East Africa. 8 2.2 QUALITATIVE DESCRIPTION Qualitative analysis reveals that the TCB portfolio has changed significantly in recent years. Interviewees from USAID headquarters staff discussed changes in the technical approach for TCB activities and changes in the portfolio’s broader focus and vision. Shifts in emphasis and the types of activities being designed include an increase in trade facilitation activities, including support for ratification and implementation of the WTO Trade Facilitation Agreement, the increased integration of TCB into large, multi-faceted economic growth or food security activities, and an increased emphasis on policy and institutional development. These changes in focus have also come with changes in the approach to TCB activity implementation. Specifically, interviewees highlighted increased direct funding to local organizations, an emphasis on cost sharing with other donors, a greater focus on value chains, and decreased availability of funding for standalone trade activities (independent of food security and other broader activity objectives.) Overall, the qualitative and quantitative data we collected portrayed a robust and diverse TCB portfolio that has developed and expanded in recent years. The portfolio comprises a geographically varied grouping of activities, reflecting significant investment in the private sector, public development and policy, and trade-related agriculture. 8 Activities were assigned to geographic regions in a multi-step process. Where regions were clearly assigned in the TCB Investment Dashboard data, those regions were used. When only countries of operation were provided, activities were grouped into TCB Investment Dashboard regions based on the regional mission for those countries. Finally, in some cases, activities were implemented in multiple countries under different regional missions so the evaluation team created additional regional categories. For example, activities in multiple continents were deemed global, an activity in Central African Republic, Cote d'Ivoire, Guinea was classified as Sub-Saharan Africa, and an activity in China, Cambodia, Indonesia, Laos, Malaysia, Papua New Guinea, Thailand, and Vietnam was classified as Pan Asia. TABLE 3. DISTRIBUTION OF TCB ACTIVITIES ACROSS GEOGRAPHIC REGIONS ACTIVITIES REGION No. % Central America and Caribbean 81 17% Central and Eastern Europe 66 14% East Africa 61 13% Middle East and North Africa 43 13% Southeast Asia 36 9% West Africa 37 8% South America 29 8% Central Asia 22 6% Sub-Saharan Africa a/ 24 5% Pan Asia b/ 20 4% Southern Africa 20 4% South Asia 15 3% Global 12 3% a/ These activities operated in multiple countries spread across multiple African regions (e.g., an activity operating in a Southern African country and a West African country such as a Trade and Investment Hub project). b/ Same as for above, but for multiple Asia regions. 9 | USAID 2017 TCB Evaluation 3. FINDINGS AND CONCLUSIONS: IMPACT ON CAPACITY IN ASSISTED COUNTRIES Question 1: To what extent have USAID programs contributed in a measurable way to improved trade capacity in the target countries? The evaluators used a variety of measures from high-level outcome data collected in a standardized way through the State Department Office of Foreign Assistance (F) indicators, Feed the Future monitoring system, and the African Competitiveness and Trade Expansion (ACTE) Performance Management Plan (PMP) for the three Trade and Investment Hubs in Africa. They also scored activities based on activity￾level indicators and evaluations to determine, at the activity level, the extent to which TCB is contributing to improved trade capacity in target countries. The evaluation team then aggregated and averaged the scores to assess aggregate performance of the TCB program. In order to qualitatively assess impact on capacity in assisted countries, the evaluators looked at the evaluation findings and conclusions for activities with a heavy emphasis on trade. It should be acknowledged that the lack of harmonized trade-focused indicators at the outcome level (such as exports) made it difficult to answer this question directly. For example, a USAID standard indicator measures jobs data, but that indicator does not specify whether the jobs are trade-specific or not. Even investment data can only be attributed to trade at the activity level if disaggregated, but not at the programmatic or “F level.” This would be difficult to offset by requiring standardized disaggregated data, since trade is not necessarily the only disaggregation for which this would be warranted. Unfortunately, while cited as an issue in reporting, there is little that can be done to improve this deficiency, other than instituting a custom indicator or a custom disaggregation requirement for projects where trade-specific jobs are an expected outcome. Often more advanced quantitative analysis with plausible assumptions for a counterfactual is needed to better understand cause and effect. The 2010 TCB evaluation estimated the effects of USAID TCB assistance on exports, and determined that every $1 of assistance is associated with $42 in exports (Hageboeck, 2010a and 201b). USAID excluded such research in its scope of work for this evaluation. Accordingly, the evaluators used a variety of other techniques to analyze quantitative and qualitative primary and secondary data to provide answers to this question. 3.1 SECONDARY DATA AND DOCUMENT REVIEW Based on an analysis of each of the data sources cited above, USAID TCB activities have generally met or exceeded their objectives and have contributed to improved trade capacity in target countries. However, issues have been cited with measurement of outcomes, progress, and contribution to trade outcomes with TCB activities due to the absence of an explicit trade-focused theory of change and corresponding M&E system. Numerous cases can be highlighted to demonstrate TCB activities’ contribution to exports, including the Guatemala Rural Value Chains Project (RVCP) as shown in Box 4. However, it is impossible to aggregate this indicator data to answer the question on the TCB portfolio level because of the various data collection methodologies used to collect and attribute the data. USAID 2017 TCB Evaluation | 10 Since relatively few TCB activities use outcome indicators specific to trade, and certainly not a harmonized set, it is difficult to answer this first evaluation question in its most precise form. This itself constitutes a critical evaluation finding. In the sample of TCB activities covered by this evaluation, the majority of indicators used to monitor activity results are standard output indicators that are not specific to trade capacity—even though, as demonstrated in Box 5, output indicators do add value when used appropriately. Therefore, the team has relied heavily on qualitative and anecdotal data from interviews, case study field visits, and evaluation reports—in addition to annual, final, and M&E reports—to assess progress on indicators to inform the extent to which activities met their objectives, and then the extent to which TCB funds have contributed to improved trade capacity. To do so, the team scored the proportion of indicator targets the activity met or exceeded in each year of the activity. The ability to score an activity signifies that indicators and other evidence existed in activity reports or indicator tables to demonstrate progress towards intended results, meaning that success was “measurable” in some way. The team found that 48 of the 58 activities for which there was sufficient information to assess performance (68%) achieved 50% or more of their objectives; 27 of the 58 (just under 50%) achieved 75% or more of their objectives. However, this scoring does not consider the appropriateness or validity of indicators selected for expected results, nor does it consider whether indicators are output or outcome indicators. Additional analysis on indicator use and type is included later in this report in response to question 4. 9 ANACAFE and AGEXPORT indicator tables from USAID/Guatemala (2017). BOX 4. FEED THE FUTURE TRADE-RELATED AGRICULTURE CASE STUDY In Guatemala, the Rural Value Chains Project (RVCP) –a Feed the Future project with an emphasis on trade- is supporting indigenous artisans and coffee farmers in 30 municipalities of the Western Highlands region to export to markets in the United States and abroad by supporting quality improvement and, once ready to export, making linkages to international importers. Hindrances to the project’s achievement of export goals include transportation infrastructure impeding movement of goods from the Western Highlands regions to Guatemala City for export. To date, AGEXPORT has increased exports from agricultural products and handicrafts by 44% each year over the baseline of $2.8m (2013) through 2015, and ANACAFE has contributed to $61.7m in coffee exports from 2013-20159 . BOX 5. PHILIPPINES TRADE PROJECT’S PERFORMANCE MANAGEMENT SYSTEM The Philippines TRADE project has a robust results framework (RF) that shows contribution to the Country Development Cooperation Strategy (CDCS) RF, the Partnership for Growth RF, and to a logical framework that clearly delineates the activity’s theory of change. To track performance, the implementation team selected a small set of outcome and output indicators (both standard and custom) that align with the work plan. 11 | USAID 2017 TCB Evaluation The team used another methodology to assess whether TCB activities considered as standalone trade activities (e.g., represented in our Philippines TRADE case study) were, on the whole, more or less successful than TCB activities that were undertaken as part of a broader activity. To do this, we developed a “trade focus score” in which activities were categorized as “solely focused on trade” to “very minimally focused on trade” with several in-between categories. This determination was based on whether IP documents explicitly mention trade across borders as an objective of the activities. We then compared the life of project (LOP) indicator score (described in Table 4) and the evaluation score from our review of external evaluations (each of which was rated on a scale of 1 to 3) to the trade focus score above. This triangulation of the data between the LOP score and the evaluation score increased the reliability of our data: activities with a high LOP score also received a high evaluation score. Figure 2 shows that overall, the average LOP indicator score for all activities for which data are available is 2.3 out of 3. Activities with a trade focus received an LOP score of 2.29 out of 3—essentially the average of all activities. Activities that were mainly (but not solely) focused on trade received a higher￾than-average score of 2.47. This finding is not statistically significant and does not imply cause and effect thus the evaluation team has no way to determine whether the presence or absence of additional non￾trade specific activities contributed positively or negatively to performance. Geographically speaking, TCB activities based in Southeast Africa received an LOP score of 2.6, markedly higher than the average; the three activities in South America each received a score of 3. TABLE 4. LIFE OF PROGRAM INDICATOR PERFORMANCE (CUMULATIVE, STANDARD AND CUSTOM) SCORE NO. ACTIVITIES % OF SAMPLE Less than 50% of indicator targets achieved (Score=1) 10 14% 50-74% achieved (Score=2) 21 30% 75+% achieved (Score=3) 27 38% Not able to assess 13 18% Note: Our sample included 71 activities from the TCB Investment Dashboard. Figure 2. Success of TCB Activities by Trade Focus 0 0.5 1 1.5 2 2.5 3 0 1 2 3 4 Trade Focus (Low -> High) Average LOP Indicator Success Score Average Evaluation Success Score USAID 2017 TCB Evaluation | 12 Activities in Southern Africa (1.67) and Central and Eastern Europe (2) scored the lowest on the LOP indicator score. Activities with total funding of less than $15 million received an average LOP score of 2.44 versus those above $15 million with a score of 2.19. Annex E contains a list of the most successful activities according to Table 4 above. The team also reviewed and scored evaluation reports (on a scale of 1 [negative] to 3 [positive]) for all TCB activities, and focused its attention on activities that were “mostly” or “exclusively” focused on trade. These activities had evaluation reports that reflected significant progress made towards trade capacity in the countries; the average evaluation score for mostly or exclusively trade focused activities is 2.5 out of 3. This indicates that most activities that have a heavy emphasis on trade as an objective are contributing to the achievement of their expected results and improving trade capacity in country in some capacity. A compilation of the key findings of these evaluations can be found in Annex F. A primary example of this contribution towards trade capacity in countries comes from the three Trade and Investment Hubs in Africa. According to the East Africa and Investment Trade Hub final evaluation report (Pragma Corporation, 2015), the Hub contributed to a reduction in the time and cost to trade through its support of joint border crossings. The Port of Dar es Salaam, with Hub support, streamlined cargo clearance times from several days to several hours, thereby reducing paperwork cleared by more than 100 agents a day to about 25 agents. In most instances, the joint border crossings pioneered by the Hub have had a significant impact on the time and cost of border clearance at the sites they operate in (highlighted by a 69 percent reduction in transit clearance times along the Northern Corridor, which resulted in estimated cost savings for private sector firms shipping goods across borders of between $44.2 million to $58.3 million per year). These efforts have tangibly streamlined trade transit procedures in key chokepoints in the region. According to surveys of beneficiaries cited in the evaluation of West Africa trade-related projects (Development and Training Services, 2012), Hub efforts resulted directly in $178 million of exports from 2007 to 2011. Per the most recent West Africa Trade and Investment Hub factsheet,10 the current Trade Hub (2014-2019) has facilitated $57 million in new sales, including $44.7 million of sheep and cattle trade, $7 million of mango sales, and over $4 million of apparel exports. The Hub also enabled $4.6 million in regional and international trade through technical assistance and support to AGOA Trade Resource Centers in the region. USAID also supports trade policy reform at the outcome level, including the negotiation, ratification, and implementation of trade agreements and efforts to reform domestic customs or competition policies. The impacts of these interventions, however, are less easily quantified at the outcome level because the only consistently-tracked standard indicator is the “number of policies, regulations… supported by USG.” Anecdotes and examples of successful cases are discussed throughout this evaluation report, but it was not possible either to aggregate the impact and determine whether it was on a whole positive or negative, or to look beyond the “adoption” of policies and regulations into the implementation. The current set of standard indicators includes one focused on WTO support, so in the future it will be easier to look across the portfolio. A prime example of USAID support for trade policy can be found in Box 6 below: the Philippines TRADE project. 10 https://www.usaid.gov/west-africa-regional/fact-sheets/west-africa-trade-and-investment-hub 13 | USAID 2017 TCB Evaluation 3.2 PERSPECTIVES FROM USAID MISSION STAFF Mission staff respondents in the survey (n=24 missions) felt that the TCB activities on which they are currently working contributed in a measurable way to improved trade capacity in the country or region that the activity supports (M = 54, Mdn = 60, SD = 23.51), as shown in Figure 3. The survey asked participants to rate the extent to which their TCB activities contributed in a measurable way to improved trade capacity. Examples were given on how to measure: indicators, feedback from beneficiaries, evaluations, or national statistics tracked but not attributed to the activity. This was a closed-ended question using a Likert scale: 1 being “not at all sufficiently,” 2 being “sufficiently,” and 3 being “to a great extent.” The software used (Survey Monkey) automatically weighted those responses and re-scaled them from 0 to 100, thereby resulting in the axis presented in Figure 3. Next, while no definition of the word “sufficient” was given in the survey, the three-point scale, reference to accomplishing TCB program objectives, and the examples of measured provided in the survey question implied a limited range (1-3) of satisfaction or acceptability (i.e., achievement). Mission staff responses to survey question of “To what extent have USAID programs contributed in a measurable way to improved trade capacity in the target countries?” Figure 3. Mission Perspectives on TCB Impact USAID 2017 TCB Evaluation | 14 The Philippines TRADE project was selected as an in-depth activity review and case study because it is one of the few remaining standalone trade capacity building projects funded and implemented by the U.S. Agency for International Development (USAID), and one that focuses exclusively on trade policy and the government capacity needed to draft, pass, and implement it. This project is not necessarily unique in its approach, but unique in that very few projects like this one remain. The evaluation team spent about one week in Manila in October 2016 and met with TRADE project staff, government beneficiaries in the Customs Administration and the Department of Trade and Industry, private sector partners, and USAID. TRADE, implemented by Deloitte and subcontractor Palladium since 2013, has four components, each supporting the Philippines Government to strengthen trade through policy reform and capacity building, including many areas ranging from customs modernization to competition law. Its trade facilitation component was successful in efforts to help the Philippines streamline Customs and integrate the border clearance responsibilities of multiple government agencies; they also helped secure passage of the Customs Modernization and Tariff Act. Further, after four years working to support the Government to pass the Philippine Competition Act, nearly 20 years in the making, Congress passed the law in June 2015 and has begun working to implement its by￾laws. The Philippines was the only country of the original five ASEAN nations without a competition law, the lack of which impeded trade, investment, and economic growth. A frequently cited anecdote that demonstrates the impact of a lingering competition law is that it costs more to ship from Manila to other islands in the Philippines than it does to ship items from Manila to Hong Kong because there is only one shipping company monopolizing the Philippines shipping industry. This in itself has impeded intra-Philippine trade and deters investment from foreign companies. The TRADE project played a critical role in the passing of the Competition Law, primarily because it provided support to the Philippines Government without overtly pushing an agenda or highlighting its USG funding. When supporting policy development and reform in another country, identifying the right champions to work with is crucial. Champions in this case included both the principal legislators and their key technical staff. One of the success factors cited in every interview with government and project staff was the understanding that because of the strong policy piece of the project, TRADE had to operate through local organizations and well-known Filipino staff, rather than operate outwardly as a USAID project. The team ensured that the legislators had a sense of full ownership throughout the process, and stayed in the background. A lessons learned paper written by the key players in passing the Competition Law would be a critical knowledge management piece for activities of this nature in other countries by highlighting success factors such as a well-known local team with respected technical prowess and existing relationships with the members of Congress integral to its passing. This kind of policy-level achievement, because of the political sensitivity, is not broadcast as widely as other trade￾specific results in success stories or public reports, but it is no less critical to improving trade capacity. The right team is likely the most critical component because of its effect on all support provided. According to the TRADE team’s experience, the team should be composed of people who collectively not only have expertise in the most important and relevant fields (in this case, law and economics), but also possess keen political instinct and genuine empathy for opposing points of view. USAID, government stakeholders, and project staff also cited the well-known, respected, and connected Chief of Party as the critical factor for success. Trust is one of the most important links in a policy project, compared with a project providing private sector-focused assistance, grants, or humanitarian assistance, where participants may receive a tangible and immediate benefit. For TRADE, building trust and rapport with the key players in Congress by staying out of the internal politics of the institution was important for its success. The TRADE project also highlights the appropriateness of output indicators where they help to tell the impact story of the work being implemented. Without the standard “Number of policies supported by USG” indicator, four years of the project’s work would not have been reflected as succinctly in the M&E system. This is not to say that the monitoring should stop there: now that the law is passed, the “real” work begins in terms of de jure implementation. Indicators reflecting outcomes should be integrated if the TRADE project continues to support the Philippines Government in implementation. Phased sets of indicators aligned with expected results can help to tell a richer story, starting with outputs and ending with outcomes. Herein lies the criticality of a well-defined theory of change and logical framework that adapts with an activity. BOX 6. PHILIPPINES TRADE POLICY CASE STUDY 15 | USAID 2017 TCB Evaluation 4. FINDINGS AND CONCLUSIONS: SUCCESS IN ACCOMPLISHING PROGRAM OBJECTIVES Question 2: To what extent have the interventions funded by USAID since 2009 succeeded in accomplishing the program’s objectives? The evaluation team relied on three sources of information explained in the methodology section of this report (see section 1.2.1 Methodology, data collection, and analysis) to answer this question, viz., e￾surveys sent to USAID mission staff, interviews with USAID headquarters staff, and document review. Evaluators also mapped activities from the document sample to a TCB Results Framework to provide additional insights. Overall, while the responses from USAID staff varied, all responses were positive. In general, respondents from USAID overseas missions were more positive about USAID’s success in meeting TCB program objectives than were staff located at USAID headquarters. The difference could be attributed to the methodology used in our evaluation: mission staff rated this question on a quantitative scale in the survey, while USAID headquarters staff expressed their opinions during interviews with evaluation team members. The evaluation team’s analysis of the document review also resulted in a numerical score, which reflected the overall positive finding of the survey. 4.1 PERSPECTIVES FROM USAID HEADQUARTERS STAFF The evaluation team sought an answer to Question 2 by interviewing USAID headquarters staff about the desired outcome of TCB work, and specifically how they know when their work has been successful and whether it is always possible to measure success. Two important concerns emerged from this discussion, both of which relate to monitoring and evaluation. M&E systems make determining “success” difficult. The first and dominant theme about success relates to M&E systems, and specifically to the difficulties encountered with these systems, such as: (1) inconsistent indicators and unreliable reporting of them, and often subsequent reliance on anecdotal information; (2) the understanding of the dynamics of trade and the building blocks that must be working well to notice success and therefore measure it; and (3) measurement challenges due to country-specific contexts. Trade agreements are often used to measure achievements. The second theme was the use of trade agreements to measure success, and the common belief that if a country nominally complies with an agreement, which is easy to “see” and therefore measure, it will succeed. However, reliance on measuring compliance with trade agreements does not mean that the nature (volume, value, composition) of the country’s trade has changed materially. Some countries have ratified agreements that they do not have the capacity to fully implement. In other cases, adherence to trade agreement provisions may not change the speed or efficiency of the movement of goods, which would be a true outcome indicator. On balance, we found that USAID headquarters staff believe that activities generally contributed to reaching TCB activity objectives, but that improvements can be made both in setting measurable outcome objectives and in measuring performance against those objectives. USAID 2017 TCB Evaluation | 16 4.2 PERSPECTIVES FROM USAID MISSION STAFF Rather than discussing Question 2 with USAID mission staff, the evaluation team posed this question directly in the survey, which resulted in a scored response (n = 21, M = 53, SD = 24.98). Our interpretation of the scores is that overall, respondents feel that activities made good progress in achieving the TCB activity goals as a result of USAID interventions (see Figure 4). 4.3 DOCUMENT REVIEW USING INDICATORS The evaluation team then analyzed the data from our document review per the six TCB “super categories.” Based on discussions with USAID headquarters staff as well as the findings of the 2010 evaluation, we hypothesized that the combination of private sector and public sector/policy activities would do better than private sector-focused activities that lacked a policy dimension. So the team broadened the analysis to examine the successes of activities by combining the policy “super category” with other individual “super categories.” We kept in mind that the number of activities when combined is greater than the number of activities when not combined, and several activities can be classified in more than one “super category” (therefore, double counting exists). While we attempted to validate our hypothesis with the available data, we found that the sample size is insufficient to draw firm conclusions. Nonetheless, Figure 5 appears to support our hypothesis in three out of four cases for which we have indicator data. Specifically, when combined with policy, activities in the “super categories” of agriculture, infrastructure, and “other” had higher scores than those without the policy element;12 perversely, however, the results differed in the case of private sector TCB support. Activities in the super category of private sector had higher scores on average when they did not include a policy dimension. (Note that we were unable to include the super category of trade facilitation in this analysis because the three activities classified in this super category lacked enough indicator information 11 Note: The online software used automatically rescaled the x-axis by weighting the responses that ranged from 1 to 3 and rescaling them from 0 to 100. 12 “Policy element” is defined as being recorded in the public sector super category previously defined. Figure 4. USAID Mission Perspectives on TCB Activity Impact Mission staff responses to survey question of “To what extent has the intervention succeeded in accomplishing TCB’s objectives of improved trade and investment performance in the country or region in which you are working?”11 0 1 2 3 4 5 6 7 Frequency Score 17 | USAID 2017 TCB Evaluation to analyze.) The sample size is much smaller in this evaluation than it was in the 2010 evaluation, so we are cautious about the interpretation of these results. 4.4 DOCUMENT REVIEW USING TCB RESULTS FRAMEWORK The evaluation team mapped the activities in the TCB Investment Dashboard sample to the TCB Results Framework (RF), which was based on the RF presented in the 2010 TCB evaluation and updated using the new 2016 USAID TCB policy (see Annex H). Team members consulted activity evaluations and indicators to assign each activity to one or more elements of the TCB RF, and determined whether the activity components relevant to the RF element met expectations or not. The assessment team members quantified this variable by assigning a score of 1, 2, or 3 to each activity (where 3 indicates that all the activity components supporting this element met or exceeded expectations, 2 indicates that all the activity components supporting this element partially met expectations, and 1 indicates that all the activity components supporting this element did not meet expectations). The following is a summary of high-level observations of our analysis:  The average score assigned to each RF element exceeded 2, indicating that the average TCB activity mostly achieved its performance expectations. Figure 5. Indicator Performance and Success Notes: (i) 'n' refers to the number of activities for which LOP indicator data are available; (ii) our sample included three activities in the facilitation super category, none of which had enough data to analyze. Red = Without a policy component Blue = With a policy component 0 0.5 1 1.5 2 2.5 3 Average Score on LOP Indicators USAID 2017 TCB Evaluation | 18  The RF element results cluster to which most activities mapped most frequently overall was RF 2.1: Export/Import Practices Improved (44 percent of activities mapped here), which also scored the highest of the three top-level clusters (2.46).13 In short, TCB activities that focus on import and export practices are the most common form and most likely to meet USAID’s performance expectations. Indeed, the most successful activities mapped the greatest number of components to this RF cluster. Within this cluster, the most mapped RF sub-element was Improved Business Practices (sector, firms, and industries) at 54 percent.  The RF element result cluster to which most activities mapped least frequently overall was RF 2.3: More Efficient Cost-Effective Movement of Goods and Services Across Borders (21 percent of activities mapped here), whose score is the lowest of the three top-level clusters but still indicates that activities mapped here mostly met USAID’s performance expectations (score of 2.35).  The RF 2.2: Trade-Related Public Sector Practices Improved results cluster had the largest range in scores (2.2-2.7) and averaged 2.39. RF Within this cluster, 2.2.2.1. Strengthened Institutional Capacity Development for Policy Formulation/Negotiation/ Implementation/ Enforcement was mapped to 51% of activities. 13Although the database of TCB activities used in the 2010 evaluation (Hageboeck, 2010a and 2010b) was significantly different than that used in the present evaluation, the greatest number of the 2010 activities (72%) also mapped to the RF 2.1 cluster. 19 | USAID 2017 TCB Evaluation 5. FINDINGS AND CONCLUSIONS: CHARACTERISTICS OF SUCCESSFUL TCB ACTIVITIES Question 3: Which activities have been more successful in achieving their objectives, and what were the primary factors responsible for their relative success? Following the same methodology as outlined in section 1.2.1 of this report (Methodology, Data Collection, and Analysis), evaluators analyzed the answers to open-ended queries in the interviews (see Annex C, questions 7 and 8) and survey (see Annex B, questions 8 and 9) to understand factors of perceived success as well as challenges at the activity level. In addition, the evaluators analyzed the TCB document sample and assigned a performance score to each activity (see Annex D, reference to question 3). In all three information sources (interviews, survey, and document database), the factors can be grouped into four different “families:”  Activity design (keys and barriers to success)  Activity management (keys and barriers to success)  Relationship development and management (keys and barriers to success)  External contextual factors (barriers to success only) According to survey respondents (USAID staff located in missions) and the evaluators’ analysis of TCB activity documents, the factors related to activity management were the most important of the three categories mentioned above. In particular, activities that were well managed in terms of achieving stakeholder buy-in, which were adaptable or had some form of flexible-response mechanism, and which maintained a roster of high-quality activity staff and consultants, tended to be the most successful. In contrast, for the USAID headquarters staff who participated in our interviews, activity design was the most important factor. That said, all three information sources (USAID overseas staff, USAID headquarters staff, and evaluators’ review of TCB documents) observed that relationship development with all types of stakeholders (government, private sector, civil society, and other donors) was the second most important factor of success (see Figure 6). 5.1 PERSPECTIVES FROM USAID HEADQUARTERS STAFF As noted above, the outstanding theme related to success factors in achieving TCB activity objectives as reported by the 15 USAID headquarters staff who participated in interviews relates to activity design. Examples of well-designed activities that were cited include those that ratified the WTO Trade Facilitation Agreement, and because of the activity’s ability to engage government partners in a high￾profile issue. Value chain activities in general were also noted as being successful in achieving activity objectives as a result of their design that improved linkages and resulted in a high return on investment. Other examples of success as they relate to activity design include the spinning off of local organizations to create non-governmental organization (NGOs) with private sector membership, and activities that have integrated firm-level assistance (or assistance to the private sector in general) with public sector regulatory reform. USAID 2017 TCB Evaluation | 20 USAID headquarters staff also attributed success to stakeholder relationships, although much less frequently than to activity design elements (three times versus ten times). Examples cited included success in achieving activity objectives in Central America as a result of USAID’s E-CAM program forming an effective relationship with Secretaría de Integración Económica Centroamericana (SIECA), which took many years to accomplish. The third theme (mentioned twice) that was revealed in interviews about success factors relates to activity management, and specifically to USAID staff at headquarters and in missions, and also to IPs. The comment related experience to success in achieving activity objectives: the more experience in terms of type and length of service that staff and IPs have, the higher the level of success. During the interviews, the evaluation team also offered participants the opportunity to discuss activities that they did not consider a success and the underlying reasons for those challenges. The distribution of comments about challenges in the three major categories was nearly the same as those about success; however, the reasons varied quite a bit between success and challenge factors. For example, in the broad area of activity design (10 answers out of 18), funding limitations was mentioned the most frequently. Without adequate funding, four participants commented, activities are unable to engage in “deep” customs reform, trade is not given a priority, and the design of Feed the Future activities require trade-related interventions to be much more niche (e.g., veterinary inspections, sanitary and phytosanitary measures) than the general transit work needed to create functioning and cost-efficient corridors. An additional three comments on barriers to success related to USAID’s long procurement process (see Box 7). Delays have important implications, especially for sector-based activities. By the time an activity begins implementation, the situation in the country likely will have changed, so the design is not necessary relevant. Also, managing regional activities is difficult because of the structure of USAID (viz., no person in the South Asia Bureau dedicated to regional trade activities), which can lead to lack of continuity after an activity ends. Figure 6. Overall Factors of Success, by Information Source BOX 7. OPINION ON BARRIERS TO SUCCESS IN ACHEVING TCB ACTIVITY OBJECTIVES “…Because of the procurement system and the fact that we are trying to create omnibus projects, it’s almost impossible to have success across the board, whether talking about trade hubs or EG [economic growth] projects in countries worldwide. The more broadly their scopes are drafted, the more difficulty they have in accomplishing what they are set up to do. We really don’t have an ability to be nimble and quick and respond in a narrower fashion. Even though we ideally would like to have multiple trade facilitation projects targeted at specific countries, we end up including TF as one of many components of a larger effort, which creates a challenging situation for those overseeing the project to pull it together in a way that results in positive outcomes.” ----USAID headquarters staff member Relationships USAID Washington (Interviews) Missions/IPs (Survey and Document Review) Activity Design Activity Management 21 | USAID 2017 TCB Evaluation Activity management barriers to success (three comments) focused on onerous reporting requirements of field staff that prevent them from directing their activities, the high turnover of Foreign Service Officers in countries such as Afghanistan and Pakistan that prevents learning about the country and its stakeholders, and the lack of technical expertise within USAID on trade that would improve interaction with contractors and government counterparts. Challenges about relationships were mentioned twice. Countries such as Bangladesh are challenging due to the multitude of donors implementing activities there, cited one participant. Another participant shared his concern about the desire of those who manage regional activities to be well accepted at the bilateral level, without infringing upon bilateral activities. Finally, three external factors were noted as barriers to success: economic and political instability that infringed upon reforming labor regulations: (1) Central America Free Trade Agreement (CAFTA) during the mid-2000s; (2) a difficult operating environment (India); and (3) technological challenges while trying to communicate across countries (El Salvador). 5.2 PERSPECTIVES FROM USAID MISSION STAFF Among the 22 responses received from USAID mission staff who participated in the survey regarding the primary success factor of a TCB activity, the most evident theme is linked to activity management (12 comments). Of those comments, seven relate to host government and stakeholder buy-in. To a much lesser extent, the high quality and skills of staff and external consultants was noted twice, as were concerted efforts to build the capacity of stakeholders; communicating best practices to beneficiaries was mentioned once. One best practice in capacity building approaches to export was highlighted by the Guatemala Rural Value Chains Project, described in Box 8. The second important theme relates to relationship development and management (6 out of 22 responses). Survey participants noted the success of leveraging the existing relationships and clout of key activity personnel, developing strong strategic alliances with influential private sector groups, and collaboration among Feed the Future activities. Lastly, success factors related to activity design (4 out of 22 responses) were included, notably adopting the value chain approach and leveraging international organizations as convening bodies for high-level officials. The evaluation team also asked the inverse question in the survey. The primary barriers to success relate to relationship development and management included in 11 of 24 comments) and external factors (included in 9 comments). Reasons cited for relationships as a barrier to success include poor donor coordination, the slow pace of implementation of partner, changes in leadership and competence of government personnel, and frequent transfer of host government focal persons. External challenges have to do with overall political and economic stability, crime, weak rule of law, and limited access to finance for SMEs. BOX 8. TAILORED ASSISTANCE TO EXPORT Anacafe, one of the implementers of the Rural Value Chains project in Guatemala, has found success in a new approach to working with coffee farmers. The team categorizes coffee growers into four groups by their level of readiness to export, thereby allowing them to tailor the assistance provided to the farmer, cooperative, or other entity. Farmers in the category of least readiness will not be linked with buyers, but will receive support for quality and growing practices; farmers in the highest category of readiness will attend trade shows, meet with buyers, and create sustainable linkages (which hopefully leads to sales). USAID 2017 TCB Evaluation | 22 Funding constraints were mentioned as the only activity management challenges (3 out of 24 comments). Finally, the lifetime of the activity was mentioned once as a flaw in activity design that challenged success. 5.3 DOCUMENT REVIEW The evaluation team conducted an inductive qualitative content analysis on the evaluation and audit narratives in order to identify the thematic concepts underlying activity accounts of success or challenge (for details, see section 1.2.1 on methodology). Tables 5 and 6 summarize the concepts that emerged from the content analysis, as well as the frequency at which these concepts are reflected across the sample of activities. In Table 6 (factors of success), 13 thematic concepts emerged representing keys to success across the activities. A few concepts were quite common and are reflected in the document narratives, as follows:  The activity’s ability to be flexible and responsive to context changes (adaptive management).  Making a concerted effort to develop local capacity (either through activities designed to capacitate certain stakeholder groups, or by prioritizing the hiring of local staff vs. ex-pats).  Developing and maintaining a positive and productive working relationship with government counterparts, other stakeholders including private sector, civil society, and other donors. The team also closely evaluated factors of success for all trade-related activities that made good progress in achieving TCB activity objectives or met or exceeded expectations (for details, see Table 5). Success themes are organized by the three broad categories of activity design, project management, and relationships, as above, and can be considered as lessons learned. Challenges to success-- Many of the concepts relating to barriers to success essentially reflect the “other side the coin” of the success themes. The following challenges relate to the influence of specific contextual realities, by theme.  Activity design: Activity lifespan was too short; some activity components are duplicative and/or are poorly aligned with work of other donors.  Activity management: Funding and resource constraints inhibit meaningful work; inadequate communication or information flow.  Relationship development and management: Key stakeholders, beneficiaries, or partners lack capacity to be effective or act as intended.  External contextual factors: Local or regional political dynamics (balance of power, elections and changes in government, corruption, bureaucracy); environmental challenge (disease, climate, natural disaster. 23 | USAID 2017 TCB Evaluation TABLE 5. DETAILS OF KEYS TO SUCCESS: THEMES PRESENTED IN NARRATIVES OF TCB DOCUMENT SAMPLE No. of Activities % of Sample ACTIVITY DESIGN Activity model is strong 10 14% Activity components introduce innovations that meaningfully fill gaps and solve problems 8 11% Activities takes a demand-driven approach and/or customizes to local context 4 6% Activity builds off of previous or other existing activities 3 4% ACTIVITY MANAGEMENT Concerted efforts to establish, partner with, or enhance local entities and/or build capacity of local staff to promote buy-in and sustainability 18 25% Flexibility and responsiveness to capitalize on or adapt to context changes (positive or negative) and/or failures in activity assumptions 18 25% Effective activity management 16 23% High quality and skilled activity staff or consultants 9 13% Activity used strategic communications media to build support and buy-in 6 8% RELATIONSHIP DEVELOPMENT Activity has effective and positive relationship with key stakeholders other than USAID (private sector, civil society, other donors) 29 41% Activity has effective and positive relationship with host government stakeholders 19 27% Activity fosters linkages and relationship building between key stakeholders (promotes sustainability) 14 20% Effective and strong relationship with USAID 3 4% USAID 2017 TCB Evaluation | 24 TABLE 6. SUCCESSFUL TRADE-RELATED ACTIVITIES: DETAILS OF KEYS TO SUCCESS PRESENTED IN TCB DOCUMENTS ACTIVITY DESIGN  Trade agreements or organizations (e.g., WTO) have influenced government counterparts to make significant progress in reforms.  A phased approach to assistance that allows beneficiaries to take on increased levels of risk as they try new approaches and cost sharing lead to stakeholder involvement and sustainability.  Using an evidence-based strategy and data helps monitor successes. ACTIVITY MANAGEMENT  Physical proximity to USAID: Moved office to Reagan Building to have close contact with USAID staff.  Seek opportunities for quick wins, such as reforming administrative procedures, to foster significant impact with approval below Parliament and Cabinet.  Concerted efforts to establish, partner with, or enhance local entities and/or build capacity of local staff to promote buy-in and sustainability.  High quality and skilled activity staff or consultants who helped government counterparts to move forward to agreed-upon support initiatives and action plans.  Strong partnerships with government counterparts allow broad participation and buy-in across relevant government stakeholders.  Successful activities require the "proper mix" of private sector involvement, government engagement, policies and services, donor support, and beneficiary/stakeholders.  Alliance-building approach can drive multiple-stakeholder initiatives. RELATIONSHIPS  Networking is a valuable asset for sustainability of future USAID interventions.  Facilitating collaborative efforts to improve trade industrial policy, investment, export policy, procurement, and particularly investor aftercare.  Identifying and collaborating with right partners, unifying all implementing partners with common goals, channeling the resources of existing practitioners with strong foundations instead of "reinventing the wheel".  Form partnerships with private sector through grants or some type of formal mechanism whereby partners have contracts that define and regulate their relationship to activity.  a/ The evaluation team tested for the influence of internal synergies of TCB activities on success, as measured by indicator performance. Results for all activities in our TCB document sample suggest that scores are higher for activities that mentioned internal synergies than those without synergies. Specifically, activity documents that mentioned positive synergies (n=30) averaged 2.40 on their indicator scores; activities that did not specifically mention internal synergies in their documents (n=25) scored an average of 2.24; and activities that had conflicting reports on internal synergies (n=3) scored 1.60 on average. The 13 activities for which there was not enough information to score were removed from this analysis.  Note: The findings in this table represent key success factors for 12 trade-related activities that scored 2 or 3 on evaluations based on indicator performance or document review. 25 | USAID 2017 TCB Evaluation 6. FINDINGS AND CONCLUSIONS: INTEGRATING M&E INTO TCB ACTIVITIES Question 4: How can USAID integrate monitoring and evaluation into the design and implementation of TCB programs more systematically? In order to make recommendations to USAID on how to strengthen M&E into TCB, the evaluators included M&E- and indicator-specific questions in the survey to Mission staff, queried Washington-based staff working on trade as to the utility of current approaches during interviews, and carried out an extensive desk review of the indicators used currently in activities (both standard and custom) to assess depth as well as breadth. The results of this data collection and analysis are presented below. The following section presents the findings and conclusions that, paired with findings in other evaluation question sections, provide the foundation for the recommendations for integrating M&E more effectively. 6.1 DOCUMENT REVIEW AND SECONDARY SOURCES Quantitative performance indicators are a key source of monitoring and evaluation data and are used by implementing partners and USAID mission and headquarters staff to report on activity and portfolio performance and impact. While achieving the ideal balance of aggregation and contextual nuance in indicators continues to be a work in progress, USAID has made progress in developing and standardizing trade capacity building indicators and more systematically investing in M&E. For each activity within the document review, the evaluation team identified the number of indicators, disaggregated by standard (F indicators or other standardized indicators such as those developed for Feed the Future projects) versus custom. When available in the documents, the team listed the indicators in order to develop a comprehensive listing of indicators used. For seven activities, the documents available through the DEC did not provide a list of activity indicators. Eighty percent of the sample (51 activities of 71) reported on both standard and custom indicators with an average of 10.4 custom indicators and 9.4 standard indicators per activity.14 Across the sample, activities reported on over 1,000 indicator datum, (556 standard and 631 custom). This count is based on the total number for each activity, even though some activities are reporting on some of the same standard indicators. While a benefit of standard indicators is their ability to be aggregated, because the TCB portfolio is so cross-cutting, the range of standard indicators reported in this sample remains quite broad. The most frequently used standard indicators across activities fall within the agricultural indicator category, likely influenced by the Feed the Future activities that receive TCB funding and are required to report on such indicators. Some of the most frequently utilized indicators in our sample are reported in Table 7. 14 The average for custom indicators removed the outlier of CAFTA-DR Environmental Chapter Requirements whose evaluation report listed 57 indicators, but did not list out the indicators. This prevents the evaluation team from verifying the standard vs. custom split. USAID 2017 TCB Evaluation | 26 Custom indicators also span the range of programming. Within the document review sample, the evaluation team was able to specifically identify 436 custom indicators.15 To provide further clarity on the types of indicators developed, the team sorted the indicators into nine larger non-mutually exclusive categories based on the super categories previously defined (for examples, see Box 9). The breakdown across these categories is provided in Figure 7. The universe of custom indicators within this sample follows the composition of the sample itself, which is reflective of the fact that nearly all activities developed some number of custom indicators to best tell the narrative of that particular activity. In an effort to best capture the impact of a program through quantitative measures, USAID activity managers are striving to develop outcome indicators, which could ideally be utilized across projects. That said, recognizing the difficulty of creating comprehensive outcome indicators, a potential secondary option is to sequence indicators (possibly a mixture of output and outcome) to develop a quantitative narrative of success during an activity’s lifetime. For example, if a significant part of an activity is to provide training and capacity building, the “Number of person hours of training” can be a helpful first level indicator of outreach. However, to only report on that indicator leaves out the heart of what the activity is striving to achieve. It may be relevant in designing the indicators to determine what skill the training is trying to impart that is critical for the participants to utilize in order to achieve the goal of the activity. For example, if the activity is aiming to provide capacity building to customs personnel to streamline processes and move goods across that particular border crossing faster, a phased indicator approach might be: 15 The number is lower than the 631 previously listed because some project documents listed the number of indicators, but did not provide a complete breakdown of the indicators themselves. TABLE 7. SAMPLE OF FREQUENTLY USED STANDARD INDICATORS INDICATOR CODE INDICATOR DESCRIPTION EG 3.9 Number of full time equivalent (FTE) jobs created with USG assistance EG 3.2-23 Value of targeted agricultural commodities exported with USG assistance EG 3.2 – 22 Value of new private sector capital investment in the agriculture sector or food chain leveraged by Feed the Future implementation EG 3.2-18 Number of private enterprises, producer organizations, water users associations, women's groups, trade and business associations and CBOs that applied new technologies or management practices as a result of USG assistance EG 3.2-5 Number of public-private partnerships formed as a result of USG assistance BOX 9. EXAMPLES OF CUSTOM INDICATORS WITHIN THE SAMPLE Trade Accession and Facilitation for Afghanistan:  # of exporters using services of the one-stop shop  #of new contracts between local producers and car vendors  # of steps eliminated at inland clearance depots and border crossing points Peru-Andean Trade Capacity Building Project  # of labor disputes resolves by the Judiciary  Cost to register a business in a local municipality  # of intellectual property applications granted Nigeria Expanded Trade and Transport Project  Reduction in the cost to move goods between Kano and Lagos as a result of US assistance El Salvador Improved Access to Employment  # of trainees who continue studies after training 27 | USAID 2017 TCB Evaluation Number of person hours of training  Number of customs personnel who complete the entire training course  Number of customs personnel who report implementing best practices from training in employment  Decreased time to move shipments through that customs post Similarly, an activity aimed at improving regional integration through a policy level approach might design an indicator narrative such as: Number of regional trade related legal acts advanced through the legislative process with activity support  Number of launched or signed initiatives on regional integration  Number of rulings issued in compliance with regional agreements. An activity may not report on all the parts of this indicator chain, but the process of building it makes explicit how the activity’s projects conceptualize working towards a trade-related goal. Figure 7. Custom Indicator Categories 6.2 PERSPECTIVES FROM USAID STAFF Data gathered through interviews with USAID headquarters staff and survey responses from USAID mission staff further highlight the tension between reporting on standard indicators that can be broad enough to aggregate across activities versus contextualized custom indicators that can be designed to specifically capture that activity’s unique goals and interventions. While interview respondents often expressed frustration with standard indicators such as number of person hours of training, survey respondents cited the same indicator as one of the most relevant and useful of the standard indicators. Concerns were also raised about the recent changes to the standard indicators that can occur mid￾activity. One standard indicator that was recently dropped by the State Department Office of U.S. Foreign Assistance Resources—the number of commercial laws and regulations simplified and implemented in accordance with international standards as a result of USG assistance--was cited by one survey respondent as incredibly relevant as it measured a necessary starting point for future reforms. On the other hand, the USAID Economic Growth Standard Indicator Working Group dropped this indicator because of low reporting from Missions globally. 27% 19% 18% 14% 10% 5% 3% PRIVATE SECTOR ENGAGMENT 3%1% IMPROVE PUBLIC SECTOR PRACTICES CAPACITY BUILDING AND TRAINING EXPORT/IMPORT IMPROVEMENT AGRICULTURE MEDIA AND KNOWLEDGE MANAGEMENT CUSTOMS GENDER ENVIRONMENT USAID 2017 TCB Evaluation | 28 There is also variation among activities and missions in indicator use and the extent to which learning is occurring in place of standard reporting practices. According to the survey, indicators are least often used to affect activity design (64%) and form mission strategy (68%) and most often used for reporting and accountability (100%), making management decisions (74%) as well as conducting portfolio reviews (86%). Comments from the respondents were mixed, but typically agreed that F indicators are not useful as a means to understand project success and results because they are typically too high level to be attributable to activities, or because they do not allow for nuances at the activity level. The recently approved smaller set of F indicators (2016) seem to allow for more useful attribution to activities and outcomes, but this falls outside of our study period and therefore cannot be assessed. While progress has been made in M&E for trade with the development of the ProjectStarter online Monitoring, Evaluation, and Learning Toolkit and the Trade Indicator Handbook, in addition to a more substantive set of standard F indicators in the most recent revision, more collaborative work needs to be done with Regional and Pillar Bureaus and mission points of contact for trade activities and project design to standardize M&E approaches for monitoring trade outcomes more systematically, especially in projects that are deemed “mostly” or “exclusively” trade focused. The Trade Indicator Handbook, while useful, does not limit the number of recommended standard indicators at each expected results level, and therefore does not support aggregation. When asked in the survey whether the M&E system currently helps USAID mission staff determine whether an activity is successful, on-track, and meeting objectives, the overwhelming majority (91%) felt that their M&E system helped them to determine whether an activity is on track, but only 63% felt that the system was helpful in determining activity success. The responses were mixed regarding the utility of M&E systems. When asked how useful the M&E systems were in the missions, survey respondents averaged 2.7 on a scale of 1 to 5 (where 5 is extremely useful and 1 is not useful at all). A few of the survey respondents said they are not tracking trade indicators because there is no direct attribution to trade in their activities. Sixty-two percent of survey respondents felt that the old Trade Results Framework that is still used by some stakeholders is helpful or extremely helpful. Without a revised theory of change or TCB results framework from the 2010 evaluation and its TCB results framework, it is difficult to know which components of a project with a small focus on trade are hypothesized to promote trade capacity, and therefore impossible to confidently estimate contribution to trade capacity in evaluation question 1. Based on interviews and the survey responses from USAID mission and headquarters staff, the evaluation team found that well-designed PMPs allow for performance tracking and learning, and that poorly designed performance management systems (with an emphasis on outputs and indicators unaligned to expected results) are a burden rather than a support. In the absence of clear theories of change and active learning plans, M&E systems can become repositories of indicators instead of learning tools. When the emphasis is on project monitoring (or management) in order to learn about what is working, make modifications, and implement more effectively in lieu of performance monitoring or, which focuses more on reporting, but where learning and utilization is not necessarily emphasized (the former is in line with the revised ADS), the utility increases. As shown in Boxes 6 and 10, Philippines TRADE offers a best practice example in how to align the Mission’s CDCS with the activity’s logical BOX 10. LESSONS IN POLICY FROM PHILIPPINES TRADE The Philippines TRADE project based in Manila highlights the criticality of output indicators where appropriate. The team spent nearly five years working with the GOP on the passing of its Competition Policy in 2016. While the standard F indicator “number of policies, laws, regulations passed” says nothing of de jure implementation, the “simple” and time consuming act of passing the law is something USAID and the TRADE team highlight as a significant achievement of TRADE. 29 | USAID 2017 TCB Evaluation framework, work plan, and indicators (as well as an appropriate use of output indicators to track performance). When asked how to incentivize a standardized approach to trade M&E, one survey respondent suggested that project and activity design teams “consult actual CORs/AORs and project staff for which indicators to use; again, the more useful the indicators are, the more they will be used (that is the incentive).” Underlying the utility of any M&E system is the ability of USAID and their implementing partner to identify the most pressing information and management needs over the life of the contract. They must also clearly define objectives and expected results, critical assumptions, and relevant indicators (standard or custom) that are feasible, reliable, and timely. This allows the mission and IP to collect and feedback data in time and on budget to make critical management decisions and track performance. This is a difficult task, but USAID is committed and making headway. Following the 2010 Aid to Trade Evaluation, USAID developed several guides and learning tools to assist with M&E across the TCB portfolio. These include the Trade Indicator Handbook and the USAID ProjectStarter online toolkit. When asked through the survey about the usefulness of these materials, only one respondent in a USAID mission indicated that they were not useful. However, 43% had not used ProjectStarter and 24% were not aware of the resource. Similarly, 24% had not used the Trade Indicator Handbook and 14% were not aware of it. Those who have used these resources appear to find them helpful, but further efforts can be directed into disseminating and strengthening the quality of these resources to maximize their effectiveness. Some additional insights regarding ProjectStarter’s utility have been gathered through a user survey currently available on the website’s homepage. Users identified the utility of guidelines and templates, the results framework information, and the monitoring and evaluation tools as some of ProjectStarter’s most useful features. However, users also commented that the website could be improved with some technical adjustments to file formatting, sharing more information on formulating impact indicators, providing online courses, and updating all materials to match ADS guidelines.16 Another notable ongoing effort is the development of the African Competitiveness and Trade Expansion (ACTE) African Trade Hub Project Management Plan that developed a common RF, core indicators, and data collection methods across the three Trade and Investment Hubs. The Hubs represent a significant focus of the TCB portfolio and this PMP offers an opportunity to compare and report on results at a higher collective level across the three regions in the areas of increased capacity of targeted non￾agricultural sector firms and associations, improved regional trade and investment enabling environment, more efficient and cost effective movement of traded goods across borders, and advanced regional trade and investment agreements. Although the development of this collaborative PMP was a long and difficult process, it speaks to the ability of USAID to develop synchronized monitoring systems across regional contexts. 16 Updating ProjectStarter guidance and templates to match the current ADS guidelines is currently ongoing under the iBEAM project. USAID 2017 TCB Evaluation | 30 7. RECOMMENDATIONS While answering the research questions given, the evaluation team identified key recommendations that were then grouped into four main categories: (1) activity design, (2) implementation, (3) monitoring and evaluation, and (4) capacity building, knowledge management, and learning. 7.1 ACTIVITY DESIGN Implement guidance under ADS 201 and 205. During the design phase of activities that include TCB elements, it would be highly beneficial to articulate explicit theories of change (pursuant to ADS 201 guidance), develop trade-related logic frameworks, and identify monitoring indicators that focus on measuring progress toward the desired trade-related development outcomes, as well as on measuring USAID’s program outputs. Employ adaptive management techniques. During the planning stage of an activity, consider the possibility of things going wrong and plan for contingencies or unexpected developments (adaptive management). Examples include considering how the activity might need to change in the event of a natural disaster or political-security threat. Integrate alternative plans into the design and contracting process or have these conversations with the IP at the beginning of implementation. Use M&E learning tools. Emphasize M&E in new trade SOWs as a management and learning tool, rather than merely as a reporting requirement. Include adequate funds for M&E in contracts. Ensure that activity and contract budgets reflect appropriate levels of M&E and learning activities. 7.2 IMPLEMENTATION Develop effective stakeholder relationships. Effective relationships among host governments and key non-government key stakeholders with USAID and IPs clearly make a difference in terms of success. The time and energy required to develop those relationships, which often includes providing targeted technical assistance to create formal relationships, are worth the effort to achieve buy-in and foster sustainability. Develop a clear and common understanding of roles and responsibilities among USG Agencies involved in TCB. A best practice is to ensure that everyone working on a given TCB activity understands the role and the value of all interagency players, even though they see trade through different lenses. This theme of establishing a common understanding follows through to the suggested sharing of annual work plans and performance measures prepared by IPs with all agencies that have a stake in the country or region. Tapng the experience of regional institutions. Regional TCB activities can benefit from collaborating with regional institutions (such as SIECA in Central America), where such institutions exist and are effective. Regional institutions can help coordinate actions and priorities at the country level, co-sponsor capacity building events, and be the focal point for multi-donor collaboration. Coordinate bilateral and regional activities. Improved coordination and alignment of bilateral and regional activities by USAID staff and IPs can magnify the positive the impact of these activities. 31 | USAID 2017 TCB Evaluation 7.3 MONITORING AND EVALUATION, AND CAPACITY BUILDING Expand use of outcome indicators. USAID headquarters staff should promote adoption of outcome- and output-level indicators across TCB activities. For all TCB activities, include outcome-level, knowledge-based training indicators. As a corollary, USAID/E3/TRR could conduct research on how to measure trade-related capacity (based on their theories of change) and develop model Performance Indicator Reference Sheets (PIRS). Integrate knowledge management and learning resources into USAID platforms. Update the Trade Indicator Handbook to include a more selective list of outcome-focused standard indicators and improve access to and awareness among USAID staff of this document. Train USAID staff on foundational elements of M&E. USAID mission staff overseeing TCB activities would benefit from training by TCB experts within USAID on the theory of change, logical frameworks, learning plans, M&E methodologies, and ProjectStarter. Improve TCB indicators by incorporating those from Feed the Future and trade and investment hub programs. Integrate and align TCB indicators and data collection methodologies with the Feed the Future Monitoring System and/or the model PMP developed for USAID’s three trade and investment hubs in Africa. USAID/E3/TRR will benefit from continued collaboration and knowledge sharing with USAID’s Africa Bureau and with its Bureau for Food Security, given that many TCB activities are part of a larger Feed the Future-funded activity, and also given that three of USAID’s largest TCB activities are the trade and investment hubs in Africa. 7.4 KNOWLEDGE MANAGEMENT AND LEARNING Promote MEL approaches for trade-specific activities. Disseminate best practices, syntheses of evaluations, and harmonized but custom outcome- and output-level indicators, through USAID KM platforms, including Learning Lab, ProgramNet, or ProjectStarter. Improve the quality of ProjectStarter as a learning tool.  Integrate more learning resources into USAID KM and Learning platforms (Learning Lab, ProgramNet, ProjectStarter). These resources, as mentioned above, can include evaluation syntheses, research, best practices and lessons learned documents.  Since ProjectStarter is being integrated onto a usaid.gov site and is being emphasized as a learning tool for trade-related MEL, when discussing it, emphasize its use as a project design, performance management, and learning tool instead of just a MEL toolkit. It is useful for both, but the former emphasizes activity design and implementation instead of a toolkit accessed infrequently for ad hoc M&E requests. (The revised ProjectStarter will include a synthesis of evaluation findings and recommendations by sector and region so that Missions and DC can more easily access information on past activities.)  ProjectStarter can also highlight more practical resources on the home and landing pages: this and the 2010 meta evaluation; E3’s sectoral synthesis on TRR and gender; the Trade Indicators Handbook; and TCB results framework. Easy access to these resources in a more highlighted fashion will improve activity design, integration of M&E, strengthened approaches to learning across the program cycle, and time savings because of easy access to a more standard approach to MEL. Document Review Sample Because of the reduced scope and period of performance of this evaluation it was necessary for the evaluation team to review documents from a systematically-constructed sample of activities drawn from the USAID TCB Database. The Data Universe To construct the sample of activities for review, the evaluation team first defined an evaluation-specific data universe. To do this the team first conducted a query of the public TCB Database for all activities that were implemented and funded by USAID between 2009 and 2014.1 The team then manually integrated the dataset provided by USAID/E3-TRR for all 2015 records. 2 The team considered the resulting dataset to be the data universe for this evaluation; it included 563 activities, comprising 3,298 records, and $3,213,394,776 USD in TCB obligated funding. Activities in the sample averaged $5,861,775 USD in total TCB funding, ranging from $113 million to $2,500 USD. The Sample Frame Upon examination, the team determined that some elements in the data universe would need to be removed in order to create an appropriate sample frame. Several criteria were applied to remove activities, including: any records for Participating Agency Service Agreements (PASAs); Participating Agency Program Agreement (PAPAs); general pools of institutional funding or contributions to initiatives (such as the African Development Fund (ADF), the Multidonor Trust Fund (MDTF), Common Market for Eastern and Southern Africa (COMESA), the East African Community (EAC), the African Union (AU), Wildlife Enforcement Network (WEN), etc.); activities whose descriptions indicated the funding was primarily for mission support or staffing; activities whose descriptions indicated they were discrete data and analysis products (i.e. stand-alone analyses or evaluations that are presumably used as inputs at the mission level and are not a part of a larger research program/project); activities whose descriptions indicated that they were "TBD"3 ; and discrete activities determined by the evaluation team’s Senior Trade Advisor to be inappropriate for inclusion in the sample fame.4 The resulting sample frame consisted of 466 activities, comprised of approximately 2,750 records and $2,813,472,552 USD in TCB Obligated Funding. Activities in the sample averaged $6,198,517 USD in total TCB funding, ranging from $87 million to $2,500 USD. Initial Sample Selection To draw a sample from the sample frame for document review the evaluation team firstsorted the sample frame by total TCB funding amount in descending order to ensure that our sample would include all of 1 Although the TCB Database includes activities funded and/or implemented by other USG agencies, the scope of this evaluation was limited to only those activities that were both funded and implemented by USAID. 2 At the time of our initial query in September 2016, the publically available TCB database only included records for activities through 2014. USAID subsequently provided data for all records for 2015 that were not yet publically available. The evaluation removed activities from the integrated dataset that appeared to have been started in 2015 (i.e. TCB funding information was provided only for 2015), because it was assumed that these activities would not have enough information available to be responsive to the evaluation aims and purposes. 3 Records for activities whose descriptions they were “TBD” seemed to not be fully planned or in implementation stage at the time they were entered into the TCB database. The evaluation team concluded that these were placeholders in the database and should be removed. 4 Discrete removals flagged by the team’s Senior Trade Advisor included BEAM and iBEAM (conflict of interest), funding for the World Bank annual “Doing Business” report (discrete analysis), Worldwide Support for TCB (general funding), the EAT and BizClir Projects, and the Juba-Nimula Road (considered an outlier, at over $113 million in funding this activity dwarfs all other activities in the universe and had little documentation available). Annex A. Summary of TCB Investment Dashboard and Sample USAID 2017 TCB Evaluation | 32 the largest activities, in terms of the total funding amount reported in the TCB database. The initial sample consisted of 202 activitiesrepresenting 2,076,009,229 USD or 74% of the total TCB funding for the sample frame. 5 Activities in this sample averaged $10,277,273 USD in total TCB funding and ranged from $87 million to $300,000 USD. Prior to starting the document review, it was first necessary to collect and obtain documentation necessary to conduct a review of each activity in the sample. Evaluation team members consulted the USAID Development Experience Clearinghouse (DEC), a broader Internet search, DEC staff, and project implementers to compile the documentation needed to complete the document review. The evaluation team hypothesized that, at a minimum, sufficient documentation needed to complete an activity review would consist of: a full or almost-complete set of annual reports, a final report (if the activity was complete by 2015), and any available evaluation reports (midterm or final). 6 Despite concerted efforts to obtain documentation from a variety of sources over the course of eight weeks, sufficient documentation was available for relatively few activities in the initial sample. Just 18.3% of activities had fully sufficient or sufficient levels of documentation as shown by the table below: Documentation Level Number of Activities Percent of Initial Sample Insufficient (none of the key documents sought were available/obtainable) 120 59.4% Marginally Sufficient (some key documents sought were available/obtainable) 45 22.2% Sufficient (all annual reports available along with a final report OR one or more evaluation reports) 14 6.9% Fully Sufficient (all annual reports available along with a final report AND one or more evaluation reports) 23 11.4% Final Sample Selection Because of significant documentation limitations, the sample of activities that was ultimately reviewed in detail was smaller than the initial sample drawn from the sample frame. The evaluation team sought to purposively construct a final sample that would reflect key characteristics of the overall sample frame, while also including activities that were actually feasible to review. The key characteristics that the evaluation team considered when constructing the sample included: the distribution of activities across five, broad activity type “super-categories” and the distribution of activities across geographic regions.7 Determination of the final document review sample was primarily driven by the sufficiency of the available documentation. Any activities for which no documentation had been identified or obtained were 5 For the initial sample, the evaluation team sought to draw a sample that would be comparable, in terms of the proportional of total TCB funding represented by the activities selected, to the first TCB portfolio evaluation completed in 2010. The sample used for that evaluation represented 75% of total TCB funding. 6 In some cases activity implementing partners reported on a quarterly, not annual basis. Where this was the case the team substituted quarterly reports for annual reports. 7 The “super-categories” the evaluation team used for balancing the sample against the sample frame were USAID 2017 TCB Evaluation | 33 created by grouping the more granular activity types used in the in the USAID TCB database into five higher-level categories. The resulting “super-categories” include: Private Sector TCB, Public Sector TCB, Trade Facilitation, Trade-Related Agriculture, Trade-Related Infrastructure, and Other Trade Capacity Building. See Annex A1 for a crosswalk between super-categories and the granular activity type labels used in the USAID TCB Database. excluded from the sample; all activities with fully sufficient or sufficient documentation were automatically included in the sample. To obtain a better, more balanced sample, activities with marginal￾levels of documentation were purposively selected for inclusion on the basis their potential to improve the sample’s reflection of key sample frame characteristics. 8 Evaluation team members continued to seek out documentation for the marginally-sufficient activities that were included in the sample, to improve the quality of the review that could be conducted for these activities.9 Sample Characteristics Using the approach described above, the evaluation team was ultimately able to construct a sample of 75 activities totaling $960,270,546 USD or 34% of the sample frame funding total. Activities in this sample had an average value of $12,803,607 USD range from $87 million to $503,000 USD. Overall, the sample reflects the key characteristics of the sample frame reasonably well, especially in terms of geographic distribution and the overall proportion of TCB funding dedicated to each TCB super-category. The sample does less well in reflecting the distribution across super-categories by the proportional number of activities with each super-category designation. On the whole, the proportional representation of super-categories among activities is higher in the sample. The reason for this is partially due to the fact that there is a higher proportion of sample activities with multiple super-category designations, which indicates that the sample activities may be more “trade intensive” than the activities in the overall sample frame. Balancing on the basis of mean and median super-category funding by activity was also more difficult to achieve; in general the activities in the sample tend to have higher super-category funding levels than the activities in the overall sample frame. 8 Since the number of available documents varied for the activities deemed to be “marginally sufficient” by evaluation team members, the evaluation team also prioritized the inclusion of activities that had the most “complete” documentation relative to other marginally-sufficient activities. 9 In some cases these efforts were fruitful and resulted in an “upgrade” of that activity to sufficient documentation status. In other cases evaluation team members were unable to obtain any further activity documentation and completed the activity review with the information that was available. USAID 2017 TCB Evaluation | 34 Annex A2 provides specific details on how the document review sample balances against the sample frame. Evaluation Super-Category TCB Database Activity Type Private Sector TCB • Trade Promotion • Enterprise Development • Financial Sector • Trade-Related Tourism • Trade-Related Services (excluding. tourism) Public Sector TCB • Competition Policy, Business Environment, and Governance • FTAs and Trade Integration • Environmental Standards and Trade • WTO Accession/Compliance • Trade-Related Labor • Trade-Related Procurement • Intellectual Property Rights Trade Facilitation • Technical Barriers to Trade • Customs Operations • Sanitary and Phytosanitary Standards (SPS) Trade-Related Infrastructure N/A Trade-Related Agriculture N/A Other Trade Capacity Building N/A USAID 2017 TCB Evaluation | 35 Annex A1: Crosswalk of Evaluation Super-Categories to USAID TCB Database Activity Categories Note: In most cases, a “good” sample balance was achieved when the difference between the sample frame and the sample is +/- 5% or less, and is denoted with green shading. Yellow shading indicates where the balance is relatively close to the threshold; red shading indicates a larger imbalance. For mean and median funding level comparisons a “good” sample balance was achieved when sample statistic is +/- 120% of the sample frame statistics and is denoted with green shading. Yellow shading indicates where the sample statistic is +/- 121-150% of the sample frame statistic; red shading indicates a where the sample statistic is greater than 150% of the sample frame statistic. TCB Super-Category Balances Table 1: Percent of Aggregate TCB Funding Dedicated to TCB Super-Category TCB Super-Category Sample Frame (n=466) Document Review Sample (n=75) Funding % of Funding Funding % of Funding % Difference from Frame Private Sector TCB $ 921,095,992 33% $ 301,995,984 29% 4% Trade-Related Agriculture $ 736,115,646 26% $ 272,151,256 26% 0% Public Sector TCB $ 650,087,314 23% $ 290,091,887 28% -5% Trade Facilitation $ 198,427,412 7% $ 100,327,158 10% -3% Trade-Related Infrastructure $ 287,824,548 10% $ 55,983,038 5% 5% Other Trade Capacity Building $ 94,982,980 3% $ 24,577,574 2% 1% The sample achieved an optimal balance in terms of the percent of the aggregate-level TCB funding dedicated across TCB super-categories. The super-categories with the most activity funding in the sample and sample frame were Private Sector TCB, Trade-Related Agriculture, and Public Sector TCB. Other Trade Capacity Building was the least intensive category represented. USAID 2017 TCB Evaluation | 36 Annex A2: Document Review Sample vs. the Sample Frame Table 2: Number of Activities with Super-Category Designation TCB Super-Category Sample Frame (n=466) Document Review Sample (n=75) # of Activities % of Activities # of Activities % of Activities % Difference from Frame Private Sector TCB 260 56% 45 60% 4% Public Sector TCB 228 49% 48 64% 15% Trade-Related Agriculture 180 38% 33 44% 6% Trade Facilitation 99 21% 26 35% 14% Trade-Related Infrastructure 68 15% 13 17% 3% Other Trade Capacity Building 64 14% 15 20% 6% The sample achieved an optimal balance in terms of the proportional number of activities distributed across two super-categories: Private Sector TCB and Trade-Related Infrastructure. The sample had a much higher proportion of activities with the Public Sector TCB and Trade Facilitation designation. Table 3: Mean Activity Funding Level by TCB Super-Category TCB Super-Category Sample Frame (n=466) Document Review Sample (n=75) Mean Activity Value Mean Activity Value % Difference From Frame Trade-Related Agriculture $ 4,089,531 $ 8,247,008 202% Trade Facilitation $ 2,004,317 $ 3,858,737 193% Private Sector TCB $ 3,542,677 $ 6,711,022 189% Public Sector TCB $ 2,851,260 $ 3,327,517 117% Other Trade Capacity Building $ 1,484,109 $ 1,638,505 110% Trade-Related Infrastructure $ 4,232,714 $ 4,306,388 102% The sample achieved an optimal balance in terms mean activity funding levels for the Public Sector TCB, Other Trade Capacity Building and Trade-Related Infrastructure Activities. Relative to the sample frame, the document review sample contained larger projects that fell into the Trade-Related Agriculture, Trade Facilitation, or Private Sector TCB super-categories. Mean funding levels for the activities falling into these categories was about twice the mean levels for similarly categorized activities in the sample frame. USAID 2017 TCB Evaluation | 37 Table 4: Median Activity Funding Level by TCB Super-Category TCB Super-Category Sample Frame (n=466) Document Review Sample (n=75) Median Activity Value Median Activity Value % Difference From Frame Trade Facilitation $ 787,735 $ 2,266,263 288% Other Trade Capacity Building $ 600,000 $ 855,000 143% Private Sector TCB $ 1,499,608 $ 1,499,215 100% Public Sector TCB $ 1,057,821 $ 1,041,641 98% Trade-Related Agriculture $ 1,080,199 $ 1,048,066 97% Trade-Related Infrastructure $ 1,071,645 $ 1,043,290 97% The sample achieved a better balance in terms median activity funding levels most of the TCB activity type super-categories. The main exception is that the median activity funding level for the Trade Facilitation category was much higher than the median level for the sample frame. This is likely due to the fact that there are more “low value” activities with Trade Facilitation categorization in the sample frame than in the sample. Table 5: Super-Category Concentration Among Activities TCB Super￾Category Concentration Sample Frame (n=466) Document Review Sample (n=75) # Activities Percent # Activities Percent % Difference from Frame 1 Category 248 53% 28 37% -16% 2 Categories 97 21% 19 25% 5% 3 Categories 62 13% 12 16% 3% 4 Categories 30 6% 7 9% 3% 5 Categories 21 4% 4 5% 1% 6 Categories 7 1% 5 7% 5% The sample was generally balanced when it came to super-category concentration among activities, with one exception. In comparison to the sample frame the document review sample was comprised of much fewer projects that fell into just one super-category. This may indicate that the projects in the sample frame were somewhat more “trade intensive” than the projects in the broader sample frame. USAID 2017 TCB Evaluation | 38 Geographic Balances The geographic balances across regions in the sample are comparable to those in the sample frame. Table 6: Distribution of Activities Across Geographic Regions Region Sample Frame (n=466) Document Review Sample (n=75) # Activities Percent # Activities Percent % Difference from Frame Central America and Caribbean 81 17% 13 17% 0% Central and Eastern Europe 66 14% 11 15% -1% East Africa 61 13% 10 13% 0% Middle East and North Africa 43 9% 6 8% 1% Southeast Asia 36 8% 6 8% 0% West Africa 37 8% 6 8% 0% South America 29 6% 5 7% 0% Central Asia 22 5% 4 5% -1% Sub-Saharan Africa* 24 5% 2 3% 2% Pan Asia** 20 4% 3 4% 0% Southern Africa 20 4% 3 4% 0% South Asia 15 3% 4 5% -2% Global*** 12 3% 2 3% 0% *Activities categorized in the “Sub-Saharan Africa” region operated in multiple countries spread across multiple African regions (e.g. an activity with activities in a Southern African country and a West African country) ** Activities categorized in the “Pan Asia” region operated in multiple countries spread across multiple Asian regions (e.g. an activity with activities in a Central Asian country and a Southeast Asian country) *** Activities categorized as “Global” operated in multiple countries spread across multiple continents (e.g. an activity with activities in a Central American country and an African country) USAID 2017 TCB Evaluation | 39 Please name Mission, Bureau/Office 1. Please select for whom you work: USAID Mission USAID/W 2. Please name the largest, most current TCB project on which you work: 3. For this project: What is your current role? (select all that apply) USAID Trade Specialist/advisor A/COR Monitoring, Evaluation and/or Learning Specialist Agriculture Specialist/advisor Economist Private sector advisor Other (please specify) 4. To what extent has this TCB project or activity contributed in a measurable way to improved trade capacity in this country or region? (i.e. with evidence of trade-related outcomes via activity- or project-level qualitative or quantitative indicators, feedback from beneficiaries, evaluations, national level statistics tracked by but not attributable to the project) 0 Not at all Sufficiently 5 To a great extent Annex B. Stakeholder E-Survey USAID 2017 TCB Evaluation | 40 Standard 1 Standard 2 Standard 3 Standard 4 Standard 5 5. Please list the most relevant and useful standard indicators in use in the project? Why are they useful/what do they help you to do? To which intended result is the indicator attached? Custom 1 Custom 2 Custom 3 Custom 4 Custom 5 6. Please list the most relevant and useful custom indicators in use in the project? Why are they useful/what do they help you to do? To which intended result is the indicator attached? 7. To what extent has the intervention succeeded in accomplishing TCB’s objectives of improved trade and investment performance in the country or region? 0 Not at all Sufficiently 5 To a great extent 8. On the project, what have been the primary factors for success: 9. On the project, what have been the primary impediments: 10. Do you currently work on or have you worked on any other TCB activity since 2009? Yes No USAID 2017 TCB Evaluation | 41 11. If yes, please name: country, years, and activity title 12. For this project: What is your current role? USAID Trade Specialist/advisor A/COR Monitoring, Evaluation and/or Learning Specialist Agriculture Specialist/advisor Economist Private sector advisor Other (please specify) 13. To what extent has this TCB program contributed in a measurable way to improved trade capacity in this country or region? (i.e. with evidence of trade-related outcomes via activity- or project-level qualitative or quantitative indicators, feedback from beneficiaries, evaluations, national level statistics tracked by but not attributable to the project) 0 Not at all Sufficiently 5 To a great extent Standard 1 Standard 2 Standard 3 Standard 4 Standard 5 14. Please list the most relevant and useful standard indicators in use in the project? Why are they useful/what do they help you to do? To which intended result is the indicator attached? USAID 2017 TCB Evaluation | 42 Custom 1 Custom 2 Custom 3 Custom 4 Custom 5 15. Please list the most relevant and useful custom indicators in use in the project? Why are they useful/what do they help you to do? To which intended result is the indicator attached? 16. To what extent has the intervention succeeded in accomplishing TCB’s objectives of improved trade and investment performance in the country or region? 0 Not at all Sufficiently 5 To a great extent 17. On the project, what have been the primary factors for success? 18. On the project, what have been the primary impediments? 19. Do you currently work on or have you worked on any other TCB activity since 2009? Yes No USAID 2017 TCB Evaluation | 43 20. If yes, please name: country, years, and activity title 21. For this project: What is your current role? USAID Trade Specialist/advisor A/COR Monitoring, Evaluation and/or Learning Specialist Agriculture Specialist/advisor Economist Private sector advisor Other (please specify) 22. To what extent has this TCB program contributed in a measurable way to improved trade capacity in this country or region? (i.e. with evidence of trade-related outcomes via activity- or project-level qualitative or quantitative indicators, feedback from beneficiaries, evaluations, national level statistics tracked by but not attributable to the project) 0 Not at all Sufficiently 5 To a great extent Standard 1 Standard 2 Standard 3 Standard 4 Standard 5 23. Please list the most relevant and useful standard indicators in use in the project? Why are they useful/what do they help you to do? To which intended result is the indicator attached? USAID 2017 TCB Evaluation | 44 Custom 1 Custom 2 Custom 3 Custom 4 Custom 5 24. Please list the most relevant and useful custom indicators in use in the project? Why are they useful/what do they help you to do? To which intended result is the indicator attached? 25. To what extent has the intervention succeeded in accomplishing TCB’s objectives of improved trade and investment performance in the country or region? 0 Not at all Sufficiently 5 To a great extent 26. On the project, what have been the primary factors for success? 27. On the project, what have been the primary impediments? USAID 2017 TCB Evaluation | 45 28. How useful are the M&E systems (databases, dashboards, approaches and processes to collect and utilize data) currently in place in the mission? (Specifically around trade, but given that trade is typically implemented alongside something else, think about activities with trade components) For example, do the systems systematically track progress? 0 Not useful at all Useful 5 Extremely useful 29. Please explain your score in the previous question. Why are they useful? What does this data help you to do? How? 30. How do you define measureable? Response Successful? On Track? Meeting Objectives? 31. With the current approach or M&E system are you able to determine whether an activity is: 32. How does the M&E specialist in the mission work with project implementers on the M&E system? Who selects the indicators? 33. How relevant are the trade indicators on which you report to the trade-related objectives, results, or outcomes you expect the project or activities to achieve? USAID 2017 TCB Evaluation | 46 Response Make management decisions? Affect project design? Form Mission strategy? Reporting and accountability? Conduct portfolio reviews? 34. Are M&E data (performance monitoring and evaluation data) from the activities used to/for: Not at all helpful Slightly helpful Helpful Very helpful Extremely helpful Not aware of this resource Have not used this resource 2003 USAID Trade Capacity Building Strategy USAID Trade Results Framework USAID ProjectStarter online Trade MEL toolkit Trade Indicator Handbook TCB Meta Evaluation, From Aid to Trade: Delivering Results 35. How helpful are the USAID trade-related guidance documents: 36. What actions could USAID take to improve the relevance of custom or standard indicators for trade￾related activities? 37. How can USAID improve monitoring and evaluation data and reports for learning, decision making, project design, portfolio reviews? USAID 2017 TCB Evaluation | 47 38. How can USAID incentivize integrating standard approaches to trade-related performance reporting to improve trade-specific outcome tracking? 39. Can we follow up with you for an interview? Yes No 40. We kindly request that you send the most recent non-sensitive performance monitoring indicator data and any completed evaluations to smonschein@dexisonline.com. We will be using this data to review potential for strengthening trade-related indicators, reducing duplicity of indicators, and improving “aggregatability” across similar trade-related outcomes and outputs. USAID 2017 TCB Evaluation | 48 [Interviewer: Start by providing a brief description of the purpose/intention of the TCB evaluation] RESPONDENT ROLES/EXPERIENCE & [MISSION/USAID BUREAU NAME] TCB WORK 1. What is your current role in the [country Mission] or [USAID bureau name]? a) How long have you been [working/posted] there? b) Where did you [work/serve] before? 2. Are you an A/COR for any trade-specific or trade-related activities? If so, which ones? 3. Please describe the ways that [USAID bureau] is supporting/funding TCB in [country or region].  Probes: specific activities/IPs who are focused primarily on TCB; activities that are not trade-specific, but have trade-related components; the types of TCB support undertaken (i.e. customs, trade-related ag, standards work, technical barriers to trade, etc.) 4. How has the TCB portfolio evolved over the years?  Probes: influence of Power Africa, Feed the Future, recent trade agreements, TFA, etc. OUTCOME OF TCB WORK IN [COUNTRY OR REGION] 5. Thinking about the things that this [mission/USAID bureau name] is doing to build trade capacity in [country or region], how do you know when the work has been “successful”?  Probes: measures/indicators/signals used 6. Is it always possible to measure a TCB “success”? a) If not, how do you capture/ascertain that there has been a success when it’s not as easy or possible to measure it? 8. What activities/efforts in [country] have not been as successful as anticipated? a) How do you know that the activity has not been as successful? Location of Interview (City, Country): Date of Interview: Name, Title, Organization of Interviewer: Name, Title, and Organization of Interviewee(s): Annex C. Interview Questionnaire Interviews with USAID/W and Mission Staff (Trade Advisors/Officers, AORs/CORs, M&E Officers, other related staff) USAID 2017 TCB Evaluation | 49 7. What trade-related activities/efforts in [country] do you think been particularly successful? a) How do you know that the activity has been successful? b) What factors contributed to the activity’s success? b) What factors contributed to the activity’s struggles? 9. Please describe the ways in which [USAID bureau name] collaborates with bilateral and regional missions and/or other USAID/W bureaus or offices on project and activity design and implementation. a) What are the primary challenges? b) Are there any supportive synergies? USING M&E SYSTEMS TO TRACK/MEASURE TCB ACTIVITIES’ PROGRESS/SUCCESS 10. What kinds of M&E approaches/systems does this [mission/USAID bureau name] use to track and measure progress toward trade capacity building [in country or region]? 11. In general, what are the standard (F) indicators that the TCB activities [in country or region] report on? a) Have IPs experienced any difficulties reporting on these indicators? If so, what kinds of difficulties have they had? b) What do the results from F indicators tell you about an activity’s trade-related outcomes? 12. In general, what kinds of custom indicators do TCB activities in [country or region] report on? a) Do all TCB activities in [country or region] report on custom indicators? Why/why not? b) How do you use custom indicators to characterize/report on TCB work in [country or region]? 13. Do the indicators provide you with enough information? If not, what other information do you wish you had? 14. How else do you learn about what works/doesn’t work when it comes to TCB in [country or region]?  Probes: evaluations, meetings, desk research, etc. 15. Based on your experience to date, how would you characterize the usefulness of M&E tools USAID/this mission is currently using to understand and improve TCB activities in this [country]? a) What do you think USAID can to do to better integrate M&E reporting improvements in the design and implementation of TCB-specific or trade-related programs? Supporting Trade Capacity in [Country or Region] in the Future 16. What do you see as the most pressing priorities/needs for TCB in [country or region]? a) How is USAID positioned to address these priorities? b) What is needed specifically to address the priorities? USAID 2017 TCB Evaluation | 50 [VERSION: 11.18.16] TCB EVALUATION DOCUMENT REVIEW AND ASSESSMENT STRATEGY OVERARCHING PARAMETERS/ASSUMPTIONS Team members involved in document review Greta, Lauren, Jillian, Rachel and Stephanie Activities to be reviewed during document review 75 Total number of documents to review ~225-375 documents total (presuming 3-5 documents per activity) LOE needed 4 hours per activity to skim content and populate Excel table with data; with 75 activities this is about 300 hours LOE. • With a team of 5 each reviewer would review 15 activities which would take about 7-8 days to complete the review. Purpose of document review The primary aim of the review is to cull relevant data that is responsive to the evaluation questions following a structured procedure to ensure team members pull the right data. Document Review Database The “results” of each activity’s document review will be entered/captured in a structured database that can be subsequently used facilitate synthesis of a cross-activity outcomes using both quantitative and qualitative analysis. An initial prototype of this database will be circulated for comment when this strategy is circulated to team members for comment. Annex D. Document Review Guide USAID 2017 TCB Evaluation | 51 DESCRIPIVE PROJECT INFORMATION For each project reviewed, populate the relevant columns in the tracker to reflect descriptive information on each project you review. The columns you need to populate are described in the tables below. Follow the coding instructions for each element as written. Unless there is a specified “numeric code” to select from and apply to your activity, presume the entry is a “string” entry (i.e. text). Turquoise-shaded tables indicate fields that are auto-calculated; you do not need to do anything with these columns. Column D Prime IP Definition: The name of the prime IP executing the project Column Header: IP Notes: The prime IP is usually the entity that submits reports; do not include names of sub-contractors or non-prime partners. Column E Country Definition: The names of the country or countries where the project took place Column Header: Country Notes: This column is pre-populated. Verify with documentation and correct if inaccurate. Use full country name not an abbreviation, separate multiple countries by a comma. Column F Region Definition: The names of region where the project took place Column Header: Region Notes: This column is pre-populated. Verify with documentation and correct if inaccurate. Use full region name not an abbreviation. Column G Start Date (MM/YYYY) Definition: The month and year a project began. Scale: Month and Year Column Header: Date_Start Column H End Date (MM/YYYY) Definition: The month and year a project ended Scale: Month and Year Column Header Date_End Column I *Duration (months) Definition: The number of months a project was active Source: calculated Column Header Duration [Columns J-S provide pre-populated columns detailing the status of documentation and TCB-specific funding for the project; reviewers should not need to update these fields]. USAID 2017 TCB Evaluation | 52 Column T Total Project Funding (Life of Project) Definition: Total funding for a project across the life of that project. Scale: U.S. dollars Column Header: Funding_Total Notes: Total funding for all components (TCB and non-TCB) and all years of a project, including prior to 2009 and after 2015. This may involve adding the initial value of a contract to any amendments or future supplemental funds. Enter only the funding total. Column U *Proportion of TCB Funding Definition: Proportion of project funds that are identified as “TCB” Source: calculated Column Header Proportion_TCB Columns V-Z Sector 1, 2, 3, 4, 5 Definition: The sector(s) targeted by the project Scale: 1 – Agriculture (commodities, livestock, horticulture, wood, 2 – Manufacturing (goods, chemicals, machinery) 3 - Energy and uncultivated natural resources (electricity, mining, water) 4 – Services (not including tourism) 5 – Tourism 6 – No specific sector (i.e. economy-wide policy development or reform or trade-related infrastructure) Column Header: Sector1, Sector2, Sector3, Sector4, Sector5 Notes: Populate one code per column in order of strength of emphasis (i.e. Sector1 is primary emphasis, Sector5 has least emphasis). If you can’t tell which sector the project works in, leave this cell blank. Column AA Number of Sectors Definition: The number of sectors targeted by the project Source: Calculated Column Header: #_Sectors Columns AD-AF Institutional Beneficiary 1, 2, 3, 4, 5 Definition: The direct target/beneficiary of the activity. The entity that is expected to change as a result of the intervention. Scale: 1 – Private Sector goods producers/exporters/importers (including Co-ops) 2 – Private Sector services (including tourism) exporters (including Co-ops) 3 – Private Sector support firms (firms that provide training & assistance for other firms to assist with exporting/importing, including standards, marketing, export promotion, finance, policy advocacy. This includes consultants, freight forwarders, trade financiers, etc.) 4 – Non-profit entities doing activities similar to #3 above 5 – Government entity doing activities similar to #3 above (Export Promotion agency, for example) 6 – Government agencies/entities/labs, except those included in #5 above USAID 2017 TCB Evaluation | 53 7 – Other non-profits (environmental protection, anti-corruption, etc.) 8 – Consumers (via lower prices, etc.) 9 – women 10 – the poor 11 – other marginalized group 12 – Public-Private partnership doing similar to #3 above Column Header: Beneficiary1, Beneficiary2, Beneficiary3, Beneficiary4, Beneficiary5 Notes: Populate one code per column in order of strength of emphasis (i.e. Beneficiary1 is primary emphasis, Beneficiary5 has least emphasis). If you can’t tell which beneficiaries the project targets, leave this cell blank. Columns AG Number of Beneficiary Targets Definition: Number of different beneficiary groups targeted by the project Source: Calculated Column Header: #_Beneficiaries Columns AH-AL Other Actors 1, 2, 3, 4, 5 Definition: Documents mention other actors working toward the same or complimentary objective in the same population or geographical location. Scale: 1 – Host government program 2 – Host country foundation/NGO 3 – Other bilateral donor (DFID, etc.) 4 – 3rd country foundation/NGO (Oxfam, etc.) 5 – The World Bank 6 – Other multilateral development bank (ADB, etc.) 7 – U.N. Agency (UNDP, etc.) 8 – Regional organization (ECOWAS, etc.) 9 – U.S. foundation/NGO (Gates Foundation, IESC, etc.) 10 – Other USAID program 11 – Other U.S. government program 12 – Corporation 13 – None 14 – Mentioned, but not specified Column Header: OtherActors1, OtherActors2, OtherActors3, OtherActors4, OtherActors5 Notes: Code according to the funder of said project (if you can tell). For example, if Aid to Artisans was implementing a DOL program, it would be coded 11. If it implemented its own program, code it 9. Columns AM Number of Other Actors Definition: Number of different actors working in a similar space to the project examined Source: Calculated Column Header: #_OtherActors USAID 2017 TCB Evaluation | 54 Columns AN TCB Focus Definition: Captures the extent of the focus devoted to trade capacity building in a project. Scale: 1 – The project contained a small trade component but was primarily focused on other issues 2 – The project balanced trade and non-trade activities relatively equally 3 – The project was mostly focused on trade, with a few non-trade elements 4 – The project was exclusively focused on trade 0– The project was not focused on trade AT ALL. Column Header: TCB_Focus [Columns AM through BJ provide pre-populated information on the TCB categories attached to the project in the USAID TCB Databases, as well as our defined “TCB Supercategories”. These should not need to be updated.] Column BK-BP Feed the Future and Trade-Related Agriculture Details Definition: Captures more detail on projects which are explicitly Feed the Future and/or have been labeled in the TCB Database to be “Trade-Related Agriculture.” The indicators in these columns are 1s and 0s. The database will include fields for the reviewers to indicate whether activity addresses cross-border trade in some other specific way or if the reviewer determines that the activity does not have a specific cross-border trading component. Scale: BJ – 1 if the project is FTF or TR-Ag BK – 1 if the project focuses on facilitating cross-border ag-trade (not internal ag trade) BL – 1 if project focuses on increasing access to imported inputs needed for agriculture (seeds, fertilizer, machinery, genetic material) BM – 1 if project provides capacity building/institutional strengthening support for ag-trading entities (i.e. farmers groups and associations) BN – 1 if project has some other ag-trade component that doesn’t fit in the other categories (i.e. financing for cross border ag trade) BO – 1 if the project does not appear to have any specific components related to cross border agricultural trade. Column Header: Feed the Future USAID 2017 TCB Evaluation | 55 EVALUATION QUESTION 1: TO WHAT EXTENT HAVE USAID TCB PROGRAMS CONTRIBUTED IN A MEASUREABLE WAY TO IMPROVED TRADE CAPACITY IN TARGETED COUNTRIES? Overall Approach • We have found that there do not seem to be consistent indicators reported across activities (even F indicators). • Using the table of indicators tracked in the first evaluation (see Table 26 in the evaluation report (page 112-118)) is not a tenable option; we found it extremely difficult/time consuming to trace indicators in current projects to this list and found that the F indicators have changed significantly since the evaluation was finished. • In the interest of time we’re proposing an abridged way to assess progress on indicators that can at least give a sense about the extent to which activities were on track with their indicator targets. Documents to Consult • In most cases, the annual reports will be the primary source for indicator data. They typically include some sort of indicator table/section on M&E section. • Final reports *may* report indicators, but they are more likely to report cumulative results (which won’t necessarily provide information on whether targets were met). • Evaluation reports may draw on but will likely not report out on indicators specifically. Indicators (Columns BQ￾BU) • This phase of the document review will assess and score the proportion of indicator targets met/exceeded each year of the program. • This assessment will be cumulative indicator results for the project, and reviewers will assign a score based on the extent to which the project met or exceeded its life-of-project (LOP) indicator targets (BQ). • Reviewers will summarize the kinds of the standard indicators tracked/reported in a descriptive narrative (BR); indicate the number of standard indicators in column BS) • Reviewers will summarize the kinds of the custom indicators tracked/reported in a descriptive narrative (BT); indicate the number of standard indicators in column BU) Indicator Scores • The reviewer will assign a single score of 1, 2, or 3 or 99 for all custom and standard indicators reported. o 3 – Indicates that the IP has met/exceeded 75%-100% of its “life of project” (LOP) indicator targets reported o 2 – Indicates that the IP has met/exceeded 50-74% of its “life of project” (LOP) indicator targets reported o 1 – Indicates that the IP met/exceeded targets less than 50% of its “life of project” (LOP) indicator targets reported o 99 – Indicates that the IP did not report on indicators with targets for the entirety of the project. Considerations • Note: If you cannot tell if the IP reported standard indicators, just provide a description and count of indicators in the custom indicator columns (BT/BU); leave the standard indicator columns blank. USAID 2017 TCB Evaluation | 56 EVALUATION QUESTION 2: TO WHAT EXTENT HAVE THE INTERVENTIONS FUNDED BY USAID SINCE 2010 SUCCEEDED IN ACCOMPLISHING THE PROGRAMS OBJECTIVES. (PART 1) Overall Approach • To address this evaluation question with document review we will do four things: 1. Summarize the activity objectives. 2. Using the activity objectives map the project to specific components of the (updated) TCB results framework developed in the first TCB evaluation. 3. For each TCB results framework component that can be mapped to an activity, team members will review relevant documents and use their judgement to assign a score indicating whether the activity met its objectives vis a vis each pursued results framework component. 4. Where available, team members will also review evaluation reports to assess/summarize the nature of the findings. Documents to Consult • An activity’s final report will likely be the best source of information on project objectives and high level outcomes; where final reports are not available team members will have to consult annual reports to piece together objectives. • Where available activity evaluation reports will also provide information on activity objectives and an assessment of outcomes for at least some of the trade-related objectives. Assessing/Defining Objectives (Column V) • Team members will begin this phase of document review by skimming through an activity’s final report/evaluation reports to get a general sense of the activity’s objectives. o Reviewers will summarize objectives in narrative form (~3-5 sentences) in the Document Review Database. USAID 2017 TCB Evaluation | 57 EVALUATION QUESTION 2: TO WHAT EXTENT HAVE THE INTERVENTIONS FUNDED BY USAID SINCE 2010 SUCCEEDED IN ACCOMPLISHING THE PROGRAMS OBJECTIVES (PART 2) Mapping the Activity to TCB Results Framework Elements • The next step in the analysis is to map the activity to specific subcomponents of the TCB results framework. • Using the understanding of the activity objectives developed in the previous step, reviewers will determine the Results Framework elements that are most relevant to the activity. • Note: Many but not all project or activity objectives will map most directly to the lowest level of the results framework and indirectly to upper levels. There will be cases, however, where the objective does not map to the lowest level but directly maps to the next level up. If this is the case, do not worry about having to “shoehorn” the objective into the lower level. • Note: Because different activities will map to different aspects of the RF, we do not expect that all of the Results Framework columns will be populated for most (if not all activities). Assessing Progress on Results Framework Elements • For each Results Framework component that is relevant to the activity being reviewed, team members will consult activity evaluation reports and evaluation reports and use their judgment to determine whether the activity components relevant to the results framework element met expectations or not. Based on this assessment team members will provide a score of 1, 2, 3 or 99. Results Framework Element Scores (Columns BW-DB) • For each element mapped, the reviewer will assign a single score of 1, 2, or 3 or 99 for all custom indicators reported. o 3 – Indicates that all of the activity components supporting this element met or exceeded expectations (i.e. was reported by IP to be fully successful). o 2 – Indicates that all of the activity components supporting this element partially met expectations (i.e. was reported by the IP to be mostly or partially successful). o 1 – Indicates that all of the activity components supporting this element did not meet expectations (i.e. was reported by IP to be a challenged activity that struggled). o 99 – Indicates it was not possible to assess progress on this element for this activity. • For each results framework component scored, the reviewer should provide a brief narrative justification for why they assigned the score they chose. USAID 2017 TCB Evaluation | 58 Evaluation Review (Columns DC-DD) • A second, high-level approach to gauging overall “success” of TCB activities will be achieved through a general assessment of evaluation results (where available). For each activity that has been evaluated, the reviewer will score evaluation results on a 1, 2, 3, or 99 and provide a brief narrative summary for the results of the evaluation. Evaluation Review Scores • For each activity, the reviewer will assign a single score of 1, 2, or 3 or 99 for all of the evaluation reports reviewed. o 3 – Indicates that the evaluation(s) determined the program to be largely successful (i.e. more positive results than negative). o 2 – Indicates the evaluation(s) generated mixed results (generally a split between positive and negative results). o 1 – Indicates the evaluation(s) found that the activity was largely unsuccessful (i.e. more negative results than positive) o 99 – Indicates it was not possible to assess the success of the activity based on evaluation reports. EVALUATION QUESTION 3: WHICH ACTIVITIES HAVE BEEN MORE SUCCESSFUL IN ACHIEVEING THEIR OBJECTIVES, AND WHAT WERE THE PRIMARY FACTORS RESPONSIBLE FOR THEIR RELATIVE SUCCESS? Overall Approach • At this point we do not have a good way of identifying, a priori, the successful vs. less successful activities. Because of this, as well as the fact that the client may also want to know whether there were any factors contributing to TCB activity challenges, we will capture challenge and success factors on all activities we review. • For the document review phase, reviewers will summarize • These narratives will then be subjected to a qualitative thematic analysis which will aim to describe common factors across projects. Documents to Consult • Final reports, evaluation reports, and annual reports often have specific sections in which IPs detail particular challenges and successes. o Success/challenge factors are more likely to be systematically documented in evaluation and final reports. o Annual reports may contain reports of challenges to explain why indicator or implementation targets were missed. Success and Challenge Tabulations ( Columns DE- DL) • Reviewers will summarize the success and challenge factors reported by IPs in the Document Review Database. • See tables below for specific instructions for each type of data field. USAID 2017 TCB Evaluation | 59 Columns DE-DG Instability 1, 2, 3 Definition: Document mentions a significant external change that affected the project (eg natural disaster or severe drought, severe political upheaval, etc). Scale: 1 – Major or minor political upheaval that affected the project 2 – Natural disaster that affected the project 3 – Economic crisis that affected the project 4 – Other instability that affected the project 5 – No mention of a significant external change that affected the project Column Headers Instability1, Instability2, Instability3 Notes: This must be something that occurred after the project was planned or began implementation that forced the project to alter itself in some way. For example, a project being implemented in Indonesia during the 2004 tsunami might be coded “2” if the disaster affected the project. However, a project that began in Afghanistan in 2003 would not be coded “1” despite the ongoing conflict there. This is because the instability existed even before the project began. However, if an unstable situation took a dramatic turn for the worse and the increase in instability affected the project, it could be coded “1”. Be prepared to Google references to incidents that you don’t understand. Columns DH Design Modifications Definition: The documents state that the project was modified in a significant way during implementation (ie. changed direction, scaled up/down, etc.) Scale: 1 – Yes 0 – No Column Header: Design_Mods Columns DI Sustainability Planning Definition: Documents describe plans or lack of planning for sustainability Scale: 1 – Documents mention planning for sustainability 2 – Documents mention a lack of planning for sustainability 3 – Documents are inconsistent about sustainability planning 4 – Documents do not address sustainability Column Header: Sustain_Plans USAID 2017 TCB Evaluation | 60 Notes: Inconsistencies may exist between reports prepared by the prime implementing partner vs. those prepared by outside evaluators or auditors. Code inconsistent information “3”. If project is still ongoing, there may not be sufficient information to score the project; if this is the case leave this field blank. Columns DJ Internal_Synergies Definition: Projects with multiple components may experience “synergy” if the components work well together. They may also experience conflicts or difficulties stemming from the presence of multiple components. Scale: 1 – Documents mention positive synergies between various components of the project 0- Documents do not mention positive synergies between various components of the project Column Header: Internal_Synergies Notes: Inconsistencies may exist between reports prepared by the prime implementing partner vs. those prepared by outside evaluators or auditors. Code inconsistent information “3”. Columns DK External Synergies Definition: Interactions with other projects working toward the same or different results. Scale: 1 – Documents indicate some partnership or relationship between project and projects implemented by other actors but do not indicate a positive/negative result 2 – Documents mention positive synergies between the project and projects implemented by other actors 3 – Documents mention conflicts between the project and projects implemented by other actors 4 – Both positive and negative relationship issues are described. 5 – Documents do not mention any relationship between the project and projects implemented by other actors Column Header: External_Synergies Notes: Inconsistencies may exist between reports prepared by the prime implementing partner vs. those prepared by outside evaluators or auditors. Code inconsistent information “4”. USAID 2017 TCB Evaluation | 61 Columns DL External Relationships Definition: Interactions with external stakeholders. Scale: 1 – Documents describe positive external relationships 2 – Documents describe problematic external relationships 3 – Documents describe both positive and negative external relationships 4 – None described Column Header: External_Relationships Notes: Examples of problematic relationships might include resistance to trade liberalization from certain businesses, sectors, or political leaders; resistance to anti-corruption efforts by customs officials; or any difficult relationship between project personnel and a host-country stakeholder. Columns DM Keys to Success Narrative Definition: Did the coder include narrative regarding what the project documents perceived as particularly important factors in the success of the project? Column Header: None Notes: Keys to Success may include descriptions of the various factors believed to be primarily responsible for the success of a project. Examples of this might include a particularly unique or successful approach or model, the receptivity of beneficiaries to the particular interventions undertaken, a strong relationship with external stakeholders, etc. Avoid including language that emphasizes the quality or helpfulness of the USAID client or staff, U.S. embassy or other government personnel, etc. The credibility of such statements is hard to establish due to the likelihood that the government personnel being lauded are also the target audience for the report. You must differentiate between text taken from a report prepared by the prime implementing partner (like a final report) and text taken from an evaluation or other 3rd party document (IG report, GAO report, etc). If it’s a third party document, put “evaluation” or something similar in parentheses after the text entry. Columns DN Barriers to Success Narrative Definition: Did the coder include narrative regarding what the project documents perceived as particularly important factors in the success of the project? Column Header: None Notes: Barriers to Success may include descriptions of the various factors believed to be primarily responsible for the failure of a project. Examples of this might include a particularly unsuccessful approach or model, the lack of receptivity of USAID 2017 TCB Evaluation | 62 beneficiaries to the particular interventions undertaken, a weakened relationship with external stakeholders, etc. Avoid including language that emphasizes the quality of the USAID client or staff, U.S. embassy or other government personnel, etc. The credibility of such statements is hard to establish due to the likelihood that the government personnel being lauded are also the target audience for the report. You must differentiate between text taken from a report prepared by the prime implementing partner (like a final report) and text taken from an evaluation or other 3rd party document (IG report, GAO report, etc). If it’s a third party document, put “evaluation” or something similar in parentheses after the text entry. EVALUATION QUESTION 4: HOW CAN USAID INTEGRATE MONITORING AND EVALUATION INTO THE DESIGN AND IMPLEMENTATION OF TCB PROGRAMS MORE SYSTEMATICALLY? Overall Approach • This question will be primarily address through other means (i.e. Case Studies, KIIs with USAID/Washington). • However, the ease and completeness with which team members are able to complete the steps of the document review will be heavily influenced by the quality/completeness of data and reporting of the TCB activities (which is a reflection of the quality of M&E for TCB). • We will document our experiences with document review to inform our answer to EQ4. USAID 2017 TCB Evaluation | 63 MOST SUCCESSFUL TCB ACTIVITIES INCLUDED IN TCB EVALUATION SAMPLE a/ Region Country Start Date End Date Project lifetime funding US$ b/ Central America and Caribbean DR-CAFTA Support Dominican Republic Jan-07 Jan-12 $6,209,216 Improved Access to Employment El Salvador Dec-09 Nov-13 $8,397,040 Small and Medium Enterprise Development El Salvador Oct-11 Oct-16 $11,866,809 Support Implementation of CAFTA-DR Environmental Chapter Requirements C. America Region & Dom. Republic Sep-06 Sep-12 $28,793,800 Municipal Competitiveness El Salvador Sep-10 May-15 $5,046,790 Nicaraguan Enterprise and Employment Activity Nicaragua Oct-09 Aug-13 $7,361,031 Central and Eastern Europe Azerbaijan Competitiveness and Trade (ACT) Azerbaijan Oct-10 Sep-13 $14,765,616 Competitiveness Project in Macedonia (MCP) Macedonia Aug-07 Aug-12 $8,396,272 Enterprise Energy Efficiency (3E) Bosnia and Herzegovina May-10 May-14 $4,413,192 Central Asia Incentives Driving Economic Alternatives: North, East, and West (IDEA-NEW) Afghanistan Mar-09 Sep-15 $20,798,024 East Africa AGP AMDe Ethiopia Oct-11 Dec-15 $51,323,509 East Africa Trade Hub (EATH) East Africa Feb-09 Aug-14 $78,088,645 Innovative Agricultural Research Initiative (iAGRI) Tanzania Mar-11 Mar-17 $4,000,000 Middle East and North Africa Cultural Resource Management in Luxor Egypt Oct-11 Dec-14 $4,422,262 Enterprise Development and Investment Project (EDIP) West Bank/Gaza Sep-08 Sep-11 $15,199,073 Tunisia ICT Competitiveness Project Tunisia May-12 Sep-13 $7,500,000 South America Paraguay Productivo (Improved Economic Opportunities for Paraguayans, IEOP) Paraguay Sep-09 Sep-12 $5,955,110 Peru-Andean Trade Capacity Building Program Peru Jun-10 Oct-13 $21,415,399 MIDAS (Más Inversión para el Desarrollo Alternativo Sostenible) Colombia Jan-06 Sep-11 $120,584,989 South Asia Poverty Reduction by Increasing the Competitiveness of Enterprises (PRICE) Bangladesh Feb-08 Sep-13 $3,890,000 Southeast Asia Agribusiness Market and Support Activity (AMARTA) Indonesia Sep-06 Apr-11 $14,105,064 Governance for Inclusive Growth Vietnam Dec-13 Dec-18 $7,060,921 Trade-Related Assistance for Development (TRADE) Project Philippines Oct-12 Oct-17 $3,766,000 Sub-Sahara Africa AGCI-Knowledge Sharing and Analysis Project - KSA Jun-07 Jun-13 $11,275,485 West Africa Annex E. Most Successful TCB Activities included in TCB Evaluation Sample USAID 2017 TCB Evaluation | 64 African Cashew Alliance (Toward Inclusive Markets Everywhere (TIME) Western Africa Region Jan-14 Sep-16 $1,286,203 USAID West Africa Trade Hub Western Africa Region Jan-07 Jan-12 $10,028,000 a/ These activities achieved 75% or more of their LOP targets based on the most recent available activity reports. For those activities for whom an evaluation was available, the evaluation scored a 2-3 (on a 3 point scale). b/From TCB database; actual project funding could be more. USAID 2017 TCB Evaluation | 65 EVALUATION DATA Under the 2017 TCB Evaluation, 54 available evaluations were reviewed to gather lessons learned and recommendations for future programming. These included performance and impact evaluations (mid￾term and final), self-reflective documents, and OIG audits. While synthesized recommendations are included in the evaluation report, Annex F aims to be a centralized resource, organized by region, in which to find the most relevant TCB recommendations from across these reports. The information highlighted focuses on findings that may be relevant for future programming or beyond the current activity (if available), rather than summarizing activity specific results. It is meant to identify and highlight broadly applicable findings, recommendations, and lessons learned in a more detailed format than an evaluation report allows. As a result of this approach, various kinds and amounts of information were pulled for each evaluation; what is written below is meant to be thought provoking highlights but not holistic representations of evaluations or activities. All the information contained below is drawn from the evaluations reports listed before each entry, and the majority is direct quotes with any comments from the compilers in italics. Quotation marks in this document denote pieces of text that were quotes in the original evaluation. The titles of all the reports are included, and are publicly available on the DEC, should further information be desired.1 The regions2 are listed alphabetically and include: • Central America and the Caribbean…………………………Pages 2-10 • Central and Eastern Europe………………………………….Pages 10-20 • Central Asia…………………………………………………..Pages 20-24 • East Africa…………………………………………………….Pages 24-31 • Global…………………………………………………………Pages 31-32 • Middle East and North Africa………………………………...Pages 32-34 • Pan Asia……………………………………………….……....Pages 34-36 • South America………………………………………………...Pages 36-37 • South Asia…………………………………………………….Pages 37-42 • Southeast Asia………………………………………………...Pages 42-45 • Southern Africa……………………………………………….Pages 45-48 • Sub-Saharan Africa…………………………………………….Pages 48-49 • West Africa …………………………………………………...Pages 49-53 1 Since this document is mostly direct quotes it may reference acronyms or figures from the complete evaluation. When possible this has been avoided and it should not affect the overall message of the excerpt. Footnotes from the original evaluations were not carried over. Please refer to the complete evaluation for more information. 2 Activities were assigned to geographic regions in a multistep process. Where regions were clearly assigned in the TCB Investment Dashboard data those regions were used. When only countries of operation were provided, activities were grouped into TCB Investment Dashboard regions based on the regional mission for those countries. Finally, in some cases, activities were in multiple countries under different regional missions so the evaluation team created additional regional categories. For example, activities in multiple continents were deemed global, an activity in Ghana and Zambia would be classified as Sub-Saharan Africa, and an activity in Indonesia and China would be classified as Pan Asia. Annex F. Compiled Evaluation Findings and Recommendations USAID 2017 TCB Evaluation | 66 Central America and the Caribbean Caracol Industrial Park Power Plant Haiti Reconstruction: USAID Infrastructure Projects Have Had Mixed Results and Face Sustainability Challenges – U.S. Government Accountability Office Report to Congressional Requesters The OIG report covers all three parts of the Caracol Industrial Park. The activity in the TCB database referred only to the Caracol Industrial Park Power Plant. The sections of the OIG report below are highlighted as directly relevant to the power plant. USAID has allocated $170.3 million to construct a power plant and port to support the newly developed Caracol Industrial Park (CIP), with mixed results (Page 1). The USAID mission completed the first phase of the CIP power plant, with a designed capacity of 10 megawatts, for $17.0 million, 11 percent less than the $19.1 million allocated (see table 3). The power plant project benefited from the mission having a Senior Energy Advisor on staff from April 2011 through February 2013 who used his background in electrical engineering to oversee and manage the project (Page 14). The study concluded that the CIP will only succeed if expanded, efficient port facilities are developed nearby (Page 22). The port and power plant depend on revenues from the CIP: CIP tenants will generate a substantial portion of the revenue for the power plant and port, so the sustainability of these projects will depend on the Haitian government finding additional tenants and maintaining the park (Page 23-24). All three projects depend on Haitian government capacity: The Haitian government will be responsible for maintaining and managing the CIP and power plant and for overseeing the private company that will operate the new port. Studies of the CIP have cited concerns about the relevant Haitian government ministry’s ability to manage and maintain the infrastructure in and around the CIP given their limited staff and technical resources. Aware of such concerns, the CIP has contracted a professional industrial facility management firm to operate and maintain the park (Page 24). Obtaining revenue for the power plant from electricity distribution outside the CIP: As of February 2013, the few customers connected to the power plant outside the CIP had largely paid their initial bills on time. However, according to a 2010 report on the Haitian energy sector, 64 percent of Haitians do not pay their electricity bills in a timely manner and 33 percent do not pay at all. In addition, USAID officials have recognized that it is common throughout Haiti to tap into lines without paying, and this practice is unlikely to have repercussions. As a result, the USAID operations and maintenance contractor plans to provide training to local communities on the use and value of electricity (Page 24). DR-CAFTA USAID/Dominican Republic Tri-Project Performance Evaluation Dominican Republic – Central American Free Trade Agreement Implementation Project (DR-CAFTA IP) – Final Evaluation The objective to which DR-CAFTA IP is an ‘improved enabling environment’ that will contribute to achieving USAID/DR’s Economic Growth Strategic Objective (SO-1), “Increased sustainable USAID 2017 TCB Evaluation | 67 economic opportunities for the poor” and particularly SO IR 1.1, “Improved conditions for a more competitive Dominican Economy.” The Project also became the principle trade capacity building (TCB) vehicle for meeting a broader USG agreement to help the DR, its Central American partners and other developing country World Trade Organization (WTO) members to honor their commitments to freer trade (Page 7). The evaluation finds that at the end of three years of implementation, the Project is meeting or exceeding the activity or output targets set in its PMP. These output targets included the number of events, number and type (sex and industry sector) of event participants, trade-related studies, and number of operational and communications tools created (Page 1). Strengths • Project support for consultants, studies, and events has released DICOEX budget resources to focus on pressing needs to build and train staff, equip its facilities and develop outreach and communications mechanisms, and track Treaty compliance through the GODR legislative and regulatory processes. • Project-sponsored awareness and training events have reached a broad spectrum of audiences (Page 3). Weaknesses • The USAID decision to design two ‘Option Years’ into the DR-CAFTA IP project has proven to be disruptive to implementation momentum’. • USAID branding policy has constrained building DICOEX visibility and credibility. • USAID could better lead and administer coordination between DR-CAFTA IP activities and other USAID Economic Growth (EG) office projects. • USAID introduced (or approved) the use of activity and output level indicators as proxy measures of outcomes. • The Project has made only limited use of its event evaluation process for planning, follow-up and feedback for the GODR and the private sector. • The DR-CAFTA IP team’s central role in events management activities appears to have fostered DICOEX dependency on the Project as its administrative arm. • Project-funded support to DICOEX website development has yet to succeed in designing an Internet presence that is easy to navigate and transparent in its information content (Pages 3-4). Lessons Learned and Best Practices • International trade treaties can be useful tools for USAID’s development assistance kit. DR￾CAFTA is more than a trade agreement as are almost all regional and bilateral trade agreements under the WTO today. DR-CAFTA also calls on the Dominican Republic to adopt a number of domestic fiscal and economic reforms that internally make the country more efficient and competitive. Best practice. Design and implement economic growth development assistance programs to include compliance with appropriate trade treaty measures. • Avoid option years as they can disrupt project momentum. Best Practice. Contract projects for their entire planned implementation period. USAID 2017 TCB Evaluation | 68 • Exercise branding waivers when appropriate as they can lead to more cost effective use of USG funds for institution building. USAID has a mechanism for waiving or modifying its branding policy where justified. Best Practice. Train Mission, implementing partner and counterpart staff on how to know when waivers are appropriate and in how to justify and apply them (Page 6). Evaluation of The Regional and Bilateral Programs to Support Trade Compliance Under CAFTA-DR – Final Evaluation This is the assessment report on the assistance funded by USAID that helped the six CAFTA￾DR beneficiary countries in complying with provisions of Chapters 3, 4, 5, and 6 of the Agreement (Page i). The consultants recommend continuing working on activities to expand Private Sector Awareness of CAFTA regulations with tasks similar to the ones conducted by the [Bilateral Program Implemented by Booz Allen Hamilton] project. Customs personnel rotation affects sustainability and needs to be addressed in future initiatives. Multilateralism is a great opportunity for the country and the region and needs to be understood by government and private sectors. A next level of training programs should be introduced including ―train the trainers activities or certification courses with established institutions following a public-private partnership approach (Page 3). Various activities carried out by SIECA should be evaluated to be executed in the future in cooperation with the private sector in a public private partnership approach. A plan of assistance by other international organizations active in Central America, such as IDB and the European Union, should be coordinated. It is essential to continue with the training of private sector and government officials in rules of origin because there are many provisions that are still not well understood. GEDOEL needs short-term corrective actions; many potential users expressed interest in using this program. It is important to review some manuals developed and deepen their contents because some are too basic or do not help operators (Page 4). The team of evaluators found essential to continue with training in rules of origin. Workshops must respond to the needs of each country and not be designed in a general format that is applied across the board in all countries. It is necessary to support the process of updating the rules of origin to the current Harmonized System as well as the tools to facilitate trade processes such as the GEDOEL and the virtual rules of origin course (Page 4). Future project design, in general, needs more specific objectives and measurable indicators; next project must include reliable mechanisms to assure stakeholder participation and cooperation. Design of regional projects must include bilateral concerns for USAID Missions to develop a sense of ownership to facilitate their participation in implementation. Country specific strategies and agendas are required for greater impact. The private sector must be included as beneficiary an implementer to complement the work of the local government agencies through public￾private alliances. Inter-regional trade must be given high priority to increase CAFTA-DR utilization i.e. accumulation of CAFTA-DR quotas for export to the US The National Sanitary and Food Safety Systems in CAFTA-DR Countries need to be improved and addressed as integrated systems to facilitate compliance with the new FSMA of the US (Page 6). During the past five years of USAID assistance to implement the CAFTA-DR Agreement, many stakeholders have developed to become effective partners in development and should be USAID 2017 TCB Evaluation | 69 considered for future program activities. This local capacity is available for the next round of projects and should be utilized not only as support for future projects but also to promote them in PPP arrangements that can ultimately serve as the main force to sustain the projects in the future. These institutions and individuals serve as a repository of knowledge and as a reference for future development activities (Page 7). Feed the Future West: Winner – Mango Portion Feed the Future West / Winner: Increase in Mango Exports due To Project Interventions – Self Reflective Document/Survey FTF WEST/WINNER seeks to sustainably increase incomes for rural households in the Cul de Sac and the Matheux corridors (Haiti) by focusing investments on five targeted value chains: four domestic (corn, rice, beans and plantains) and one export (mango) (Page1). Based on this survey, mango producers rank the use plastic crates as the most significant practice to reduce post-harvest losses, followed by mobile collection centers (Page 6). The FtF West/WINNER project supported the development of the mango value chain through the introduction of improved post-harvest techniques to reduce losses; assistance to producers with mango tree planting, pruning and grafting; the implementation of a mango drying facility in Mirebalais; and the implementation of a pilot system for the traceability of mangos. These efforts have resulted in an increase in mango exports from farmer associations supported by the project and in significant reductions in post-harvest losses. However, there is still much work to be done to fulfill the potential of the mango value chain in Haiti (Page 8). Key tasks to undertake in the near term include: • Complete and expand the mango traceability system; • Continue to set up nurseries and to plant large areas of hillsides with mangos in targeted areas; • Expand the distribution of mobile collection centers, crates, and post-harvest materials to mango producer associations; Improve roads and transportation conditions from mango producing areas; Increase and diversify the number of processing facilities to capitalize on unsold mangos (dried mangos, juice, preserves, etc.); and • Modernize the value chain by training intermediaries to be more efficient (Page 8). Food Security and Nutrition (ACCESO) Program Audit of USAID/Honduras’s ACCESO Project - Office of Inspector General Audit The goal of the [ACCESO] project is to move “rural Honduran households out of extreme poverty and undernutrition by improving their incomes” (Page 1). The audit found that ACCESO had not made significant progress and is unlikely to meet the goal by the end of the project. As of September 30, 2013, it had moved only 1,630 households of the targeted 18,000 out of extreme poverty. This lack of progress resulted primarily from not collecting baseline data on time, a severe decline in coffee production and prices, and implementation delays (Page 1). USAID 2017 TCB Evaluation | 70 Recommendation 1. We recommend that USAID/Honduras implement a plan to complete the baseline survey and collect relevant data before follow-on Feed the Future projects start or require the implementer to collect data during the first 6 months of the project to establish realistic targets and document their decision (Page 6). Recommendation 4. We recommend that USAID/Honduras assess and document whether to implement additional indicators and targets to measure the project’s impact on increasing beneficiaries’ incomes and assets as part of the follow-on awards for Feed the Future and implement if appropriate (Page 7). Recommendation 5. We recommend that USAID/Honduras incorporate into the follow-on awards a procedure to assess the project’s effectiveness in building the capacity of local nongovernment organizations (Page 7). Recommendation 6. We recommend that USAID/Honduras evaluate and document whether to include the leveraging of private-sector funds as a requirement, and develop an indicator and target to measure the project’s success in achieving this requirement as part of its follow-on awards (Page 7). Recommendation 7. We recommend that USAID/Honduras incorporate into its follow-on awards for Feed the Future projects the requirement that implementers submit a security plan and budget for security-related personnel and equipment to USAID for review and approval (Page 8). Improved Access to Employment Performance Evaluation of the “Improving Access to Employment Program in El Salvador” – Mid-Term Evaluation US Government assistance to El Salvador recognizes this GOES-driven process and is supporting measures to overcome the pervasive barriers to increased investment and broad￾based economic growth (Page v). …in December 2009 USAID awarded a contract to Carana Corporation to undertake the Improving Access to Employment Program in El Salvador, Contract EEM-I-00-07-00006-00, for a total funding of $7.4 million. The Program, which is scheduled to end in November 2013, focuses on improving the functioning of the Salvadoran labor market to make it better capable of matching the supply of skilled workers with private sector demand for labor (Page 2). Key success factors in the engagement of public sector actors in job centers have been: a. Inclusive coverage that ensures [Ministry of Labor] buy-in; b. Appropriate training content and delivery; c. Successful introduction of a better approach to matching supply and demand (Page vii). Some of the most important elements for successfully engaging private sector partners are: • A major emphasis made on high-value certification; • The ability to incorporate a sound pre-selection model; • Appropriate and effective teacher training options; • A well-grounded approach to addressing specialized labor market niches; USAID 2017 TCB Evaluation | 71 • The ability to synchronize training with employers' hiring calendars; • A sound and replicable model for at-risk youth employment (Page vii). Municipal Competitiveness Final Performance Evaluation of the USAID Municipal Competitiveness Project In El Salvador – Final Evaluation The USAID Municipal Competitiveness Project was designed to improve the competitiveness of Salvadoran municipalities through capacity building and increased dialogue with the private sector (Page 2). The overarching recommendation is that MCP or a program similar to it should be continued with minor changes (Page vii). Continued technical assistance (TA) should remain the highest priority. For activities in 2015 or beyond, we recommend that new municipalities should also specify the amount of municipal resources committed to activities and specify the nature of and involvement of the mayor and other key officials, in activities, and include provisions for compliance monitoring of the competitiveness plans (Page vii). For both current and future MCCs, a project should: 1) train MCC members in basic skills to increase the effectiveness and sustainability of MCCs; and 2) support certificate and degree programs that promote local economic development (LED) (Page viii). Beginning in 2015, municipalities should implement a communications strategy on the results of initiatives related to municipal strengthening. Plans should incorporate improved citizen access to information on improvements on municipal services and MCC results (Page viii). Based on success of USAID-assisted grants and to promote sustainability, legislation should be promoted that would allow national organizations, such as the National Institute of Municipal Development (ISDEM) to promote and fund this process (Page viii). For 2014, the Project or any follow-on should continue providing specific technical assistance to women and women’s groups and: 1) build on the initiatives of implementing partners by providing follow-up to the cohort of women that have attended workshops; 2) encourage MCCs to devote particular attention to the role of women in local development in municipal competitiveness plans; and 3) train and provide follow-up support in establishing productive businesses to a new cohort of women entrepreneurs (Page viii). Future activities should address the documentation, systematization, and dissemination of information on Project experiences through: 1) developing a learning network to share case studies and other useful knowledge; and 2) activities that enable effective knowledge collection such as workshops, cross-sectoral committees, and applied research models to facilitate mechanisms for sharing. The reports and other information should be produced and make publicly available in English and Spanish for widest impact (Page viii). Helping municipalities learn how to raise municipal revenues is recognized as a much-needed focus of future USAID activities. At the national level, there are several issues related to decentralization financing of sub-national governments, all of which go beyond the current scope of MCP. National level recommendations are to provide technical assistance to the National Assembly to update and modernize the legal framework regulating municipal taxation and to USAID 2017 TCB Evaluation | 72 remove unnecessary barriers to diversifying funding sources. At the local level, activities should be focused on expanding fiscal training and linking it and follow-up technical assistance to an explicit commitment by municipal governments to implementing improvements. In addition, future activities should support other initiatives and competitiveness plans that help municipalities raise revenues (Page viii-ix). Measuring investment at the municipal level. Future activities should include investment measurement for providing baseline measures. As part of the recommendations to measure investment at the municipal level, the evaluators suggest three potentially complementary methods for the collection of baseline data on economic indicators. Over a longer range period, beginning in 2015, USAID, ideally in concert with other funding agencies, should consider: 1) support for a DIGESTYC Economic Survey under Central Reserve Bank (BCR) guidance; 2) including a business climate survey module in the MCI; and 3) developing capacity within municipalities to measure investment (Page ix). Audit of USAID/El Salvador’s Municipal Competitiveness Project - Office of Inspector General Audit The audit found that the project was implementing activities in 50 municipalities throughout El Salvador to help the municipalities be more competitive. During site visits to 10 of them, we confirmed that the project had improved cooperation between the public and private sectors and eliminated some of the mistrust that traditionally has affected their relationship (Page 1). However, despite the implementation of these activities, the audit noted several problems. • Performance indicators did not measure the project’s promotion of investment and trade. The audit could not determine whether the project was promoting investment and trade at the municipal level successfully because USAID/El Salvador did not establish a specific indicator to measure progress toward this important objective. • Performance data did not measure economic growth and employment accurately. The mission reported performance data that did not identify the businesses and jobs created as a direct consequence of the project, and therefore it did not measure the project’s impact on economic growth and employment accurately. • Project monitoring and oversight was weak. The audit identified ways to improve monitoring and oversight related to site visits, participant training, gender analysis, and environmental compliance (Page 2). In consideration of our audit findings, we recommend that USAID/El Salvador: 1. Document steps taken to develop a performance indicator that measures whether the project is making progress in promoting investment and trade. 2. Establish a process to differentiate new firms and jobs from those that are newly registered but have been operating informally, and collect additional data that more accurately measure economic growth and employment directly resulting from the project. 3. Prepare a schedule for conducting regular site visits to the participating municipalities over the remaining contract period, and document the visits in accordance with Mission Order 750. USAID 2017 TCB Evaluation | 73 4. Obtain access to the Training Results and Information Network, and properly monitor and report on in-country training programs and participants in accordance with Automated Directives System 253 and Mission Order 745. 5. Obtain the baseline assessment of business ownership and the gender integration plan from Research Triangle Institute, and implement the actions presented in the gender analysis. 6. Amend the contract with Research Triangle Institute to include environmental compliance language that will make the environmental mitigation plan mandatory if activating conditions are met (Page 2). Nicaraguan Enterprise and Employment Activity Mid-Term Evaluation of the USAID/Nicaragua Employment and Enterprise (E&E) Activity – Mid-Term Evaluation E&E is an economic growth project in Nicaragua, operating under the Missions GBTI II IQC. The E&E project's general objective is poverty reduction achieved through equitable economic growth resulting from higher incomes and greater employment. Its specific objective is to enhance the competitiveness of SMEs, including agricultural SMEs and small-scale producers (Page 5). Recommendation for a USAID-funded Post-E&E Activity: 1. Expand the use of the E&E model as the basis for a food security initiative in the poorest, food insecure areas of the country. This would enable USAID to employ SME development through value chains as an effective tool to improve food security in these locations. The new initiative would support traditional value chains for food crops such as grains, and cash crops such as coffee and dairy, led by anchor firms, as well as analyze new value chains as appropriate. It will also include a mother-child nutrition program. Under this scenario, in the event that no anchor firm was available for the targeted value chains (such as for grain crops), then it would be necessary to assist farmer based organizations such as marketing associations to fill this marketing role. The new initiative would use “farming as a business” model for farmer training, and would focus on business management training for other SMEs. It should be possible for the present E&E project to quickly move toward a future food security model over its remaining life by increasing its focus on food insecure areas (Page iii). Recommendations for a Post- E&E Initiative Our recommendations for the next phase of USAID assistance – after E&E – is to use the E&E model as the basis for a food security initiative in the poorest, food insecure areas of the country. This would enable USAID to employ SME development through value chains as an effective tool to improve food security. The new initiative would support traditional value chains for food crops, such as grains, and cash crops such as coffee and dairy, led by anchor firms. In addition it would analyze the possible introduction of new crops such as cacao, vanilla, and black pepper, as well as small ruminants and poultry for food and cash. It would also promote nutritious food crops such as vegetable gardens. As is the case presently, the new project would help provide production technical assistance, market linkages, and access to credit to the SMEs operating within the value chains. It would have a policy component, but this component would be directly linked to the value chains. For example, if a local municipality imposes an exit tax on USAID 2017 TCB Evaluation | 74 agricultural products, then the project would support the producers and business associations to remove the tax. There would also be a mother-child nutrition program, similar to the USAID-supported Development Activity Program (DAP) that was carried out over several years by the Adventist Development and Relief Agency (ADRA) in Northern Nicaragua. Under this scenario, in the event that no anchor firm was available for the targeted value chains (such as for grain crops) then it would be necessary to assist farmer based organizations such as marketing associations to fill this marketing role. Another element of a new project would be to support and encourage the use of on-farm storage and preservation of food crops. During our travels, the evaluation team was informed that low-cost, secure metal storage chambers for on-farm storage of grain crops are commercially available at a cost of around $200 each. The new initiative would use “farming as a business” model for farmer training, and would focus on business management training for other SMEs. The use of lead farmers with demonstration plots is an effective farmer training tool, which could be used under this a new initiative. During our field visits, the assessment team saw several packing sheds, warehouses, and greenhouses that were constructed by earlier development projects, including the Millennium Challenge Account, that are now under utilized. These facilities could possibly be used in a new food security initiative. It should be possible for the present E&E project to quickly move toward a future food security model over its remaining life by increasing its focusing on food insecure areas (Pages 41- 42). Central and Eastern Europe Azerbaijan Competitiveness and Trade (ACT) Final Performance Evaluation of the Azerbaijan Competitiveness and Trade(ACT) Project – Final Evaluation ACT was an activity designed to continue the work of predecessor activities (the Trade and Investment Reform Support Project and the Private Sector Competitiveness Enhancement Project) and improve the business environment in Azerbaijan while eliminating barriers to trade and investment in the country. The main conclusion reached regarding overall awareness is that ACT was not active enough in establishing closer links with major stakeholders at the start of the project, or creating a sufficiently high enough profile aimed at informing the donor community in particular (Page viii). One of the most effective means to ensure close cooperation between technical assistance projects such as ACT and key stakeholders in a host country is an information gathering survey that needs to be designed, planned, organized and conducted (particularly aimed at donors) immediately following the start of a project (Page ix). There is a strong case for serious consideration to be given by USAID to the way in which budget allocations for short-term experts - whether local, international or third country nationals - are calculated. Furthermore, there must be some logic as to the purpose of their planned interventions in terms of how this relates to project indicators, outcomes, and impacts. The result of their input should be measurable and directly related to the specific needs of target audiences. In practical terms, experts should be knowledgeable and have the necessary motivational as well as technical skills to make things happen (Page viii). Conclusions USAID 2017 TCB Evaluation | 75 1. Available evidence suggests that the ACT Project has successfully contributed towards key staff of beneficiaries/implementing partners, playing a significant role in their respective positions with regard to the preparation of draft laws and amendments in collaboration with project experts. 2. While it is premature to assess the outcomes or impacts of ACT operations, the findings indicate that the Contractors’ effort aimed at delivering a robust approach to ensure sustainability has started yielding some intermediate results. 3. The large gap between country economic growth indicators and quality of institutional changes remains the very serious obstacle for USAID’s on-going and future efforts for increasing the sustainability of projects. 4. ACT Project played an active role in supporting initiatives in the field geared towards future sustainability. Various examples of processes launched, systems designed and installed, and management practices adapted, tend to give confidence that such tools and mechanisms may be applied in the future without external support of the USG. Furthermore, it is possible to conclude that the employment of a cost-sharing approach might have also contributed to sustainability. 5. Cost sharing – Banking Sector Reform The outcomes of the cost-sharing approach, combined with provision of tailored STTA, are more likely to lead to systemic changes in strategic and managerial behavior of clients (Page x-xi). Lessons Learned Given the timeframe, budget originally envisaged, as well as restrictive socio‐political environment in Azerbaijan, the targets set seem too ambitious. Therefore, USAID should seriously consider the efficacy and value of designing a project that can realistically be implemented relative to the anticipated results as defined by pre-determined indicators in RFTOPs and SOWs. Changing the SOW in the middle of the project, as defined by Amendment 3 of the SOW in May 2013 agreed between USAID/Azerbaijan and the Contractor, does not necessarily lend itself to a smooth transition towards implementing the envisaged changes in direction (Page xi). International donors such as USAID often draw the assumption that by simply implementing a long-term technical assistance program in a host country means that beneficiaries and key stakeholders have ‘bought into’ the rationale and purpose of the program. This in fact is often not the case as more than one donor can be advising or working with those same beneficiaries or implementing partners at the same time. Perhaps USAID might consider a more robust stance in SOWs regarding donor coordination to avoid duplication and to ensure that direct beneficiaries understand and appreciate the differences in assistance between USAID projects and others. This would best be achieved at the start of a project (Page xi). One of the issues arising from this evaluation was the fact that there is no guarantee that by simply advising the GOAJ on the preparation of legislation and regulations, this support will necessarily lead to their implementation. In this respect, it may be more beneficial for USAID to employ a broader interpretation of business enabling environment rather than limit itself to working towards improving the legislative and/or regulatory framework of a host country. This USAID 2017 TCB Evaluation | 76 includes also working towards supporting diversification of the actors in the society dealing with economic policy issues and who could take the quality professional discourse on the economic issues further on their own. Furthermore, USAID should consider what it is in position to accomplish in the short- and long-run rather than undertake the whole menu of the necessary economic reforms, which will not fly given established political structures in the country (Page xi). Economic Prosperity Initiative Mid-Term Performance Evaluation of The Georgia Economic Prosperity Initiative (EPI) – Mid-Term Evaluation EPI is an activity in Jordan, which main objective is to improve enterprise, industry, and country-level competitiveness in Georgia. Evaluation was very specifically focused on improvements for aspects of EPI. Recommendations • Analytical work could be done in a shorter period of time in order to start focusing on implementation in the earlier period of the project. • Emphasize time of delivery of promised products and services. • Avoid selecting project activities based on likelihood of achieving performance targets. • Continue to work on good performance indicators. • Emphasize flexible planning. • Going forward, emphasize proven strengths and phase out less productive initiatives in business climate reform. • Emphasize institutionalization of PPD – Competitiveness Council, business associations, administrative procedure laws, etc. • Provide more formal training in policy analysis and development to GoG. • Current initiatives in the AG Component should be enhanced including, for example, training in marketing and aid to cooperatives. • To organize better trading channels for farmers, the project should concentrate on the last link of the value chain, marketing. • Involve more agribusinesses in AG Component activities; develop special training needed for businesses. • Elaborate workable schemes for the formation of commercially oriented Farmers’ Groups; define incentives and, at the same time, highlight the importance of obligations that they will have when joining such a group. • It is necessary to intensify technical assistance to some VC enterprises or risk losing the value of the work that has already been done. • Some companies require permanent, daily assistance in management but remain skeptical of business service providers (BSPs); a solution is needed. • EPI should focus more on small scale networking activities that aim at the development of market linkages within the value chain and broaden and deepen the value chain. • Emphasize Private Sector Leadership. • Rethink the long-term potential of the apparel sector VC. • Vocational schools should become market oriented. • Look more closely at alternatives for access to finance (Pages 4-5). USAID 2017 TCB Evaluation | 77 Audit of USAID/Georgia’s Economic Prosperity Initiative - Office of Inspector General Audit USAID is one of several U.S. Government agencies implementing a billion-dollar assistance package for Georgia, pledged at an international donors’ conference held in the wake of the invasion. In September 2010, USAID/Georgia launched its Economic Prosperity Initiative “to improve Georgia’s overall economic competitiveness through assistance designed to improve both economic governance and private sector competitiveness.” To implement the project, the mission awarded a 4-year, $40.4 million cost-plus-fixed-fee contract to Deloitte Consulting LLP. As of June 30, 2012, almost midway through the contract, the mission had obligated $37.2 million and spent $23.7 million on the project (Page 1). The project has made progress on its five goals through activities in three components: an economic governance component that targets areas for interventions selected in partnership with USAID and the Government of Georgia, and two private sector components (agricultural and nonagricultural) that target market sectors selected by the contractor (Page 1). The audit also identified the following problems: • Progress was difficult to attribute to economic governance interventions. • The project’s design makes it difficult to link work on interventions in the project’s economic 2 governance component to project goals. This fact creates an incentive to do less in that component and more in the two private sector components. The project’s property rights effort was limited to intellectual property. • Although the contract lists a broad range of issues for the contractor to address in the “Strengthen Property Rights” area that it targets for economic governance interventions, the project’s action plan for this area is limited to addressing intellectual property rights. Lack of progress on other property rights issues impedes economic growth in Georgia. The mission did not establish a steering committee to set policy priorities. • Weak coordination among Georgian Government agencies, the project, and the mission led to the project’s intervening in a policy matter without the knowledge of the project’s main government contact. Reporting on two goals was not consistent with the contract. • The project cannot report whether productivity and employment increases are “consistent with USAID-approved semi-annual targets” in accordance with the contract. The contractor reported questionable results. • The contractor counted commitments to invest in the future toward meeting its $500 million foreign investment target, and import substitution and Georgian tourism toward meeting its $150 million exports target. Ambiguous contract provisions led to impractical data collection (page 16). The contractor interpreted two ambiguous contract provisions to require data that were burdensome to collect and of questionable relevance. Some contract requirements and benchmarks were not met (Pages 1- 3). To address these problems, the report recommends that USAID/Georgia. USAID 2017 TCB Evaluation | 78 • Approve a method for attributing to economic governance interventions progress toward project goals and modify the contract as necessary to enable the method to be implemented. • Issue in writing guidance on project design incorporating into contracts measures of success tailored to the feasibility of performance data collection and attribution for each project component. • Develop a written strategy for strengthening property rights that incorporates the full range of activities described on page 12 of the contract. • Establish a mechanism to bring Government of Georgia agencies together to prioritize interventions under the Economic Prosperity Initiative’s economic governance component. • Modify the contract provision on page 9 to clarify that the contractor is required to track each high-level goal by the targeted market sectors or components that can contribute to it. • Modify the contract provision on page 26 to clarify that the contractor can collect and submit performance indicator data to USAID/Georgia annually, if the mission agrees in writing that data cannot practically be collected and submitted twice during the fiscal year. • Modify the contract to either clarify or remove the requirement on page 10 that the contractor quantify the expected change in each identified World Economic Forum indicator throughout the project and as of project completion (Pages 2-3). Investment Development and Export Advancement Support Project The Mid-Term Performance Evaluation of USAID/Macedonia’s Investment Development and Export Advancement Support (IDEAS) Project – Mid-Term Evaluation USAID’s IDEAS project, implemented by Booz Allen Hamilton Inc. (BAH) in Macedonia is a 4- year, $5.1 million project, launched is 2010. The project is expected to conclude in December 2014. The project currently consists of the following components: 1) Industrial Policy Implementation (IP) and Investment Aftercare; 2) Export Promotion; and 3) Enhancement of Public-Private Dialogue (PPD). A fourth component, namely the Continued Professionalization of the Public Procurement System was discontinued, in consultation with USAID, in 2011 (Page 1). Upon careful and extensive document review, more than 30 key informant interviews, and qualitative data analysis collected from carefully identified sources, the assessment team concludes that the IDEAS project is a well-designed, well-managed, and ambitious economic growth project led by staff who are well-respected by the stakeholders. As discussed in detail in this evaluation report, the team has been particularly effective in the following subcomponents or activities: (1) export-promotion events, (2) capacity-building support provided to CoC, (3) capacity-building support provided to business associations and clusters, (4) support for building the Competitiveness portal, (5) support for the RIA process, (6) support for the establishment of the NECC, and (7) Continued Professionalization of the PPB System (Page 37). Best-practices training sessions for all stakeholders: Almost without exception, all key informants across agencies—public, private, and business associations—stated a strong interest in receiving training regarding best practices relevant to their organizations, agencies, and sectors. The participants voiced a particular interest in learning best practices and experiences USAID 2017 TCB Evaluation | 79 emerging from elsewhere in the region, and even globally. Knowledge transfer and the application of what has already been tested in similar circumstances through best practices are successful techniques, particularly in developing new structures and strategies, which appears to be the case in previous Macedonian experience. The assessment team enthusiastically recommends that the IDEAS project provide support in the area of best-practices seminars, forums, training sessions, and coaching workshops for all stakeholders when appropriate and useful to provide them with much needed know-how (Page 31). The assessment team finds that unwavering commitment at the GoM level also signals sustainability, as the GoM provides resources for sufficient staff, which are necessary if the training and transfer of knowledge provided by the project are to have sizable effects. An alternative approach to working with such GoM counterparts would be for the project to advocate goals to the GoM to obtain its buy-in for the project or its components prior to implementation to ensure that the GoM will not cause any delays. The assessment team does not recommend this alternative approach at this mid-point in the project and recommends instead that the project work with GoM counterparts on components that already have GoM commitment and buy-in to ensure that its target outcomes will be achieved within the project’s remaining time frame. (Page 30). During the assessment team’s initial meeting with USAID/Macedonia on October 22, 2012, prior to key informant interviews, USAID shared the interest of knowing whether the stakeholders would welcome having consultants hired by the IDEAS project embedded in their agencies. The assessment team asked all relevant stakeholders this question, and with the exception of one stakeholder, the participants stated that they would very much welcome having consultants hired by the IDEAS project working at the agencies, as long as the stakeholders were included in the hiring process. The only agency that was not in favor of this proposition was the MoE. The agency’s reason against this proposition was based on not having enough working space at their agency (Page 30). Moldova Business Regulatory and Tax Administration Reform Project Business Regulatory & Tax Administration Reform (BIZTAR) Project Evaluation – Final Evaluation According to this evaluation, the BIZTAR project is designed to improve Moldova’s business enabling environment by reducing the administrative burdens on the private sector, streamlining tax administration, curtailing opportunities for corruption, and improving the access for citizens and businesses to government services and information. In support of these objectives, BIZTAR was also designed to promote public awareness and support for policy reforms to contribute to a better business environment (Page 1). Based on the above conclusions, the Evaluation Team was able to derive several important recommendations that USAID may wish to consider either for the remainder of the current project or for any future follow-on BIZTAR-type project to be implemented in Moldova. • It is highly recommended that BIZTAR continues for the remainder of the project to build on the foundations of work accomplished to date (especially the practical implementation of legislative procedures and requirements) and to cement working relationships with counterparts and beneficiaries where activities are still ongoing or USAID 2017 TCB Evaluation | 80 are soon to be accomplished (tax appeals, Case Management System, tax current accounts, Customs brokers, Law on One-Stop-Shops, policy analysis for MTSI). • BIZTAR should consider its position regarding its interaction with the One-Stop-Shops, which appear in some need of continued support in terms of their sustainability. As further assistance is not anticipated before the end of the project, perhaps additional support could be revisited in a follow-on project. • At the present time, BIZTAR does not view continued support to Moldovan Custom s as a worthwhile or productive endeavor. However, given the fact that the project has already submitted a number of useful proposals to Customs officials and the MoF, it may be worthwhile for USAID to consider further support in any future possible follow-on project in order to move the process of harmonization with the Kyoto Convention a step forward. • BIZTAR should continue to collaborate closely with major stakeholders in the pursuit of common goals, especially those organizations such as the International Monetary Fund (IMF) and Sweden International Development Cooperation Agency (S IDA) that are involved in advising the GoM in similar activities to BIZTAR, e.g. tax administration policy and strategy, in order to avoid duplication of effort and providing mixed or conflicting messages to GoM. • There is a pressing need to ensure that projects of this nature have in place a recognized system of M&E reporting in order to track actual performance against pre￾determined objectives and targets on an ongoing basis. This will enhance project management decision-making, ensure a functional system of data collection and retrieval is in place, and ensure that Team members are aware of progress and achievements in each-other’s area of activity. • Whilst the World Bank’s Annual Doing Business ranking surveys are important they should not be the sole point of reference as a key economic indicator in highlighting a country’s year on-year performance, nor be used by projects to assess their respective contributions towards those indicators due to the difficulties in measuring them. Other indicators that can be used are Transparency International, the Millennium Challenge Corporation (MCC) Scorecard and the Global Competitiveness Ranking. A combination of these ranking systems would without doubt provide a country with a broader and fairer picture of the true extent of economic progress over time (Pages 2- 3). Regulatory and Energy Assistance Project (REAP) Performance Evaluation of The Regulatory and Energy Assistance Program (REAP) – Final Evaluation As part of the USAID mission’s strategy to harmonize the sector with European Union (EU) energy directives and help establish a regional energy market, USAID has been supporting Bosnia and Herzegovina’s (BiH’s) efforts to meet the provisions of the Treaty signed in Athens in 2005 and further bolstered by subsequent EU Directives, including the Second and Third Packages (Page v). The REAP project is part of USAID’s committed assistance, begun in 2000, to the BiH energy sector. As part of the USAID mission’s strategy to harmonize the sector with European Union USAID 2017 TCB Evaluation | 81 (EU) energy directives and help establish a regional energy market, USAID has been supporting BiH’s efforts to meet the provisions of the Treaty signed in Athens in 2005 and further bolstered by subsequent EU Directives, including the Second and Third Packages (Page v). Obstacles have been political in nature (or politically caused) and have included (i) the politicization of energy in BiH and the use of energy laws and energy institutions as a political tactic by certain factions in the RS as part of the ongoing fight over the nature of the State and the three-way structure of its institutions; (ii) a general lack of interest or sense of urgency at the state and entity levels concerning energy as an issue, due to a lack of understanding of the long-term potential benefits that could accrue to the country; and (iii) the absence of progress towards a national and integrated strategy for energy (Page vi). Some continued assistance beyond REAP is desirable, but it should be limited in scope and focused on facilitating, mediating, and providing neutral technical expertise where disagreements might arise between the various stakeholders, leading up to 2015 and carried over into assistance implementation on market opening. Continuing with the theme of restructuring and commercializing the electric power companies, as contained in the original SOW for REAP, should at least be explored, as, according to stakeholders, there is an ongoing need for the companies to better understand how to function in the future environment (Page vii). A fuller assistance package, which might include support in the development of a national electricity development strategy, should be considered only in light of subsequent political developments in BiH, including the internal political will to deal with the outstanding policy issues and the political will of the EU, since, according to informed respondents, they hold the power to break the deadlock of the current political struggles by withholding funds and applying sanctions (Page viii). USAID Agribusiness Project Serbia Agribusiness Project (SAP)Evaluation – Final Evaluation Overall, per this evaluation, SAP was specifically designed to increase the efficiency and competitiveness of Serbian agribusiness enterprises all along the value chain in targeted sub￾sectors complemented by efforts to improve the overall enabling environment for agribusiness. According to the project‘s website “this 5-year, $25.8 million economic development project, provides assistance to Serbian agriculture and agribusinesses with the aim of increasing agricultural sales and exports by Serbian firms and creating new employment in the six selected agricultural subsectors” (Page 1). From the available evidence, including commentary from the Ministry of Agriculture, there appears to be sound economic argument for any future technical assistance project to seriously consider the dairy and livestock/meat sectors in Serbia whose potential for real income and job creation seems an ideal opportunity for further analysis and pursuit. Given the importance of the livestock and meat sector to the economy of Serbia and with livestock numbers decreasing, it is strongly recommended that SAP revisits its strategy for this sector focusing on helping improve the quality of existing production, especially pig meat and beef (domestic market) and young beef and lamb (export markets). Farmers need to be updated in modern animal USAID 2017 TCB Evaluation | 82 husbandry techniques employed on similar smaller and medium sized commercial family enterprises across the EU and SAP should offer practical support to make this happen (Page 39). Given the almost universal acknowledgement of the need for a National Dairy Association to represent the interests of dairy farmers it is recommended that greater effort be placed on this endeavor by the SAP team since it is vital that such a body be established sooner than later (Page 39). SAP has identified a number of key areas where it could contribute to the growth vegetable subsector and help Serbia become a net importer of fresh vegetables. One of them was the need to encourage all-year-round production using greenhouses and polyethylene tunnels and use of cold chain storage facilities. This is a commendable approach and due to its importance, it is recommended that SAP takes a more active and leading role in bringing stakeholders and potential investors together to build on achievements to date (Page 39). Any future project should focus on how real impact can be achieved by planned interventions and, therefore, not be as wide-ranging in scope as SAP. A leaner, more results- driven project, will be easier not only to deliver on the ground but also in its management and administration including reporting structure as mentioned in the body of the report (Page 40). Such a project should also limit itself to activities with specific time parameters related to specific interventions. This means precluding activities related to policy-making and business enabling reform, which are time-consuming and dependent on too many factors outside of the control of technical assistance project (Page 40). A further suggestion worth considering would be building into the project a facility to recruit and train potential business consultants at the start of the project and retain them throughout the lifetime of the project to work with experienced international and local experts. This will serve to ensure a certain level of sustainability of service provision, particularly in agriculture and agribusiness. Demand for experienced locals to work with companies on local issues and delivery of certain types of training would advance this cause (Page 40). In the design of such a project, avoid the possibility of job “surfing” by local staff by offering a package of remuneration beyond simply a daily rate or monthly salary…Too often, local members of project teams are sidelined when it comes to working directly with clients on the false premise that the advice of international experts is more likely to be accepted. While this may be true in some instances (e.g. training in HACCP, ISO etc.) it is not the case in many other aspects where more than competent local experts in specific fields can easily deliver professional advice and training. This should, therefore, be encouraged at the start of any future project (Page 40). USAID Business Enabling Project Mid-Term Performance Evaluation of USAID/Serbia Business Enabling Project (BEP) -Mid-Term Evaluation The Mid-Term Evaluation conducted on BEP characterized it as a 5-year, $15.48 million initiative launched in January 2011. As stated in the SOW, the purpose of BEP “is to help the GoS improve the competitiveness of the Serbian economy and its private sector businesses.” The project supports Assistance Objective 1: More Competitive Market Economy, IR 1.1 Business USAID 2017 TCB Evaluation | 83 Enabling Environment Improved, and IR 1.4 Macroeconomic Environment Improved. BEP aims to improve the ability of all businesses in Serbia to compete in global, European Union (EU), and domestic markets. (Page 1). BEP has been very effective in identifying stakeholders and focusing their attention on key issues in reform groups, and there are good examples where they have molded them into effective partnerships for reform. The project has also built support through outreach programs to inform stakeholders and public. On the operational level, BEP has worked very effectively with a number of its grantees to gain deep insight into the nature of problems. It has also provided support to its partners in public bodies through training programs and the Internship Program, which are designed to bolster their capacity. Finally, BEP has paid special focus on building relationships with women’s organizations and ensuring they are represented in its initiatives. A review of the list of project beneficiaries and partners shows that BEP has been successful in identifying all important stakeholders in the public and private sectors, and has integrated them into activities. Where this was not possible (e.g. Competition Commission), the project withdrew from the activity with USAID’s agreement (Page 26). Recommendations Include: • Extend partnerships to include a wider base of stakeholders representing small businesses and exporters. • BEP should assess what else could be done to ensure that all stakeholders are sufficiently familiar with the objectives. • BEP should continue to improve the quality of economic journalism and develop new initiatives. • Grants should be continued to provide essential inputs for BEP activities. • Provide advocacy training to less experienced partners (Page 28). • USAID Competitiveness Project USAID Macedonia Competitiveness Project Evaluation -Mid-Term Evaluation For the evaluation, the Team focused on assessing the extent to which MCP activities are increasing the competitiveness of Macedonia’s firms (Page 1). Recommendations include: • More emphasis should be placed on helping firms implement improved managerial and production practices that the market indicates. • Emphasize export opportunities and access to credit as lead ―carrots to motivate firms to participate in Project activities. • Continue FDI activities to strategic investments and pre-existing leads. • Portfolio managers and all Project staff should ensure all client firms are fully aware of available Project services on a continuous basis. • A future USAID vocational training activity should work on the task of reforming vocational education and training for the apparel industry, including collaborating with the Ministry of Education. USAID 2017 TCB Evaluation | 84 • Spin off the internship program, for instance to a new USAID workforce development program. • Consider adding a new indicator: Number of firms implementing improved management or production practices (Pages 2-3). Central Asia Afghanistan Trade and Revenue (ATAR) Project Mid-Term Evaluation (June 4 – August 23, 2015): The Afghanistan Trade and Revenue Project – Mid-Term Evaluation ATAR is designed to strengthen the business climate of Afghanistan to enable private investment and enhance trade, job creation, and fiscal sustainability through Afghanistan’s critical Transition Period, and continuing into the Transformation Decade (Page 1). Despite its dissatisfaction with some aspects of ATAR’s support, the MoCI is overall pleased and grateful for ATAR’s support regarding Afghanistan’s WTO accession process (Page 8). We found more capacity within ministries to draft procedures than laws, amendments, and regulations, although this also varied from ministry to ministry. Again, this variation may be attributed to the method ATAR used. ATAR usually drafted procedures in a working group setting, where relevant stakeholders discussed the reason for the particular procedure and the best process to accomplish the stated reason. With ATAR’s assistance, MAIL, MoPH, and ANSA all stated that they had developed some level of capacity to draft procedures. They believe they can sustain this capacity but need more support to further develop it (Page 16). ATAR should adopt a policy of providing technical assistance and capacity building on site, if possible, and in a working group setting. If security restrictions limit the implementation of such a policy, experts should be available to provide in-depth overviews of the model laws and regulations, followed by technical assistance when needed (Page 4). Senior MoCI officials should adopt a policy of engaging middle-level staff when discussing negotiating strategies, have them participate in meetings to the extent feasible, and placing some responsibility on senior staff to develop the negotiating skills of middle-level staff (Page 4-5). EPAA might consider establishing an online/web-based market place that links regional businesses and traders. ATAR should also consider working more closely with Afghanistan’s Offices of Commercial Attaché in the CARs relating to business-to-business linages. (Page 5). Agricultural Modernization Grants (ACE Program) Agricultural Modernization Grants: Midterm Impact Evaluation – Mid-Term Evaluation The Agricultural Credit Enhancement (ACE) Program was launched in July 2010 with the fundamental purpose of building the institutional infrastructure of the Agricultural Development Fund (ADF) (Foreword). Over the past few years Afghanistan agribusinesses have witnessed the highest rates of countrywide urbanization. Is ADF up to speed on how such changes represent opportunities for USAID 2017 TCB Evaluation | 85 farmers and food processors? Not yet, but perhaps it is time to invest some grant money in evaluating how Kabul and other cities demonstrate the speed consumer preferences are evolving across different economic strata (Page 31). ADF should be poised to support agribusinesses that understand these dynamics and that can provide their own share of resources. A method needs to be put in place so that new agribusiness ideas are promoted and more loan packages are facilitated. This is not a reference to businesses in an embryonic stage (like the examples of business incubators). Rather, it is a recommendation to identify successful business ideas that can be scaled up and diversified vertically down to other nodes and horizontally across nodes to involve more participants (Page 32). Feasibility studies and market-potential assessments are two additional ideas that can be financed by the existing grant mechanism to shed light on good opportunities and attract private sector participation (Page 32). A more detailed categorization of potential agricultural credit users in both traditional and innovative work areas has to be developed by learning from the existing, positive examples gleaned from the first two years (Page 32). Agro-ecological conditions to produce high quality outputs certainly matter, but the buying capacity of markets and the existence of actors to move a product seamlessly to the consumer is a work of strategy, access to information, and collaborators. Selecting a number of leaders from successful companies and using them as sounding boards to analyze innovative credit ideas would be beneficial. In addition to ADF’s established loan evaluation process, future decisions of that projects are granted with ADF loans could use expert opinions from successful entrepreneurs (Page 33). At the input level, it has been widely documented that investments in seed and fertilizers are the largest expenses when generating higher productivity and output (Page 33). It is important to keep in mind that farmer access to inputs at competitive prices depends entirely on the business environment in which the farm stores operate. If the farm stores are isolated from efficient input procurement networks, the high prices transmitted to farmers are equally non￾competitive. Exploring innovative models on how increased input financing can be done should be a priority in ACE’s 2013 strategy and beyond (Page 34). While there is practically no knowledge of similar services available in Afghanistan for other industries, a grant with a commercial bank can be initiated where, for example, ADF can put in $500,000 with the bank putting in another $500,000, for an initial factoring fund of $1M… As might be expected from a banking sector poorly experienced in agribusiness, only certain products will be selected for this service. In general, those products will be the least perishable items such as nuts, dry fruits, spices and juices non-perishable items such as cashmere (Page 36). One of the lessons ADF learned is the creation of CMUs If learning the ropes in agribusiness takes a special set of skills where commercial banks may not have an edge- CMUs are the solution to fund small, low-cost units that have the minimum expertise to understand agribusiness. CMUs are a tailor-made remedy to fill the knowledge gap between agricultural cycles and business relationships (Pages 35-36). Final Performance Evaluation (March – May 2015): Agricultural Credit Enhancement (ACE) Program – Final Evaluation The objective of USAID’s Agricultural Credit Enhancement (ACE) project was to provide holistic support to advance agricultural modernization through financing, technical assistance, USAID 2017 TCB Evaluation | 86 and policy reform, which would contribute to increased agricultural jobs, income, and Afghans’ confidence in their government (Page 1). Modernization of Agriculture and Effects on Jobs and Income: • Grants should not be used where ADF can give loans, but be used to complement its work through capacity building and support. • Use grants as a one-off strategy to enable recipients’ graduation to economic sustainability. Hence, avoid recycling grant recipients across USAID projects in order broaden the scope of US government support and impact. • Explore and grow the range and capacity of intermediaries able to deliver credit in-cash or in-kind to the agricultural sector (Page 5). Efficiency of the Non-Financial Intermediary Approach: • To scale agricultural credit uptake, future US government efforts should support ADF in the identification of viable NFIs with capacity to intermediate agricultural credit effectively and on a large scale. • Intensify awareness campaigns to market ADF and the acceptability of Sharia-compliant loan products. Engage with religious leaders whose opinion is respected (Page 5). Inclusion of Women: • Identify and support promising women-owned businesses and NFIs that support women in order address any challenges and constraints they face and expand opportunities for women. • As an affirmative action, adjust the minimum lending limit specifically for women from the current $100,000 to a lower limit, as appropriate. • Consider the possibility of expanding the Zahra loan product to other microfinance institutions (MFIs), and opening it up to include women-operated businesses in other sectors. • Tap into the work of the MAIL Directorate of Home Economics as well as other women’s organizations to identify more women for capacity building, and link them to financial services such as Zahra. • As a way of empowering women, future support should be used to promote corporate social responsibility in agri-processing businesses that employ more women (Pages 5-6). Availability of Agricultural Credit: • Embark on an aggressive awareness building campaign to further stimulate demand for agricultural credit. • Finalize the issues related to ADF’s legal status with experts well-versed in Afghanistan’s legal system, and its non-bank financial intermediary status as it relates to tax issues with partner FIs/MFIs (Page 6). Business Profitability and Contribution to the Agricultural Economy: • Consider supporting more food safety trainings to encourage food producers, processors, and exporters to meet international standards. USAID 2017 TCB Evaluation | 87 • Support Afghan businesses in general with trainings on how to handle international business transactions to safeguard their business interests (Page 6). Incentives Driving Economic Alternatives: North, East, and West (IDEA-NEW) Audit of USAID/Afghanistan’s Incentives Driving Economic Alternatives for the North, East, And West Program - Office of Inspector General Audit IDEA-NEW was created to prevent the growing of poppies, in Afghanistan, by providing greater access to licit income sources. The report recommends that USAID Afghanistan: • Assess the focus and location of project activities, as necessary, to maximize the program’s contribution to dissuading Afghans from growing poppies; and define and use intermediate results, performance indicators, baselines, and targets to assess progress toward that underlying objective. • Modify the cooperative agreement to reflect changes in the program to date. • Assess and realign the program budget, as necessary. • Remind its staff in writing about the importance of documenting significant meetings, discussions, and decisions that affect USAID programs. • Develop and implement a risk-based monitoring plan for the program that includes periodic site visits. • Require that its implementing partner to develop and implement a formal monitoring system that includes (1) the development and execution of annual monitoring plans covering the programmatic and financial aspects of the program; (2) reporting and analysis against those plans; (3) the inclusion of subimplementers in those monitoring plans, reporting, and analysis; and (4) the verification of reported results, including supporting documentation. • Provide its implementing partners with training covering (1) the definitions of, the collection of, and the reporting on performance indicators for the program and (2) the requirement for and the maintenance of evidence, including documentation, supporting reported results. • Establish procedures to verify on a periodic basis the accuracy of the reporting of its implementing partners, including the verification of reported results with supporting documents at the lowest operational level. • Require that its implementing partner (1) develop a detailed written strategy for building sustainability into its activities and projects, (2) develop indicators of sustainability for its activities and projects as well as for the program’s overall objectives, (3) develop targets for its sustainability indicators, (4) include in its annual work plan both a narrative addressing sustainability during the work plan period and annual targets for its sustainability indicators, and (5) report in its quarterly and annual reports on its progress in the area of sustainability and against its sustainability indicators and targets. • Have a gender assessment performed for each component of the program, and incorporate the results of those gender assessments in its IDEA-NEW activities to increase the participation of women. USAID 2017 TCB Evaluation | 88 • Require that its IDEA-NEW implementing partners provide training to their program personnel on how to integrate gender issues into activities, the constraints women confront to participation in the economy, and ways to overcome those constraints. • Develop mission-wide policies and procedures governing cash-for-work and community constructed infrastructure projects. These policies and procedures should include, but not be limited to, the employment of minors, safety of workers, and responding to injuries in compliance with Afghan labor law. (Pages 3-4). Trade Accession and Facilitation for Afghanistan Mid-Term Evaluation Trade and Accession Facilitation for Afghanistan (TAFA) Project – Mid-Term Evaluation TAFA’s goal is to improve the trade environment of Afghanistan with a specific focus on cross border trade and transit to foster increased investment, exports, and market-based employment opportunities (Page1). Recommendations • Focus on trade, tariff reforms and WTO accession supported by robust economic analysis. • Deepen engagement with stakeholders to gain buy-in with other governmental agencies and private sector for Trade Agreements and WTO Accession, targeting new emerging business leaders and groups. • Increase the emphasis on standardization of reforms. • Facilitate private sector logistics services such as cargo handling, insurance, warehousing, storage facilities, transportation and trade finance in order to expedite cross-border movement. • Assist in resolving issues created by guilds, certificates of origin, and licensing to reduce transaction costs. • Engage female business-owners, faculty and government staff by instituting a program at a university or think tank to build a cadre of well-informed women who can lend their voices to WTO accession, trade agreements, standards, and customs. • Leverage journalists in the Press Club to develop media products that promote the trade reform agenda and the benefits of WTO accession (Pages 2-5). East Africa Agricultural Growth Program: Livestock Market Development Agricultural Growth Program: Livestock Market Development External Mid-Term Performance Evaluation Report– Mid-Term Evaluation AGP‐LMD, in concert with the crop value chain program (AGP‐AMDe), serves as part of USAID’s contribution to the Government of Ethiopia’s Agricultural Growth Program (AGP). The AGP goal is “to end poverty and enhance growth”, which aligns with the FtF goal to “sustainably reduce poverty and hunger” (Page iii). USAID 2017 TCB Evaluation | 89 Recommendation • Develop custom PMP indicators to measure and record the outcome and impact of the training programs and the grants program; and their contribution to the achievement of project goal and objective. • Greatly expanding support for micro‐ and small enterprise (MSE) development in the rural areas adjacent to livestock producers (first point of sale) as the focus of implementation during the remaining Life‐of‐Project. • Substantially increasing the role and responsibilities of the local Implementing Partners, making them directly responsible for the design and implementation of the MSE development program recommended above (Pages v-vi). CNFA is doing a credible job of implementing the project, and we believe that AGP‐LMD is contributing to the achievement of the AGP goal. But the project contribution to the achievement of the FtF goal is not clear, project activities lack synergy; and project results and impact are not being methodically measured. Therefore, in summary we recommend that implementation during the remaining LOP focus on MSE development in areas adjacent to rural livestock producers to increase impact and project contribution to the achievement of AGP and FtF goals (Page vii). East Africa Trade Hub (EATH) East Africa Trade Hub (EATH) 2009-2014 Final Evaluation Report - Final Evaluation The Program was tasked with promoting regional growth and integration in EAC affiliated countries, as well as expanding East African exports to the United States under the AGOA Trade Preference Program (Page 1). EATH successfully drove improvement in key aspects of the regional policy and regulatory environment for trade through the EAC Secretariat… As these procedures begin to be fully implemented, it will be very important to ensure that implementation practices in these areas (customs procedures, customs valuation and post-clearance audit) are being widely and consistently applied across the EAC member states (Page 18). Future Trade Hub initiatives could effectively capitalize on the momentum EATH generated in this area through a strong emphasis on EAC-level interventions that promote fuller administrative compliance with mandatory harmonized customs procedures (Page 18). Future programs may want to consider the advantages of embedding national advisors in each focus country, in order to further strengthen the trade policy and institutional reform dialogue and consensus-building process with stakeholders (Page 19). It may be useful for future programs to look at strengthening harmonized border procedures with trading countries outside of the EAC (given the important facilitative role this could play in further expanding trading opportunities for EAC countries and encouraging cross-regional harmonization of core trade and transit procedures). In particular, with Southern Sudan coming on board as a partner state and the establishment of the Lamu Port-South Sudan Ethiopia Transport Corridor (LAPSSET), these countries represent an important target for technical/training support (Page 19). USAID 2017 TCB Evaluation | 90 While EATH has significantly raised stakeholder consciousness, generated Secretariat level support in critical areas (e.g., RFBS and Harmonized Staple Food Standards) and launched important pilot implementation initiatives, comprehensive implementation of key reforms will require greater coordinated buy-in on the part of key national level policy institutions. Full implementation of EATH-sponsored reforms in critical staple food policy/institutional reform areas will require a major recalibration of support strategy – with greater targeted emphasis on technical/policy dialogue support in key national-level sector ministries. The placement of new EATIH project representatives at the national level may represent a key part of the solution to better addressing these needs (Page 29). … it may be important to consider more strategically focused warehousing and bulk purchasing solutions that can impact larger numbers of small-holder beneficiaries (Page 29). USAID and EATIH should utilize the recent WTO Trade Facilitation Agreement as a policy dialogue and technical engagement lever to promote follow-through implementation progress across EAC member states on key phytosanitary compliance themes (e.g., joint protocols on pest risk mitigation, traceability, and inspection processes); building off of the institutional momentum achieved in the PRA standards area through EATH (Page 30). Follow-on export promotion programs should be in a position to take EATH’s export outreach support activities to the “next level”. This can be achieved both by significantly scaling up the projected purchasing volumes that are expected to emerge from these market-making initiatives; and by promoting a sustainable plan for direct self-sustaining sponsorship over time of this participation by regional sectoral partner organizations (along the lines achieved with ACTIF and AFCA). This could reasonably become a key explicit conditionality feature of future outreach support to key partner entities under follow-on Trade Hub support initiatives (Page 37). Future export promotion programs would benefit from focusing to a lesser degree than did EATH on input/output-based targets, and more on sustainable impacts on both firm and sub￾sector level outputs/outcomes. This, for instance, could include a hybrid-focus on critical ‘anchor firms’ whose enhanced competitiveness can drive improved branding and supply chain competitiveness impacts across the value chain. It would prove useful to express such targets in percentage export growth terms in relation to historic AGOA export performance, since it appears that the dollar volume targets set for export expansion under EATH were not particularly ambitious given (1) the relative magnitude of overall export and non-traditional export volumes for the region; and (2) the substantial support efforts already implemented under prior regional programs in this area (Page 37). Explicit financial and operational autonomy goals for key regional representative organizations should be established under follow-on trade-related support programs, including EATIH (page 37). In retrospect, a greater focus on driving consensus and building implementation capacity within key national level technical counterpart units (e.g., customs, trade, and agriculture) may have yielded greater/more systemic implementation progress at the national level. This should be taken effectively into account in the allocation of technical assistance/training resources under the follow-on EATIH Program (Page 39). USAID 2017 TCB Evaluation | 91 …future programs should consider building in stronger performance incentives with associative partner institutions that encourage more intensive support to high potential firms. This could, for instance, entail a focus on concentrated support to critical anchor firms in key sectors, whose growth can spur supply chain development at the national/regional level (Page 39). Future programs should establish a clear and well-defined set of sustainability objectives in the PMP, and related institutional autonomy/viability related performance targets for key partner institutions (Page 40). Future programs should establish more transformational performance targets for export growth that are directly tied to projected percentage growth in AGOA export revenues over the prior year’s performance, or in relation to a multi-year average (Page 40). Africa Trade Hubs Export Promotion Evaluation – Evaluation Report The United States Agency for International Development (USAID) contracted with Social Impact, Inc. (SI) to conduct an evaluation to assess the effectiveness of firm-level export promotion activities of the Africa Trade Hubs Projects, which had been in operation for more than a decade (Page 1). Nontraditional export sectors (i.e., non-petroleum and non-mineral) increased from $232 million in 2001 to $607 million in 2011. Noteworthy in this respect is that the apparel and textiles sector, which declined following derogation of the Multi-Fibre Arrangement in 2005, and then remained fairly stagnant for several years, appeared to be back on an upward trend starting in 2011. Of the other sectors, food and food ingredients appeared to be the most dynamic (Page 38). USAID should consider requiring the Trade Hubs to continue market-linkage work with firms either directly or through associations. Quantitative evidence from both the WATH and ECATH shows that firms report that this type of assistance has had the most impact on increasing exports. Qualitative interviews demonstrated that the best method for assisting firms in this regard is through a step-down process, meaning firms are supported more heavily (financially and technically) when they first begin working with the Trade Hub (or association) and then, slowly, support is lessened until firms are supporting themselves entirely (Page 1). USAID should consider requiring the Trade Hubs to expand efforts to reduce firm constraints to exporting by facilitating firm level access to finance and reducing regulatory barriers to exports. Results of regression analysis from the ECATH show that firms that have received support in accessing finance were more likely to have increased exports than firms that did not. In addition, access to finance and regulatory barriers were highlighted by firms in both ECA and WA as the main barriers to firms’ export growth (Page 1). Despite these quantitative results, during qualitative interviews, the majority of firms reported that assistance with trade shows was the most important type of assistance that they had received from the Trade Hub… Firms that asked for additional technical assistance from the Trade Hub were looking for assistance with tailoring their products to the international market, ensuring the quality of their products for the international market, building the physical capacity of their firms to fill orders (and to understand how many orders are feasible), benchmarking, conducting market research and liaising and closing deals with potential buyers (Page 45). USAID 2017 TCB Evaluation | 92 While evidence does not show that ECATH assistance is more effective in generating jobs, sales and exports in some countries than in others, it is more effective at increasing exports to the United States than to any other region or country, likely because of AGOA (Page 51). Ethiopia-WTO Accession Plus Project USAID/Ethiopia WTO Accession Plus Project Evaluation - Final Evaluation The USAID/Ethiopia Accession Plus Project is designed to continue the pursuit of Ethiopia’s WTO accession while simultaneously promoting broad business environment reforms that will increase growth and economic development and enhance the benefits of WTO accession (Page 2). The Project should maintain its focus on WTO accession, but should focus on commercial law revision only to the extent such laws relate to or impact specifically on Ethiopia’s obligations and commitments under the WTO. Other donors are currently working on regional integration and the Project’s intervention in the area would be a duplication of what is being done (Page 18). More focus on the private sector is necessitated by two important considerations. First, members of the business community are the ones who would directly be affected by the consequences of the accession and making sure that they understand such consequences is important. Second, the business community lacks the capacity to pinpoint its positions and priorities and to influence government negotiation agenda (Page 18). Building the capacity of the core team of negotiators through a high level training on negotiation is a critical intervention with a huge benefit to the accession process… The Project should strive for a more synergetic sharing of assistance activities with other donors and agencies directly or indirectly working on WTO accession. Some donors are now working on Ethiopia’s WTO accession. These include, the European Union, World Bank, USAID AGOA Plus Project, World Trade Organization, GTZ, Japanese Embassy, JICA, Finland Embassy, and UNCTAD. A more synergetic collaboration with such projects would enhance the impact of Project activities and avoid duplication of activities (Page 19). Some likely key issues in the negotiation include: transparency, telecom, financial sector, intellectual property, customs, quality and standards, and state trading enterprises. A focus on these and other areas likely to be contentious would help in expediting the accession process (Page 19). It is absolutely important that the Project works very closely with the MOTI staff and in particular the Trade Relations and Negotiation Directorate and to facilitate this close interaction the Project need to remain in the premises of the MOTI… The COP need not be a lawyer since WTO rules and agreement-specific issues are more likely to be conferred on short-term technical assistants (Page 20). Feed the Future Agricultural Inputs Activity Feed the Future Uganda Agriculture Inputs Activity: Strategic Assessment Report June – July 2015 – Strategic Assessment USAID 2017 TCB Evaluation | 93 The goal of the Feed the Future Uganda Agricultural Inputs Activity is to increase the use of high quality agricultural inputs in Uganda (Page i). The Activity reported “evidence of a small number of instances where Ag-Inputs directly influenced business changes” among agro-dealers… The assessment found that the current dysfunction of the agricultural inputs market has conspired against the adoption of customer service business strategies by agro-dealers and their retail customers. The expanded presence and increased sophistication of counterfeits in the market is one major reason for this market dysfunction... With a little bit of capital and a simple trading license, it is relatively easy for an individual to enter this business. As a consequence, there is a high degree of business births and deaths among start up agro-dealers (Page ii). The Government – in this case the Ministry of Agriculture, Animal Industry and Fisheries (MAAIF), Ministry of Trade Industry and Cooperatives (MITC), and Ministry of Local Government (MOLG) -- has been ineffective in regulating the ag-inputs industry: its own licensing and standards regimes (e.g. E-Verification and seed certification) are weak; and direct government interventions in agricultural inputs supply (e.g. the free distribution of seeds) crowds-out private sector initiative (Page iii). The Ag-Inputs Activity should focus interventions on increasing barriers to entry into the agro￾inputs business, to limit further growth of less credible agro-inputs dealers, including briefcase firms (Page iii). Kenya Horticulture Competitiveness Project (KHCP) Compendium Report Multi-Stakeholder Evaluation of Agriculture and Livestock Value Chain Activities in Kenya - Multi-Stakeholder Evaluation The KHCP, with its private-sector partnership, is a model whereby very small producers can participate in the lucrative export market for cut flowers. KHCP has improved the lives of its participants. The team observed recently built new homes, children in fresh uniforms attending schools, and improved rural communities as a direct result of participation in the project. (Page 13). Actively engaging the private sector in horticulture value chain intervention activities was seen to contribute considerably to the success of the project. KHCP involvement of the private sector begins when the project invites stakeholders to a national or regional project launching forum. Thereafter, the stakeholders interested in participating in the project send concept notes detailing how they would like to participate and how they will contribute to achieving the project objectives. The selected partners enter into a formal agreement with KHCP, which spells out the relationship and roles between the two (Page 219). All the partners involved in the projects derive benefits from the partnership. The private companies improve on their business performance, the smallholder farmers improve their household incomes, and the project achieves its objectives. The project builds on what the private sector company is doing and helps the company do even better as long as it contributes to meeting the project objectives. In other words, the partnership does not move the private sector partners away from their core business, but rather strengthens them. (Page 222). USAID 2017 TCB Evaluation | 94 The most obvious recommendation identified by the team as related to donor-funded horticulture projects is to make collaboration and cooperation a priority; rather than an afterthought (Page 232). What were the greatest strengths of successful activities, and the most important lessons which can be learned from them regarding the design and implementation of new agriculture and livestock value chain activities in Kenya? Strong and well-defined private sector participation in project implementation and design of partnership arrangements. The private sector partners were selected through competitive bidding and developed proposals for the partnership support. Building the capacity of both the smallholder producers and private sector partners helps in improving value chain performance as well as enhancing sustainability (Page 246). Mozambique Agribusiness and Trade Competitiveness Program Performance Evaluation of the USAID/Mozambique Agriculture Portfolio – Final Evaluation This is a report on an independent, external evaluation of the agricultural program funded by the United States Agency for International Development (USAID) Mission in Mozambique, Office of Agriculture, Trade and Business (ATB) (Page 1). This evaluation covers seven programs including AgriFUTURO. PPPs created under the AgriFUTURO grant mechanism are an effective means to leverage ATB resources and activities. They provide more sales by small farmers and increase agricultural exports (Page 3). Conservation agriculture is a highly promising method to improve small farmer yields and food security, but commercial production requires advanced delivery mechanisms for input supplies, demand-driven markets, market linkages, and the availability of finance (Page 2). The international research centers are effective for new technology development but their impact is limited due to weak government and private extension as well as weak input supply channels, which limit uptake (Page 2). The lack of credit for agricultural production and marketing is a major problem that has not been effectively addressed by the ATB projects (Page 2). The micro-lending programs carried out by Banco Oportunidade are an important pioneering effort that merits further support (Page 3). In view of weak government and private extension services, all the international research centers operating through the PARTI platform provide extension and outreach services that benefit small farmers. Linkages between farmers and suppliers of agricultural inputs and services are weak, as a result of weak input supply chains. The greatest developmental impact is obtained by linking farmers through their producer organizations with reliable markets for their products (Page 3). A commitment to institutional “ownership” by their members, along with effective management, marketing, and financial skills of their leaders, are key requirements for the sustainability of ATB￾supported farmer-owned businesses (Page 5). USAID 2017 TCB Evaluation | 95 Although local governments desire better coordination with NGOs and donor projects, their present level of coordination is largely ineffective. In general, local governments have neither the resources nor the skills for effective coordination (Page 5). Audit Of USAID/Mozambique’s Agriculture, Trade, and Business Portfolio - Office of Inspector General Audit AgriFUTURO aimed to increase the competitiveness of Mozambique’s private sector by developing agricultural value chains (Page 2). AgriFUTURO had been under way for 2.5 years and was 15 months away from completion when, according to ATB’s director, USAID/Washington insisted that FTF be incorporated into the mission’s existing portfolio… Mission staff said it was appropriate to incorporate FTF into an ongoing project because it was seen as a “refinement” or small change. However, closing and opening offices in new provinces, removing and adding value chains, and shifting the focus to small farmers were significant, disruptive changes that diluted the focus (Page 10). The grants program was mismanaged because (1) the project’s staff did not have much experience managing grants, especially those that involved construction projects, (2) they were pressured by USAID/Mozambique to award grants, (3) grantees were not vetted properly, and (4) administrative bureaucracy led to delays in approving the grants manual and processing the awards (Page 14). Global Program for Biosafety Systems Performance Evaluation of the Program for Biosafety Systems – Mid-Term Evaluation This evaluation looked at the first 5 years of Program for Biosafety Systems (PBS) to help provide direction for the next 5 years. USAID has been funding efforts in biotechnology and the regulation of agricultural biotechnology for nearly two decades across several partners and host countries in Africa and Asia. The Program for Biosafety Systems (PBS) is the largest of USAID’s investments that support the development of science based regulatory systems that govern the application of these new agricultural technologies (Page 3). In achieving the success that it has in the technical areas, PBS has focused quite narrowly on the relatively few people who need to be involved in the process of implementing confined field trials and addressing the technical aspects of risk assessment. This has been successful because the governments in the selected countries were supportive of the technology – as long as there was relatively little political cost. While the funding was from external sources and there was little negative public reaction, the activities in implementing a confined field trial system had little downside to government officials or politicians. However, as the products moved further towards commercial approval, decisions on legal frameworks (potentially requiring laws to be passed) and funding for regulatory activities now have the potential to be controversial. As the situation evolves, PBS has become more focused on broadening its outreach to other stakeholders that may impact the regulatory system. This has required an augmentation of communication skills, expertise and resources in the program.program and has resulted in a USAID 2017 TCB Evaluation | 96 more integrated approach such that technical capacity building and communications capacity building are now developed in tandem. PBS has a challenge to effect this change in their focus – while at the same time retaining the credibility and trust which has been built up during the technical training and development (Page 28). As the situation is changing, the future role of PBS could be considered through how to strike a balance between two extreme options, detailed below. It is not suggested that PBS chooses one of these two extreme positions, and in reality, it is expected that some course between these two will be chosen, but in choosing the particular course, the potential for conflict between these goals needs to be recognized (Page 28). Option 1. PBS remains an agency for technical support, narrowly focused on getting a relatively small number of people trained in the review and monitoring of confined field trials and risk assessment of applications for commercial release, staying away from the political advocacy areas needed to get such cultivation applications approved. PBS is certainly good at this – and has a track record in a number of countries. This would retain the reputation of PBS as an honest broker and might even have benefit for USAID as it will be seen not to be an arm of big US companies. However, this will miss the major goal of USAID – giving farmers in these countries more options to improve their productivity and profitability and thus not be compatible with the Feed the Future mandate (Page 29). Option 2. PBS converts to an advocacy organization, utilizing its linkages and contacts to push for an enabling environment through the adoption of (possibly) new laws, regulations and the setting up of regulatory agencies. This will require quite a large alteration in the make-up of PBS staff and also – quite likely – a different kind of budgetary system, given the potential need for increased funds to be spent on public relations and – possibly – covering the funding of positions within local government agencies to facilitate the process. This is something which will more directly advance the Feed the Future goals, but is quite different to the current PBS structure and may be difficult to operate within the current system of an award to IFPRI (Page 29). Middle East and North Africa Investment Climate Improvement (ICI) Mid-Term Evaluation of Investment Climate Improvement Project: Fiscal and Policy Components– Mid-Term Evaluation The ICI project provides the technical assistance needed to respond to necessary business enabling environment reforms within the West Bank and Gaza, as well as assist in transforming the Palestinian Authority’s fiscal sector (Page 8). Based on USAID’s successful past experience in countries such as El Salvador and Egypt, the capacity-building process for major improvements in tax administration – major upgrades and development of the IT system, changes in business processes, and the recruitment, training, and retraining of staff to implement the new systems – may take a decade or more of on-going resident technical assistance by USAID project teams. However, the ultimate payoff is potentially extremely high and can be measured in many cases by both improved revenue performance for the government and reduced burdens on taxpayers. Alternatively, the USAID USAID 2017 TCB Evaluation | 97 assistance could be provided in a project that combines both tax and business regulatory components (such as had been the design for ICI and has been done for similar recent USAID projects in Georgia and Moldova). However, in the West Bank, the potential synergy between the two components is rather limited to a few areas – such as (a) simplifying combined registration requirements for business licensing and tax purposes (for example, implementation of one-stop shops) and (b) development of data interfaces and exchanges between the MoF revenue authorities and other PA ministries (including MoNE and MoI) (Page 26). Experienced development practitioners know that personal relationships between advisers and their counterparts is one of the single most critical ingredient in successful delivery of technical assistance. Given its history, the ICI project has been challenged in its ability to create a strong foundation of mutual trust and respect with its PA counterparts. ICI’s staff turnover and that which seems to be endemic within the PA creates additional hurdles for effectively implementing a program of technical assistance. Ongoing and future efforts in supporting the MoNE would benefit from building strong relationships both at higher political levels as well as at the working/technical level (Page 27). Timely technical assistance depends on resourcefulness in providing highly qualified short term expertise. The range and complexity of the specific tasks often requires highly specialized knowledge and experience. In order to reduce costs, experts can be drawn from USG and local government agencies in the United States. Many of these experts can provide long-term assistance through follow-up engagements to ensure consistency and completion of the endeavors they supported—and with the added benefit of developing relationships that can improve outcomes. Other USAID missions around the world routinely call upon experts from U.S. federal and local departments, agencies, private institutions, as well as foreign experts to render technical assistance. Best practice would be to employ this technique on an expanded scale in order to maintain relationships with PA counterparts throughout the entire project and cycle (Page 27). Jordan Fiscal Reform Project II Performance Evaluation of the USAID/Jordan Fiscal Reform Project II – Final Evaluation The goal of FRP II is to “create a stable macroeconomic environment that fosters economic growth by improving economic policy, public financial management, and the business environment.” To achieve this goal, the project was designed to provide technical assistance to the Government of Jordan (GOJ) in the areas of ministerial capacity development (including the Audit Bureau and Ministry of Public Sector Development (MOPSD), tax reform (including drafting new tax legislation), tax administration reform, budget reform, instituting Results￾Oriented-Budgeting (ROB), establishing a new government wide budgeting system, budget forecasting and to conduct economic analyses related to policy, customs and trade facilitation. (Page iv). USAID should continue support to GOJ to build capacity, consolidate reform gains, and build upon them (Page vii). USAID 2017 TCB Evaluation | 98 USAID and project implementers should assess the commitment level of government officials/institutions before engaging in future reform efforts. USAID should use measured conditionality as a means to advance a reform agenda in Jordan (Page vii). USAID should continue its investment in supporting institutional and capacity building improvements taking into account the lessons learned through the FRP II project (Page vii). USAID and project implementers must plan an adequate integration of the trilogy of the main fiscal and PFM reform players: ISTD, GBD, and the Macro-economic/fiscal Department of the MOF. The integration of USAID assistance in these three fiscal components is a cornerstone for the success of any fiscal reform project (Page vii). A comprehensive fiscal reform should include local governments. USAID should assist the local governments in developing local PFM systems where local governments have some revenue raising powers and financial autonomy in managing local affairs and developing the local economy (Page vii). Audit od USAID/Jordan’s Fiscal Reform Project II - Office of Inspector General Audit We recommend that USAID/Jordan: • Require DAI to prepare a sustainability plan so the Ministry of Finance will be able to continue the project management office without continued support from USAID/Jordan during any new follow-on or stand-alone project. • Adopt USAID’s standardized mission order on performance monitoring earlier that December 10, 2013; establish penalties for noncompliance with the mission order; and inform all individuals responsible for performance monitoring of their responsibilities and the repercussions for noncompliance. • Perform an evaluation or assessment that covers the remainder of FRP II’s implementation and looks at previous assistance, accomplishments, and lessons learned from other USAID/Jordan projects to provide a vision and design for any new fiscal reform or other related project after FRP II ends (Page 3). Pan Asia Asian Regional Response to Endangered Species Trafficking (ARREST) Mid-Term Performance Evaluation of Asia’s Regional Response to Endangered Species Trafficking Program – Mid-Term Evaluation ARREST is exceptionally well positioned to help address rising USG interest in and concern about wildlife trafficking both regionally and globally, including its intersection with a broad range of other high-priority issues such as transnational drug and human trafficking crimes and emerging pandemic threats resulting from the illegal transport of live animals across borders without health inspections (Page 3). Individual states cannot be expected to solve the problems of transnational wildlife crimes and threats to indigenous species without engaging the support of neighboring countries (Page 5). USAID 2017 TCB Evaluation | 99 Reducing demand for illegal wildlife is a long-term endeavor that involves changing knowledge, attitudes and practices… During the remainder of the program period, Freeland could build a foundation for the achievement of long-term outcomes by refining its demand reduction strategy to focus on Behavior Change Communication (BCC) (Page 4). Though some countries have yet to benefit from the institutionalization of the ARREST LE [Law Enforcement] training process, others, such as the Philippines, have made substantially more progress, both with respect to the functioning of their [Wildlife Enforcement Network] WENs and movement toward independently conducting their own trainings. Other WENs appear to have lost forward momentum both in their internal and external coordination efforts. This is in part due to lack of support by the respective central government and in part due to the PCU’s inability to provide more substantial support and mentoring due to understaffing and budget limitations, which has also hampered the PCU’s own progress as an ASEAN-wide coordinating entity (Page 4). Advocate for the review of the ASEAN-WEN and the PCU’s location in the overall ASEAN structure. This includes supporting the sovereign decision-making processes and protocols of ASEAN leadership in exploring options that will give the network the higher visibility and increased political and financial support it must have to become more fully effective and sustainable (Page 5). Freeland should integrate USAID’s global experience with [behavior change communication] BCC into its project design and approach to demand reduction in China and Vietnam, considering detailed recommendations outlined in Annex V (Page 5). Increase the attention and resources devoted by ARREST to partnerships in ways that promote sustainability, learning and shared ownership, thereby building the strongest possible base for USG support of Asia’s efforts to reduce and eliminate wildlife trafficking for the remaining years of the program and beyond (Page 5). Explore mechanisms for shifting management responsibility for USG support for wildlife-related law enforcement from USAID and the U.S. Department of State (DoS)/Bureau of International Narcotics and Law Enforcement (INL) to USFWS in light of the recent USG Executive Order on Combatting Wildlife Trafficking and the preparation of a whole-of-government National Strategy for Combating Wildlife Trafficking (Page 5-6). Consult with a gender specialist to identify opportunities to approach demand reduction and law enforcement capacity building in a more holistic manner that addresses the different roles of men and women in both sustaining and combating wildlife trafficking as appropriate (Page 6). Focus ARREST implementation on the ASEAN region, on the ASEAN-China relationship, on increased work in Lao PDR and on initiating activities in Myanmar. Limit support to other regional groups (e.g., South-Asia WEN [SA-WEN]) to information sharing and lessons learned (Page 6). Increase efforts to build shared local ownership and ensure that ARREST is in fact Asia’s Regional Response to Endangered Species Trafficking, not just in aspiration (Page 6). USAID 2017 TCB Evaluation | 100 Responsible Asia Forestry and Trade (RAFT) Responsible Asia Forestry and Trade: Final Program Evaluation – Final Evaluation The biodiversity of Southeast (SE) Asia is threatened by deforestation and forest degradation, driven in part by unsustainable timber harvesting. These—often illegal—practices breed corruption and undermine governance. To address this problem, USAID’s Regional Development Mission for Asia funded the Responsible Asia Forestry and Trade (RAFT) Alliance (Page 5). Programmatic lessons applicable to the responsible management of commodity chains: • Importance of a conducive regulatory environment; for timber in SE Asia regulatory signals were stronger than market signals • Importance of enforcement to send the deterrent signal necessary to compel compliance with regulations and to provide a level playing field for responsible actors • Importance of speaking industry’s language to understand their needs in order to better make the case for adopting best practices • Land clearing, in part for plantations, is the biggest driver of deforestation in SE Asia (Page 8). Managerial lessons applicable to complex regional programs: • Importance of flexibility in RAFT’s ability to respond to rapidly evolving regulations and markets and to extra-sectoral threats (like plantation agriculture). • Importance of regular monitoring and evaluation to facilitate: o Adaptive management to ensure that activities are efficiently achieving project aims o Management of partners’ activities so they do not drift from the project’s aims • Importance of communication, cooperation, coordination & collaboration to keep partners (and importantly, their activities) aligned with each other and with project aims • Compared to other initiatives, RAFT was relatively cost effective. • Importance of USAID brand; RAFT benefited from direct USG involvement: o Greater credibility with governments and industry, and goodwill for the USG itself. More direct linkage to USG policymakers and vice versa; o Greater donor cooperation, e.g., with the EU over Due Diligence Regulation outreach • USAID funding allowed increased gender equity, e.g., RECOFTC was able to train more women in conflict mediation than would otherwise have been possible (Page 8). South America MIDAS (More Investment in Sustainable Alternative Development and Areas) and ADAM (Areas for Municipal Alternative Development) Post-Implementation Evaluation Report: Evaluation of the More Investment in Sustainable Alternative Development and Areas for Municipal Alternative Development programs – Final Evaluation USAID 2017 TCB Evaluation | 101 More Investment in Sustainable Alternative Development and Areas (MIDAS)and Areas for Municipal Alternative Development (ADAM) provide technical assistance and support to organizations in Colombia to improve conditions for rural citizens. Sustainability. The evaluation found many sustained organizations, infrastructure works and productive activities, in all departments. Communities and projects were highly motivated, even in the face of adversity, to succeed and expand. Newer associations, especially with unfamiliar crops, needed more support, and rates of failure were higher (Page vii). Associativity. ADAM and MIDAS realized great gains in building associativity among some of their productive and public works activities, helping communities begin to reconstruct social fabric damaged by conflict, illicit livelihoods, poverty and the absence of the state. The common goals, training, and support were essential, yet took time to build. Notably these successes were also replicated across communities – when projects involved some combination of campesinos, indigenous groups, and Afro-Colombians (Page vii). Culture of legality. Families and communities embraced licit opportunities to make a living and avoid the social disintegration, risks to life and livelihood, and shadow markets of illicit crops. Scaling up. Established enterprises and associations provided fertile ground for productive activities that change mentalities and scope from subsistence, household levels to efficiency, marketing and a regional economy. Given the ambition and global scope of the proposed peace process, working with the GOC at all levels will be essential (Page vii). Monitoring and Evaluation (M&E). The output-level indicators by which ADAM and MIDAS are known – families supported, hectares planted, jobs created – were flawed both for telling the story of the programs and for monitoring their progress to improve implementation (Page vii). South Asia Agribusiness - Agriculture and Private Sector Competitiveness Programs (The Agribusiness Project - TAP) Building Sustainable Agribusinesses in Pakistan: Final Evaluation of The Agribusiness Project – Final Evaluation USAID/Pakistan awarded the five-year USD 89.4 million cooperative agreement for TAP to the Agribusiness Support Fund (ASF), a local organization with no previous USAID experience, in November 2011. A 2013 Office of the Inspector General (OIG) audit concluded that ASF did not have the capacity to implement the project. In consultation with USAID, ASF scaled back the project, focusing on fewer value chains and a more limited geographic area, cutting the budget to USD 39.9 million, and reducing the timeframe to four years. TAP provides grants and technical assistance to small farmers organized into farmers’ enterprise groups (FEGs) and individual grants to agri-businesses, e.g., larger farmers, processors, and exporters. Grants help recipients adopt new technologies and practices to enhance competitiveness and ultimately “support improved conditions for broad-based economic growth, enhanced profitability, and employment opportunities and contribute to poverty alleviation through product and process transformation” (Page 1). Recommendations USAID 2017 TCB Evaluation | 102 • If and when USAID/Pakistan engages local partners, it should do so slowly with relatively modest projects where the negative consequences associated with up-front learning are small. Developing the necessary capacities after the award may delay implementation and limit results. The agency might also consider placing an experienced international firm in a mentoring or capacity-building role and giving it the authority, at least initially, to manage parts of the project. • USAID/Pakistan should adapt the value of exports indicator to reflect a more meaningful measure of enhanced access to export markets. A more meaningful measure of export capacity might include trends in overall export sales, sales to new buyers or markets, or offering new products to export markets. Any of these would be more direct and feasible measures of increased access to export markets that are not as misleading as the current value of exports indicator. • If USAID/Pakistan intends to continue supporting agricultural value chain development, it should consider continuing support to at least some of the more promising TAP￾supported value chains and beneficiaries to give beneficiaries the additional technical assistance necessary to cement the lessons they have learned and address new challenges that arise as they gain experience with new technologies and practices. The most promising interventions seem to be HV/OSV and grape (FEGs and larger farmers), apricot orchards, and potato cellars. • If USAID/Pakistan wishes to continue supporting the meat sector with a focus on developing export markets, it needs to take a more holistic approach. A development process that starts by building the capacities of processors to meet a small demand for quality, value-added meat products and linking these processors to export markets may create a demand for better quality meat animals that donors can then support by investing in building feedlot capacity and improving herd quality. • To improve prospects for sustainability, USAID/Pakistan should require implementing partners to develop sustainability plans during the design phase, review the plans periodically to determine whether sustainability hypotheses are valid, and adapt implementation as necessary if the hypotheses do not hold true. Sustainability plans should be specific to each activity, i.e., in the TAP example, value chains. • When building and/or strengthening institutions is an important component of a sustainability strategy, it needs to start early in the project so the institutions are engaged, strong, and sustainable. In similar future work, USAID should ensure that building and strengthening institutions occurs early in the project work plan and follow up to make sure it happens (Page 5). Audit of USAID/Pakistan’s Agribusiness Project - Office of Inspector General Audit To strengthen the Agribusiness Project, we recommend that the mission: • Develop an action plan to make sure project goals can be reached by the end of 2016, or, if they cannot, determine whether the agreement signed with the Agribusiness Support Fund should be discontinued. • Prior to making future awards, implement mission-wide procedures to verify independently that corrective actions are sufficient to address weaknesses identified in pre-award assessments. USAID 2017 TCB Evaluation | 103 • Implement a plan to confirm that the policies and procedures of the Agribusiness Support Fund related to governance, procurement, human resources, information technology, and agribusiness operations comply with USAID regulations. • Obtain a revised annual plan from the Agribusiness Support Fund to include specific grant activities that it plans to implement during the year, with benchmarks and milestones, and expected outputs and outcomes. • Require the Agribusiness Support Fund to record the nonrefundable application fee received from the grantees as income in the project accounts (Page 2-3). Nepal Economic, Agriculture and Trade (NEAT) Nepal Economic, Agriculture, and Trade Activity Performance Evaluation USAID/NEPAL – Final Evaluation Chemonics International (Chemonics), the implementing partner for conducting the NEAT program, focused on enhancing food security; increasing agricultural production, exports, and sales; improving capacity; increasing government revenue; and improving trade and commerce in 21 districts in the East, Far West, Midwest, and Western regions of Nepal (Page iii). One main example includes the need for a longer program, which was requested specifically for the agricultural development program. People who are planning future interventions should carefully study the recommendations in this report to increase their programs’ impact. However, the NEAT program’s intervention was generally reported to be positive (Page vi). Consider splitting large, multi-faceted programs to narrow topic focus and reduce implementation complications. NEAT was a large and diverse program, which could understandably present size-related complications. It is therefore recommended that future, similar programs be divided into two separate calls for proposal. In the case of NEAT, for instance, the agricultural and food security program could have easily been split from the enabling policy and business environment component. Smaller programs would allow implementers to better focus on the specific components needed to promote success in those areas (Page v). Ensure stakeholder engagement with all key stakeholders and from the onset of programming. Respondents from the business sector shared their dissatisfaction in not being involved in decision making from the onset, stating that the programs were very government-focused. It is recommended that future programs spend time in identifying the key stakeholders from the various sectors and involving them in the decision making processes from the onset, particularly during the planning stages (Page v). Subsidies for seed and fertilizer can be helpful for farmers involved in the food-security program. However, they should be limited to farmers who will have the ability to purchase these inputs in the future (Page v). Encouraging farmers to use new hybrid seeds needs to be carefully considered. Although productive, hybrid seeds usually must be repurchased each year. For cash-strapped, geographically isolated poor farmers, hybrid seeds may not be the best solution. Improved, non￾hybrid varieties are needed, but efforts should also be made to work with open-pollinated USAID 2017 TCB Evaluation | 104 sources of seed so that farmers can be taught how to harvest and handle seeds for next year’s crop (Page v-vi). Future training programs for farmers should give a higher priority to controlling plant diseases. One of the NEAT program’s goals was to help farmers increase production through the use of better seeds, fertilizer, and improved cropping methods. Production did increase for all crops, but with this increased production came increased disease pressure, for the two are often linked. Plant-disease diagnosis and treatment should be included with future agricultural development programs (Page vi). To improve the sustainability of agricultural development projects, there should be a greater inclusion of all key stakeholders, particularly local district agricultural development officers (Page vi). Access to finance initiatives within similar programs should be included, as these activities empower the farmers to participate in other program interventions, such as training, capacity building, and purchasing of appropriate seeds, among others. Both the banks that received training and the beneficiaries who were able to access financial products ranked this component very highly. Ultimately, providing opportunities for financial independence to the NEAT beneficiaries would promote sustainability of the program’s interventions (Page vi). Pakistan Trade Project Pakistan Trade Project Midterm Performance Evaluation Report – Mid-Term Evaluation The United States Agency for International Development’s (USAID’s) Pakistan Trade Project (PTP) is a countrywide project designed to promote initiatives that help businesses trade across borders and improve the trading environment in the country (Page iv). A detailed review of the background material generated by PTP before activities initiated as well as minutes of meetings with stakeholders during implementation clearly establishes the relevance of PTP activities to improving the trade environment in the country (Component 1) and increasing cross-border trade (Component 2). Interviews with stakeholders also supported the relevance of many of PTP’s completed and ongoing activities under Component 1 and all activities under Component 2 (Page 3). USAID should ensure that PTP continues to make all efforts possible to institutionalize a sustainable and workable trade portal within an indigenous Pakistani public or private sector entity such as TDAP or an appropriate alternative. USAID should ask PTP to consult with the Ministry of Commerce to redesign or refocus the research studies on Pakistan–India trade to respond more directly and more comprehensively to the Ministry’s interest and needs (Page 4). Under Component 2, USAID, through PTP and its successor project, could consult relevant stakeholders to identify any assistance USAID can provide, including activities already planned by PTP, to facilitate realization of the Single Window and Risk Management System in the next two to three years (Page 5). There is evidence that PTP has made small, but significant progress in supporting internships and employment for women under the Management and Mentorship Program. In other ways, USAID 2017 TCB Evaluation | 105 however, PTP’s activities have had limited engagement with women and limited influence on the participation of women in trade activities. One reason for this is that the Women in Trade (WIT) Portal is not operational. Secondly, there is, as yet, no evidence (except for two case studies by PTP) that training women exporters in export processes, rules, and regulations has influenced their engagement in trade activities. However, this training is reported to be a useful tool for knowledge sharing (Page 4). Poverty Reduction by Increasing the Competitiveness of Enterprises (PRICE) Poverty Reduction by Increasing the Competitiveness of Enterprises (PRICE) Final Performance Evaluation – Final Evaluation [PRICE’s] main mission was to reduce poverty sustainably by increasing enterprise competitiveness across three main sectors in Bangladesh: horticulture, aquaculture, and leather (Page vi). Given the low domestic demand for horticulture products, USAID should emphasize linking horticulture producers with international exporting processors (Page viii). To aid small-scale horticulture and aquaculture farmers, future USAID projects should help banks or microfinance institutions ensure that loan products have grace periods long enough to cover at least one crop cycle (Page viii). Agricultural loans should also facilitate sustainable credit practices through support that identifies and develops business models to provide adequate training and knowledge among beneficiaries. In addition, an agriculture voucher system could be incorporated into loans, as the evaluation team discovered that a significant portion of loans is spent on household expenses rather than agricultural activities (Page ix). BMO [Business Membership Organization] partnerships should continue to receive support to bring project benefits into geographic locations where non-project beneficiaries reside. Nodal organization partnerships are also recommended to establish networks from the national to village level, including local technical specialists (Page ix). Group production is a sustainable option to increase production levels, which qualify producers for linkage with national or regional market intermediaries. Group production ensures that farmers harvest a sufficient volume of products and encourages large marketing intermediaries to directly collect from the farmers (Page ix). New USAID projects could identify whether the number of farmers with requisite farming knowledge has increased in the south, follow the contract-farming model implemented in the north, and replicate in southern districts the direct link between farmers and buyers (Page ix). Established organizations, such as Murail Rural Development Multipurpose Co-operative Society Ltd. are able to promote continued implementation of PRICE project practices following project completion. Through continued partnerships, local horticulture project beneficiaries will continue mutually beneficial arrangements even when they do not have direct access to project technical assistance (Page ix). USAID 2017 TCB Evaluation | 106 USAID needs to examine how to apply the COEL job creation model in other sectors, as COEL contributed to women’s empowerment and strengthened job creation for women. COEL should also expand to regional centers in Chittagong, Rajshahi, and Khulna (Page ix). In the leather sector, future projects should assist large manufacturers in finding price￾competitive international markets. Domestic sales of leather products can also be improved through continued assistance with organizing leather product fairs and innovative marketing programs (Page ix). Southeast Asia Agribusiness Market and Support Activity (AMARTA) The Agribusiness and Market Support Activity (AMARTA): Final Evaluation – Final Evaluation The objective of the AMARTA project was to improve the quality and quantity of commodities produced within selected value chains and establish strong and transparent linkage between producers and other stakeholders, resulting in enhanced knowledge, greater access to appropriate technology, improved access to markets and improved incomes (Page 1). AMARTA worked with a number of private sector partners/exporters to introduce quality standards for cocoa. These quality standards included threshold levels for moisture content, foreign matter and bean size. This activity is a model that can be used in developing an industry wide quality standard and pricing system (Page 2). AMARTA’s approach to high-value horticultural value chains was sound and should be continued. More emphasis should be placed on post-harvest handling, marketing and processing, beginning with the supply chains already developed. More emphasis should be placed on continuous economic evaluation/analysis of selected value chains focusing on marginal increases of income, employment and competiveness relative to USAID’s investment for each value chain (Page 4). More emphasis should be given to increasing productivity in West Java because farmers own and operate small plots (0.3 HA). In North Sumatra, more investment in cost-efficient cold chains and transport is needed to reduce field and transport loss and to improve shelf life for commodities. More development of branding was requested by buyers and retail outlets. Development and/or use of commodity prices, grades, and cultural practices are needed to improve price discovery. Grading for price is not widely practiced throughout the chains even though this information is generally known (Page 4). More assistance to improve horticultural supply chain efficiency should be provided to exporters, traditional markets and retail supermarkets. Buyers of farm produce seem willing to invest in post-harvest handling operations. Future AMARTA activities should consider ways to support private sector investment in cold chains, packing lines and transport systems. Future AMARTA activities should place emphasis on increasing the number of partnerships and to secure cost-sharing investment in post-harvest handling (Page 4). Partnerships and even communication with agencies in the central government were difficult to develop in part due to GOI policies since the early 2000s to decentralize responsibilities to USAID 2017 TCB Evaluation | 107 provincial and local government authorities. Stronger efforts to develop central government interaction are needed (Page 4). At the farm level, future activity needs to focus on continued training of farmers, and expanded training into new geographies… Future post-harvest activities should include training in grading and sorting of products (for coffee, cocoa and horticulture), as well as value-addition techniques such as cocoa fermentation, fully-washed coffee/coffee fermentation, as well as fruit and vegetable grading, packing, cooling and labeling… Future activities should work with partners to create a traceability system that maintains product identity throughout the supply chain. Developing a quality grade bonus system that is broadly accepted throughout the industry is another important focus area that should be included in any future activity (Page 5). In terms of institutional recommendations, future activities need to form better relationships with national-level government. Additionally, emphasis needs to be put on training RACAs [Regional Agribusiness Competitive Alliance] in advocacy to allow them to become effective change agents with the ability to influence policy and regulatory outcomes (Page 6). Micro, Small and Medium Enterprise 2 / Business Enabling Environment Final Performance Evaluation of USAID/Cambodia Micro, Small and Medium Size Enterprises II/Business Enabling Environment Project - Final Evaluation The Cambodia Micro, Small and Medium Size Enterprises II/Business Enabling Environment Project was implemented in Cambodia with the goal of improving the business enabling environment and increasing the productivity of help micro, small, and medium enterprises (MSMEs) in rural areas. • The model of minimal intervention should be used in similar future projects. • Further work in some value chains. MSME was not necessarily able to bring the skills of all clients in all areas and sectors up to the same level. Some assisted sectors (e.g. honey, latrines, fish) appear to be rapidly moving toward self-sustaining growth, while others (eco-tourism, resin collection) could profit from further training in management and financial skills, for example. Still others (e.g. swine, water and fish) need to address issues in the sectors - seasonality, illegal imports, environmental protection, and government support - that may affect their long-term growth. • Further institutionalize advocacy organizations. While good progress was made in creating local and provincial business associations that may have a long-term influence on policy development, these organizations may be limited in their reach and capacities, particularly in technical policy development and legislative skills at the national level. Further work can be done in taking the local organizations to the next level by assisting in the creation of a sectoral business membership organization of national scope that may be able, through economies of scale and membership support, to retain technical skills and take a larger role on the national stage. • Further institutionalize advocacy training. These could include further development and dissemination of the training materials produced by MSME. Efforts could be made to induce certain NGOs to adopt advocacy training as part of their usual offerings. Work could be done with local educational institutions to develop courses on USAID 2017 TCB Evaluation | 108 communications and advocacy for the private sector, including courses on policy analysis and legislative process. • Seek to institutionalize opportunities for public-private dialogs. • Perseverance in the legislative arena may ultimately pay off and ways should be found to continue support for the legislative initiatives promoted by MSME. Efforts can be made to determine which ongoing projects of USAID and other development partners will pick up the tasks of monitoring and contributing to the development of the key legislative acts that remain on the table. Too much time and effort has been invested in those to allow them to go without champions. • Further institutionalize opportunities for training in policy and legislative development (Pages 5 to 6). Support for Trade Acceleration (STAR) Program and Support for Trade Acceleration Plus Project (START Plus) Performance Evaluation of the USAID/Vietnam Support for Trade Acceleration (STAR) Project – Final Evaluation The acronym STAR refers to Support for Trade Acceleration: a series of technical assistance projects designed, funded and implemented by USAID/Vietnam through institutional contracts between 2001 and 2010 (Page 4). USAID/STAR pushed the envelope of what technical assistance can accomplish. The STAR programs delivered value as well as results. Credits for the legal reform program go to the Government of Vietnam, to USAID, to the USG stakeholders, to STAR DAI management and the STAR Vietnamese staff, and team of consultants that raised the performance bar for technical assistance. Taking time and resources into consideration, it is important to differentiate between the new laws and procedures put in place by the GVN and facilitated by STAR technical assistance and from the translation of the new legal regime into everyday practice. STAR I’s effectiveness and reach were about equal. That is, STAR I supported the GVN in carrying out a large array of changes in the body of commercial law and in the workings of legal system. STAR II expanded in the direction of implementation by helping the GVN to increase dissemination of new laws and regulations while continuing to facilitate the GVN legal reform program to meet BTA and WTO requirements. STAR II did not and could not reach deeply and broadly to support the GVN’s administration of the new statutes and procedures nor make a large impact on day-today operations of the private sector. Implementation is a national responsibility; technical assistance can only support the national effort. Application and enforcement of legal statutes, such as IPR rules or a well-functioning judiciary and court system, continue to be an objective rather than an accomplishment. The success of the program should not become an anchor to the past. With STAR+ support: • There is an opportunity for the GVN under the new project to examine how Vietnam’s economic transformation can be deepened. There is a significant need for institutional capacity building for policy analysis and policy formulation; and • There is an opportunity for the GVN to devise ways to convert and quicken the flow of potential benefits from the legal reforms into direct, tangible benefits for the Vietnamese private sector (Page 35). USAID 2017 TCB Evaluation | 109 Audit of USAID/Vietnam’s Support for Trade Acceleration Plus Project - Office of Inspector General Audit The audit determined that while the project has made some progress, it was difficult to determine the extent of any progress because of performance monitoring problems (Page 2). The report recommends that USAID/Vietnam: • Establish custom indicators that would capture critical activities adequately and allow the mission to track the project’s progress. • Work with the contractor to review and revise its performance targets. • Perform a data quality assessment identifying the contractor’s source of reported results, analysis of calculations, and completeness of data. • Through the contracting officer, determine the allowability and recover, as appropriate, $58,524 in unsupported costs pertaining to the total approved days that a short-term technical adviser worked. • Require the contractor to quantify, track, and report on all forms of cash and in-kind contributions contributed by counterparts and other organizations toward STAR Plus activities (Page 2). Southern Africa Integrating Nutrition in Value Chains (INVC) The Feed the Future Integrating Nutrition in Value Chains Project Performance Evaluation– Final Evaluation The goal of INVC is to sustainably reduce rural poverty and improve nutrition through the integration of agriculture and nutrition interventions (Page iii). Farmers apply new technologies and practices at different times. The first to adopt are innovators or early adopters. Women farmers in Malawi are perceived as innovators and early adopters, a characteristic acknowledged as a cultural distinction between sexes. Women predominantly grow groundnuts and soybeans to feed their families and increase household income. They are viewed as patient and willing to wait for positive results. In contrast, men want to see results prior to adoption. The Consortium and sub-partners should emphasize reaching out to women farmers as early adopters, while men should receive training and reinforced training to influence them (Page iv). Seed Distribution: Distribution should continue each season throughout the life of the project. However, farmers should be restricted to receiving seeds for only two years, which would allow them enough time to appreciate the benefits of these new varieties. Distribution of seed should be an incentive, not a subsidy. After the first distribution, AEDOs and AFOs should link farmers with certified seeds suppliers who carry these improved varieties so that by the second year farmers have a relationship with a supplier from which they can purchase seeds in the third year (Page v). Across all districts, farmers reported they could not afford herbicides and pesticides even if they knew what product to buy; therefore, INVC should emphasize other pest/weed management practices (Page v). USAID 2017 TCB Evaluation | 110 There are many obstacles to measuring and estimating productivity and/or profitability of groundnut and soybean enterprises in Malawi. The country does not have a culture of systematically measuring/estimating crop performance per unit of area or labor force used. This severely restricts informed decision-making and the management of a crop portfolio (Page vi). The Consortium should be able to tailor interventions to each partner, and should invest more time in change management, knowledge management, stakeholder involvement, and developing new opportunities. These are new concepts—ones sub-partners do not yet appreciate. The Consortium and the sub-partners should use the capacity development plans as the terms of reference for both parties. Delivering capacity development through provision of staff, training, coaching, and mentoring creates synergy, but this should be well-linked and coordinated. This can be achieved by establishing practices and systems for reflection and learning (Page ix). South African International Business Linkages II (SAIBL II) Evaluation of the South African International Business Linkages Program – Final Evaluation The specific objective of the SAIBL2 program is to improve the competitiveness and capacity of black businesses to integrate them into domestic and international markets through increased productivity, improvements in corporate governance, and the development and adoption of modern technologies and marketing practices. (Page 2). One can try to help individual SMEs, but to scale up the process you need structural transformation like the SASDC (Page 4). Ensure the proportionality of the analysis to the business problems being addressed. If the corporations that are members of the SASDC are not requiring a level of analysis beyond something comparable to the Picking a Sound Company (PSC) tool, it is doubtful something more rigorous for business development purposes is necessary (Page 4). It would be appropriate to continue a selected “pilot” approach, such as assistance to incubators, as an experimental effort to help USAID identify alternative approaches to assist black SMEs following SAIBL2 (Page 4). USAID should review together with the FSP [Financial Sector Program] and other implementing partners how they can coordinate their respective efforts to produce a mutually acceptable BSP [Business Service Provider] rating system (Page 6). Southern Africa Trade Hub (SATH) Mid-Term Evaluation of the Southern Africa Trade Hub – Mid-Term Evaluation The United States Agency for International Development (USAID) Southern Africa Trade and Competitiveness (the Trade Hub) project, implemented by AECOM, commenced in September 2010. The Trade Hub’s overarching goal was originally “Increased International Competitiveness, IntraRegional Trade, and Food Security in the Southern African Development Community (SADC) Region.” This objective was to be accomplished through the advancement USAID 2017 TCB Evaluation | 111 of the regional integration agenda and increased trade capacity of regional value chains in selected sectors (Page iv). Greater Competitiveness in Agricultural Value Chains - After a slow start, this activity gained momentum during the past 18 months, and shows promise of achieving increases in production and trade of the selected commodities. Important progress has been achieved in establishing storage capacity for grains, training, particularly for aflatoxin removal, and in stimulating private sector investment by value-chain partners (Page v). Increased Trade and Investment in the Textiles and Apparel Sector - The SATH countries increased exports of these products to the United States under the African Growth and Opportunity Act (AGOA). Source Africa, the flagship activity, made progress during its second year of operation, with the number of exhibitors increasing from 170 last year (2013) to more than 220 this year (Page vi). Strengthened Regulatory Capacity for the Clean Energy Sector This activity has introduced the concept of clean energy to the region with considerable success. Energy regulators have begun, with training from SATH, to analyze methods to incorporate production of clean energy into their systems, including tariff rates and other concerns. Current work in Swaziland shows promise for reforming energy policy there. The Regional Energy Regulatory Association (RERA) values the work of the Hub highly. This work should be continued through the remainder of the current project, but should be spun off into a different vehicle, given its limited relationship to the rest of the work of the Hub. That subsequent effort might be linked to Power Africa, and might be expanded to include emissions reductions (Page vi) A Better Enabling Environment With collaboration from the governments of Swaziland, Lesotho and Namibia and with varying degrees of success (as noted in the full report), the project provided investor roadmaps for these countries. It also provided a document to the Southern Africa Customs Union (SACU) on implementation of the Trade and Investment Development Cooperation Agreement (TIDCA) with the United States. We recommend that the work of the Enabling Environment IR be taken up within the other IRs for the remainder of the project, with the work on standards migrating to Agriculture Value-Chains (AVC), Trade Enabling Environment (EE) work to Trade Facilitation (TF), and Investment Promotion to Textiles and Apparel. Any new project should include a more robust activity in the enabling environment area, focusing on the concerns of promoting policy actions that improve the prospects for trade creation, specific to the work the project intends to carry out (Page vi). Africa Trade Hubs Export Promotion Evaluation – Evaluation Report The United States Agency for International Development (USAID) contracted with Social Impact, Inc. (SI) to conduct an evaluation to assess the effectiveness of firm-level export promotion activities of the Africa Trade Hubs Projects, which had been in operation for more than a decade (Page 1). • USAID should require future Trade Hub implementing partners to submit a detailed PMP supported by a clear development hypothesis and results framework at the project’s outset that includes outcome-level indicators in addition to input- and output￾USAID 2017 TCB Evaluation | 112 level indicators and also includes indicators aimed at measuring whether or not theory￾of change assumptions hold true. • USAID should require the Trade Hubs to monitor both surviving and failing firms for results reporting purposes. This should include follow up with site visits and qualitative surveys to both types of firms. • Given the multitude of potential methodological biases and restricted sample size, any future IE of Trade Hub programming should be designed alongside the project. • Collection of longitudinal data on client exports and employment should be required of Trade Hubs’ clients so that cost-effectiveness of support to different value chains can be documented. • USAID should maintain a centralized database of all Trade Hub reports, including RFPs, work plans, annual and quarterly reports and PMP data. • In line with its policy of encouraging “collaborating, learning and adapting (CLA),” USAID should periodically assess whether the value chains that the Trade Hubs are supporting actually are those with the greatest potential to generate exports and employment. • USAID should consider requiring the Trade Hubs to continue market-linkage work with firms either directly or through associations. • USAID should consider requiring the Trade Hubs to continue market-linkage work with firms either directly or through associations. • USAID should consider requiring the Trade Hubs to expand efforts to reduce firm constraints to exporting by facilitating firm level access to finance and reducing regulatory barriers to exports (Pages 58-59). Sub-Saharan Africa AGCI-Knowledge Sharing and Analysis Project - KSA African Global Competitiveness Initiative AGCI End of Initiative Report, 2006-2011 – Self Reflective Document Launched in 2006, the Africa Global Competitiveness Initiative (AGCI) was a $200-million, five￾year Presidential Initiative that supported efforts between 2006 and 2010 to: • Improve the business and regulatory environment for private sector-led trade and investment in sub-Saharan Africa; • Strengthen the knowledge and skills of African private sector enterprises to help them take advantage of market opportunities; • Increase access to financial services for trade and investment; and • Facilitate investments in infrastructure (Page 1). Through AGCI, USAID made important contributions to realizing short- and long-term progress in improving the enabling environment for trade and investment, boosting firm￾level skills and capacity, increasing access to finance, and overcoming infrastructure constraints. Projects have had an impact on the countries eligible to trade with the United States under AGOA. Over the life of AGCI, significant progress was made in reaching the four life of initiative macro performance targets, as outlined below by component (Page 2). USAID 2017 TCB Evaluation | 113 USAID’s partnerships with a wide range of African institutions and individual entrepreneurs over the last five years have made AGCI a success. The Agency’s focus on identifying and removing the barriers to freer trade—be they at the policy level or imposed by a lack of firm-level expertise or knowledge—has contributed to Africa’s steady economic growth and falling rates of poverty (Page 19). With its conclusion, AGCI leaves behind a legacy of accomplishment and partnerships with African institutions, companies, and individual entrepreneurs. The evidence for USAID’s success under AGCI can be seen in the fact that not only were the initiative’s targets met or exceeded, but that several activities initiated under AGCI will continue based on demand from the Agency’s partners in Africa. Under AGCI, USAID’s three Trade Hubs have become institutionalized sources of demand-driven technical assistance for companies, host governments, and regional institutions, including major African RECs. In particular, under AGCI USAID has cemented its reputation for being able to deliver flexible and high-quality services to its African partners that advance regional integration on the continent. AGCI has helped multiple countries and regional institutions design and implement market-friendly economic policies and procedures aimed at spurring private sector-led growth. Thousands of individual companies and entrepreneurs benefited from training and services under AGCI that have helped them become export competitive worldwide, leading to the creation of thousands of jobs and increased incomes (Page 19). West Africa Energy Sector Policy and Electricity Provision (LEAP) Audit of USAID/Liberia’s Energy Program Activities - Office of Inspector General Audit LEAP was designed to increase access to affordable energy supplies to foster economic, social, and political development (Page 1). LEAP implemented several viable pilot projects as potential solutions to the energy shortcomings in Liberia, including the use of solar energy and prepaid metering to reduce electricity theft. In addition, LEAP contributed significantly to the adoption of the Liberian National Energy Policy and other pieces of legislation, as well as to the electrical transmission and distribution equipment used in conjunction with EPP to further electrification of Monrovia (Pages 1-2). Recommendation 1. We recommend that USAID/Liberia develop and implement a policy requiring all contracting and agreement officer’s technical representatives to conduct and document site visits and, when feasible, verify reported indicator data during these site visits (Page 4). Recommendation 4. We recommend that USAID/Liberia establish policies to ensure compliance with environmental requirements for its entire portfolio and that these requirements be communicated to its staff and implementing partners (Page 6). Integrated Initiatives for Economic Growth in Mali (IICEM) Integrated Initiatives for Economic Growth in Mali (IICEM) Performance Evaluation – Final Evaluation USAID 2017 TCB Evaluation | 114 IICEM (Integrated Initiatives for Economic Growth in Mali) was designed in 2009 to confront the problem of poverty (Page 1). Promoting gender-equitable opportunities in value chain development. IICEM’s assistance to and support for women’s farming organizations has helped many women to exercise real entrepreneurial and managerial skills. The women’s bas-fond rice OPs are viewed by both banks and suppliers as viable rural enterprises. These OPs are increasingly weaving their own value chain networks and diminishing their reliance on support from projects and/or government (Page 39). Improving production and productivity. The use of and working through breakeven analysis with the farmers so that they understand the cost of inputs and the labor that they put into production is perhaps the best practice for business attitude development in preparation for negotiating an appropriate price for crops. This is a crucial first step when going from selling surplus subsistence crops to systematically producing for a market. However, we did note that land value and the managerial or entrepreneurial factors of production were not included in the analysis. Nevertheless, this part of business training is often assumed and yet not part of the culture (Page 39). Building capacity in value chain development. Support given for small and medium scale processing by IICEM adds value to production as well as developing local enterprises. The small rice mills in Mopti and the parboiling of rice by women bas-fond farmers are the best examples of this in the IICEM program. At the same time, IICEM investments in the rehabilitation of Irrigated Village Perimeters (PIV) and flood plains (bas-fond) are increasing productive capacity to meet these new demands (Page 39). Marketing and transport facilitation. Building relationships between farmers and grain traders has created the incentives to produce more for market. The contracting and grouped selling has tapped economies of scale that aid traders and transporters to expand their market purview and better address the logistics of the value chain (Page 39). Enhanced financial services. The BNDA has shown a willingness to provide the production credits needed for increasing cereals marketing. The next challenge in this process will be to build OP capacity to a point where the bank can have confidence enough to continue to risk financing after the withdrawal of the facilitating monitoring umbrella currently provided by IICEM and its sub-contractors (Page 39). Supporting an enabling environment for agricultural trade and private sector development. IICEMs strategy of integrating private traders in to the value chain has been a crucial link. Besides helping to organize the market, IICEM has provided services directly to companies. Business Development Services (BDS) and training have helped them to improve management and access new information. The Special Activities Fund and Innovation Fund are providing incentives for these businesses to upgrade their infrastructures and equipment, especially in the area of quality control (Page 39). The development of an appreciation for quality seed. In a mostly subsistence cropping system where seed is simply grain saved from the previous year, such as millet, quality seed is a key element so that productivity and quality would increase so that famers would increase their own food security as well as incomes also led to an important value added for farm businesses and groups producing seed instead of simply grain for feed or human consumption. This focus USAID 2017 TCB Evaluation | 115 effectively created or at least strengthened a new link in the millet and sorghum value chains – the commercial seed industry (Page 39). Nigeria Expanded Trade and Transportation (NEXTT) Mid-Term Performance Evaluation of The Nigeria Expanded Trade and Transport (NEXXT) Project – Mid-Term Evaluation. NEXTT’s principle objective is to improve Nigeria’s overall trade and transport competitiveness, through addressing Nigeria’s trade and transport challenges along the Lagos-Kano-Jibiya (LAKAJI) Corridor; improving and facilitating trade policy; and expanding exports (Page 1). Trade Policy appears to be the most successful component of NEXTT. Four out of six indicators had values high above targets. The adoption of Food Safety Policy approved by the Federal Executive Council was a key outcome. Lakaji Corridor Improvement was the least successful component (Page 1). Conceptually the NEXTT design is relevant, however, … A key flaw in the design is the lack of financial, institutional, and human capacity by the Government of Nigeria to successfully improve trade and investment and to develop the LAKAJI corridor. In addition the business environment in Nigeria is extremely poor, as ranked by the World Bank. USAID and other donors are funding trade capacity programs to address the institutional and human capacity constraints, but there is no formal coordination process. No donor appears to be comprehensively addressing the poor business environment, which is a key constraint to meeting the objectives of NEXTT (Page 1-2). USAID should explore addressing business environment. It is essential that the Mission work through the appropriate existing GON policy setting mechanisms and establish inter-ministerial committee, under the Ministry of Finance, to coordinate and support trade policy, infrastructure development, and private sector investment… Work should be through existing Nigerian entities and mechanisms (Page 3). Regarding the Lakaji Corridor, USAID first assess USAID/Washington’s appetite for the long￾term commitment of resources required to develop the corridor, at the same time it determines the commitment of the Nigeria’s government – Ministry of Finance, National Planning Commission, Office of the President and the private sector. A first step would be to assist the GON and the private sector to develop a strategic plan for corridor development. Support should also be given to develop an effective partnership between federal and state governments and the private sector (Page 3). USAID West Africa Trade Hub Evaluation of West Africa Trade-Related Projects – Final Evaluation This report presents the results of an evaluation of the performance of the West Africa Trade Hub (the Trade Hub), which focuses on export-ready companies, and two regional trade projects the Agribusiness and Trade Promotion Project (ATP) and the Extended-ATP (E-ATP), USAID 2017 TCB Evaluation | 116 which focus on promoting regional trade within West Africa for value chains including grains, cattle, poultry, and onions (Page 1). In sum, Trade Hub activities are working well, and there is substantial scope for continuing these programs, building capacity, and helping export-ready companies to access American and European markets (Page 3). Continue to support regional programs that are needed to help make the marketing system more efficient and bring some new players into the market information system (Page 3). Work to facilitate transport should be continued by the Trade Hub, expanding beyond sealed transit cargoes to include activities with truckers of other commodities such as bulk grains, previously covered by E/ATP (Page 4). Strengthen industry advocacy (Page 4). Formal computerized systems to book backhaul freight should be in introduced in major commercial centers throughout the region and a linkage to the USAID-supported MIS [Market Information System] network should be considered (Page 4). Truck certification and assistance to promote industry certification of conformity (weight, documents) at point of origin for trucks carrying grain and livestock and other bulk products should also be considered (Page 4). Future projects should consider improving the RESIMAO price collection system currently sponsored by 10 West African governments and review country by country whether it continues to be desirable to support a parallel for-profit network for gathering and publishing the basic market price information or whether integration of efforts is feasible (Page 5). Consideration also should be given to providing support to the private system because it can have the advantage of “real time” prices–without a week’s delay and can tailor information to client needs. Synergies could be found between the public and private system (Page 5). Weekly radio announcements of wholesale and retail process in major markets in local languages should be encouraged. Also, consideration should be given to the continuation and expansion of Internet programs to announce sales (preferably of truckload quantities) (Page 5). Helping companies to become bankable, make sensible decisions on credit based cash flow expectations, and achieve new and cheaper access to finance should remain a preoccupation of USAID regional projects (Page 6). Continue programs of the Trade Hub and E/ATP that help companies develop business planning and bankable proposals (Page 6). Educating bankers about opportunities and seeking new sources of financing are helpful and should be continued preferably with some non-USG money at risk (Page 6). USAID should support sustainable commodity associations that will take a role in promoting policies that provide an attractive enabling environment for their value chain (Page 6). Educating officials on sensible economic policies should be a continual process supported by donors and by local industry associations (Page 6). USAID 2017 TCB Evaluation | 117 Coordination among donors on programs and policy messages should be actively pursued (Page 6-7). Regional activities should be coordinated with other donors and focus on key policies (Page 7). The regional project should continue the work of a regional policy expert and dedicate one session of each regional workshop to examine national and regional status of key policies, and agree on an action plan to address important issues at both the national and regional levels, and as appropriate include important stakeholders in countries without USAID bilateral value chain programs (Page 7). Africa Trade Hubs Export Promotion Evaluation – Evaluation Report The United States Agency for International Development (USAID) contracted with Social Impact, Inc. (SI) to conduct an evaluation to assess the effectiveness of firm-level export promotion activities of the Africa Trade Hubs Projects, which had been in operation for more than a decade (Page 1). USAID should consider requiring the Trade Hubs to continue market-linkage work with firms either directly or through associations. Quantitative evidence from both the WATH and ECATH shows that firms report that this type of assistance has had the most impact on increasing exports. Qualitative interviews demonstrated that the best method for assisting firms in this regard is through a step-down process, meaning firms are supported more heavily (financially and technically) when they first begin working with the Trade Hub (or association) and then, slowly, support is lessened until firms are supporting themselves entirely (Page 1). Trends in West African exports to the United States in sectors that have received support from the West Africa Trade Hub are shown in Figure 4.23 Nontraditional export sectors (i.e., non￾petroleum and non-mineral) increased from $838 million in 2001 to $2.7 billion in 2011. Noteworthy in this respect is that the apparel and textiles sector has played a relatively insignificant role in exports to the United States, whereas food and food ingredients, as well as manufactured and other goods (including handicrafts) have all seen substantial increases during the past decade (Page 26). In addition to asking firms questions about key Trade Hub project outcomes, the WATH survey also asked firm level respondents about their perceptions of major constraints to export growth and types of assistance received (from the WATH and other donors). Interestingly, the most commonly cited challenges to export growth were economic shocks and lack of access to investment and trade financing. In addition, when asked what type of assistance received from outside entities has most impacted firm sales and exports in recent years, more than 50 percent of firms reported that trade shows and business-to-business forums and financing had the most impact (Page 35). USAID 2017 TCB Evaluation | 118 Annex G. List of Key Interviewees Name Affiliation Cory O’hara West Africa Regional Mission Steve Orr Regional Economic Integration Office, Kenya and East Africa Carol Adoum West Africa Trade and Investment Hub Kofi Boakye Bureau for Asia Patterson Brown Bureau for Food Security Christian Gomez + Doug Pulse Bureau for Latin America and the Caribbean David Cowes Bureau for Europe and Eurasia Lee Williams Bureau for Europe and Eurasia Bama Athreya Bureau for Democracy, Conflict and Humanitarian Assistance David Marsden Office of Afghanistan and Pakistan Affairs Lori Rakoczy Bureau for Asia Stephen Little Bureau for Europe and Eurasia Dany Khy Bureau for Asia Elizabeth Santucci Bureau for Economic Growth, Education and Environment Paul Fekete Bureau for Economic Growth, Education and Environment Carrie Gray Bureau for Africa Erhland Herfindahl United States Trade Representative William Baldridge Bureau for the Middle East Bhavani Patha Bureau for Economic Growth, Education and Environment USAID 2017 TCB Evaluation | 119 Annex H. TCB Results Framework: Success Scores RF 1.1 Trade Performance/ Foreign Investment Improved in TCB Target Countries RF 2.3 More Efficient/Cost￾Effective Movement of Traded Goods across Borders (score=2.35) RF 2.2 Trade-Related Public Sector Practices Improved (score=2.39) RF 0.0 Rapid, Sustained, Broad-based Economic Growth in TCB target Countries RF 2.1 Export/Import Practices Improved (Enhancing Economic Responsiveness) (score=2.46) RF 2.3.1.1 Enhanced Skills/Awareness of trade facilitation personnel (score=2.4) RF 2.3.1.2 Expanded ICT Use/ Capacity (score=2.5) RF 2.1.1 Improved International Market Opportunities Knowledge and/or Linkages (score=2.3) RF 2.1.2 Improved Trade Requirements/ Standards and Knowledge (score=2.5) RF 2.1.3 Improved Business Practices (sectors, firms, industries) (score=2.5) RF 2.1.4 Expansion/Diversif￾cation of Exportable Value Chains (Ag/Svs/ Manufacturing) (score=2.3) RF 2.2.1 Trade-related/ Business￾friendly Policies Issued or Adopted (score=2.2) RF 2.2.2 Trade Agreements Signed and/or Implemented (score=2.3) RF 2.2.3 Public Financial Management (for domestic revenue generation) (score=2.2) RF 2.3.1 Improved Prof. Services for Exporters (FF, finance, transport) (score=2.5) RF 2.3.2 Customs/Border Procedures Modernized (score=2.3) RF 2.3.3 Improved Trade-related Infrastructure (resource mobilization, implementation, feasibility studies) (score=2.2) RF 2.1.1.1 Improved Export/FDI Promotion, Business Support Services (score=2.6) RF 2.1.1.2 Improved Capacity/ Use of ICT/Other Technology (sectors/ firms, industries) (score=2.4) RF 2.2.1.1 Strengthened Inst. Capacity Devel. for Policy Formulation/ Implem./Enforcement (score=2.5) RF 2.2.1.2 Enhanced awareness/ support for TAs and related business/trade policies (score=2.7) USAID 2017 TCB Evaluation | 120 Bibliography Abt Associates. (Undated). USAID AgriFUTURO. (2012). USAID AgriFUTURO Project. End of project report. 1 May 2009-28 February 2015. Abt Associates. (2014). USAID AgriFUTURO Project. Annual performance monitoring report. 1 October 2013-30 September 2014. October. Abt Associates, Inc. (2014). Integrated Initiatives for Economic Growth in Mali (IICEM). Final report. Itiatives Intégrées pour la Croissance Économique au Mali (IICEM) IQC# EDH-I-13-05-00005 for the reporting period: January 2010-February 2014. February. ACDI/VOCA. (Undated). Agricultural Growth Program – Agribusiness and Market Development (AMDE). Quarterly report October 1, 2011-December 31, 2011. January 31. ACDI/VOCA. (Undated). Agricultural Growth Program – Agribusiness and Market Development (AGP￾AMDe). Quarterly report October –December, ACDI/VOCA. (2011). Agricultural Development and Value Chain Enhancement Feed the Future Activity (ADVANCE). Second annual report. October 1, 2010-September 30 2011. October 30. ACDI/VOCA. (2010). Ghana Agricultural Development and Value Chain Enhancement Project (ADVANCE). Annual report, year 1. October 1, 2009-September 30, 2010. October. ACDI/VOCA. (2011). Agricultural Development and Value Chain Enhancement Project (ADVANCE). Second annual report. October 1, 2011-September 30, 2012. October 30. ACDI/VOCA. (2012). Ghana Agricultural Development and Value Chain Enhancement Feed the Future Project (ADVANCE). A USAID Feed the Future Initiative. Third annual report. October 1, 2011- September 30, 2012. October 30. ACDI/VOCA. (2013). Ghana Agricultural Development and Value Chain Enhancement Project (ADVANCE). A USAID Feed the Future Initiative. Fourth annual report. October 1, 2012-September 30 2013. October 30. ACDI/VOCA. (2014). Agricultural Development and Value Chain Enhancement Feed the Future Activity (ADVANCE). A USAID Feed the Future Initiative. Final report. June. Advanced Engineering Associates International, Inc. (AEAI). (Undated). Enterprise Energy Efficiency (E3). IQC contract # EPP-I-00-03-00004-00. Task order #12. Annual progress report. May 14, 2010-May 14, 2011. Advanced Engineering Associates International, Inc. (AEAI). (Undated). Enterprise Energy Efficiency (E3). IQC contract # EPP-I-00-03-00004-00. Task order #12. Annual progress report. May 4, 2011-May 4, 2012. USAID 2017 TCB Evaluation | 121 Advanced Engineering Associates International, Inc. (AEAI). (Undated). Enterprise Energy Efficiency (E3). IQC contract # EPP-I-00-03-00004-00. Task order #12. Annual progress report. May 4, 2012-May 4, 2013. Advanced Engineering Associates International, Inc. (AEAI). (Undated). Enterprise Energy Efficiency (E3). IQC contract # EPP-I-00-03-00004-00. Task order #12. Annual progress report. May 4, 2013-May 4, 2014. Advanced Engineering Associates International, Inc. (AEAI). (2013). Regulatory and Energy Assistance Project in Bosnia and Herzegovina (BIH) - REAP BIH. Completion report. November 29. AECOM International Development. (2011). Southern Africa Trade Hub. Annual report FY 2011. September 20, 2010-September 30, 2011. December. AECOM International Development. (2012). Southern Africa Trade Hub. Annual report FY 2012. October 1, 2011-September 30, 2012. October. AECOM International Development. (2013). Southern Africa Trade Hub. Annual report for the period October 2012-September 2013. October 31. AECOM International Development. (2014). Southern Africa Trade Hub. Annual report for the period October 2013-September 2013. October 23. AECOM International Development. (2015). Southern Africa Trade Hub. Annual report for the period October 2014-September 2015. October 23. African Cashew Alliance. (Undated). USAID TIME Project - FY2015 indicator table. African Cashew Alliance, Global Shea Alliance, Borderless Alliance. (2014a). USAID TIME. Industry alliances working together to empower poor rural women in the cashew and shea sectors. Quarterly report. May 16. African Cashew Alliance, Global Shea Alliance, Borderless Alliance. (2014b). USAID TIME. Industry alliances working together to empower poor rural women in the cashew and shea sectors. Quarterly report. August 5. African Cashew Alliance, Global Shea Alliance, Borderless Alliance. (2014c). USAID TIME. Industry alliances working together to empower poor rural women in the cashew and shea sectors. Quarterly report. July-September 2014. African Cashew Alliance, Global Shea Alliance, Borderless Alliance. (2014d). USAID TIME. Industry alliances working together to empower poor rural women in the cashew and shea sectors. Quarterly report. October-December 2014. African Cashew Alliance, Global Shea Alliance, Borderless Alliance. (2014e). USAID TIME. Industry alliances working together to empower poor rural women in the cashew and shea sectors. Quarterly report. May 16. USAID 2017 TCB Evaluation | 122 African Cashew Alliance, Global Shea Alliance, Borderless Alliance. (2015a). USAID TIME. Industry alliances working together to empower poor rural women in the cashew and shea sectors. Quarterly report. January-March 2015. African Cashew Alliance, Global Shea Alliance, Borderless Alliance. (2015b). USAID TIME. Industry alliances working together to empower poor rural women in the cashew and shea sectors. Quarterly report. April-June 2015. African Cashew Alliance, Global Shea Alliance, Borderless Alliance. (2015c). USAID TIME. Industry alliances working together to empower poor rural women in the cashew and shea sectors. Quarterly report. July-September 2015. African Cashew Alliance, Global Shea Alliance, Borderless Alliance. (2016a). USAID TIME. Industry alliances working together to empower poor rural women in the cashew and shea sectors. Quarterly report. July-September 2016. African Cashew Alliance. (2016b). Toward Inclusive Markets Everywhere (TIME) 2016 annual report data. AGP-Livestock Market Development Project. (2013). Agricultural Growth Program-Livestock Market Development. Expanding livestock markets for small-holder producers. Annual report. (Sept. 17, 2012-June 30, 2013). August 15. AGP-Livestock Market Development Project. (2014). Agricultural Growth Program-Livestock Market Development. Expanding livestock markets for small-holder producers. Annual report. (July 1, 2013-June 30, 2014). July 31. AGP-Livestock Market Development Project. (2015). Agricultural Growth Program-Livestock Market Development. Expanding livestock markets for small-holder producers. Year 3 annual report. (July 1, 2014-June 30, 2015). July 30. The Agribusiness Project. (2012). First annual progress report (APR-1). November 10, 2011-November 9, 2012. December 9. The Agribusiness Project. (2013). Second annual progress report-II. 01 October 2012-30 September, 2013. December 10. Alampay, G. (2016). Lessons learned from advocating for the passage of the Philippine Competition Act. Mimeo addressed to Ciel Habito and Rebecca Hoffman. October 13. AMEX International and DevTech Systems, Inc. (2011). Evaluation. USAID/Dominican Republic tri-project program evaluation. Dominican Republic-Central American Free Trade Agreement Implementation Project (DR-CAFTA-IP). August 29. ARD. (2007). MIDAS. Mas Inversión para el Desarrollo Alternativo Sostenible. Program level work plan - 2007. January. ARD. (2008). Programa MIDAS. Mas Inversión para el Desarrollo Alternativo Sostenible. Work plan 2008. Contract no. 514-C-00-06-00301-00. USAID 2017 TCB Evaluation | 123 ARD. (2008). Programa MIDAS. Mas Inversión para el Desarrollo Alternativo Sostenible. Performance management plan. Contract no. 514-C-00-06-00301-00. ARD. (2009a). Programa MIDAS. Mas Inversión para el Desarrollo Alternativo Sostenible.Work plan 2009- 2010. Contract no. 514-C-00-06-00301-00. ARD. (2009b). Property Rights and Artisanal Diamond Development Project - Guinea. Closeout report (January-April 2009). May. American Research Center in Egypt (ARCE). (2015). Final report (October 1, 2011 through December 31, 2014). Job creation through cultural resource management in Luxor (APS). March 11. Aramouny, P., Konar-Leacy-V., Rader, P., Sakhr, Z., Steele, C., & Yared, R. (2011). Support for Lebanon’s accession to the World Trade Organization (WTO): The way forward 2011-2012. Booz Allen Hamilton. May 5. Bear, M. & Banda, R. (2015). Feed the Future Uganda. Agriculture inputs activity. Strategic assessment report. June-July 2015. Tetra Tech ARD. September. Bonifaz, A., Hawkins, D., & Israel, R. (2010). Guide to designing workforce development programs. With a special focus on job and career opportunities for youth. Guides and toolkits series. Education Quality Improvement Program 3. Education Development Center, Inc. Booz Allen Hamilton Inc.. (2010). Support for Lebanon’s accession to the World Trade Organization. Annual project report, year 1. December 1, 2009-September 30, 2010. December. Booz Allen Hamilton. (2012). Support for Lebanon’s Accession to the World Trade Organization (WTO). Final report. November 27. Booz Allen Hamilton. (2015a). Investment Development and Export Advancement Support Project. Annual report. October 2014-December 2015. December. Booz Allen Hamilton. (2015b). Investment Development and Export Advancement Support Project. Final report. December 2010-December 2015. December. Buehrer, T. (2015). 2014 Evaluation report: Evaluation of key indicators. ASEAN Connectivity through Trade and Investment (ACTI). Nathan Associates, Inc. April 21. Buehrer, T. (2015a). 2015 Annual performance management progress report. ASEAN Connectivity through Trade and Investment (US-ACTI). Nathan Associates, Inc. October 31. Buehrer, T. (2015b). Annual report 2014. ASEAN Connectivity through Trade and Investment (ACTI). Nathan Associates, Inc. March. Buehrer, T. (2016). Annual report 2015. ASEAN Connectivity through Trade and Investment (ACTI). Nathan Associates, Inc. February. Cambronero, S., Midling, M., & Molina, F. (2012). Evaluation. Performance evaluation of the “Improving Access to Employment Program in El Salvador”. International Business & Technical Consultants Incorporated (IBTCI) and Aguirre Division of JBS International. October. USAID 2017 TCB Evaluation | 124 CARANA Corporation. (Undated). Paraguay Productivo. Final report. September 2009-September 2012. CARANA Corporation. (2008). USAID/Macedonia Competitiveness Project. Year 1 annual report July 2007-September 2008. CARANA Corporation. (2009). USAID/Macedonia Competitiveness Project. Annual report October 1, 2008-September 30, 2009. CARANA Corporation. (2010a). Rwanda Post-Harvest Handling and Storage Project. Annual progress report 2010. Project #EEMI-00-7-00006-00. Task order 09. September. CARANA Corporation. (2010b). USAID/Macedonia Competitiveness Project. Annual report October 1, 2009-September 30, 2010. CARANA Corporation. (2010c). West Africa Trade Hub. Annual report. October 2010-September 2011. November. CARANA Corporation. (2011). West Africa Trade Hub. Annual report. FY 2011-2012. October. CARANA Corporation. (2012a). USAID/Macedonia Competitiveness Project. Final report. CARANA Corporation. (2012b). Final report. Enterprise Development and Investment Promotion (EDIP). October. CARANA Corporation. (2013a). USAID Improving Access to Employment Program. Final report. November. CARANA Corporation. (2013b). Nigeria Expanded Trade and Transport Program (NEXTT). Annual report. October 2012-September 2013. October. CARANA Corporation. (2013c). Promoting Equitable Economic Growth Nicaragua Enterprise and Employment Program Final Report 2013. August. CARANA Corporation. (2014). Nigeria Expanded Trade and Transport Program (NEXTT). Year 2 annual report. October 2013-September 2014. October 31. CARANA Corporation. (2015). Nigeria Expanded Trade and Transport Program (NEXTT). Year 3 annual report. October 1st, 2014-September 30th, 2015. October 31. CARANA Corporation. (n.d.) West Africa Trade Hub. Annual report. October 2008-September 2009. CARANA Corporation. (n.d.) West Africa Trade Hub. Annual report. October 2009-September 2010. Cardno Emerging Markets USA, Ltd. (2011). USAID’s Business Enabling Project. Final Report: 2011. Checchi and Company Consulting. (2012). Evaluation. Mid-term evaluation Trade and Accession Facilitation for Afghanistan (TAFA) Project. March. USAID 2017 TCB Evaluation | 125 Checchi and Company Consulting, Inc. (2015a). Final performance evaluation. (March-May 2015). Agricultural Credit Enhancement (ACE) program. May. Checchi and Company Consulting, Inc. (2015b). Mid-term evaluation. (June 4-August 23, 2015). The Afghanistan Trade and Revenue Project. August. Chemonics. (Undated). Governance for Inclusive Growth Program (GIG). Year 1 work plan. 1 October 2014-30 September 2015. Hanoi, Vietnam. Chemonics. (Undated). Governance for Inclusive Growth Program (GIG). Year 2 work plan. 12 December 2013-30 September 3014. Hanoi, Vietnam. Chemonics International. (2008). CAFTA-DR Regional Trade Program. Work plan for calendar year 2008. January. Chemonics International. (2009a). Poverty Reduction by Increasing the Competitiveness of Enterprises (PRICE). First annual report: February 2008-March 2009. Chemonics International. (2009b). Competitiveness and Trade Expansion (COMPETE) Program. Performance monitoring plan. December 23. Chemonics International. (2010a). Trade Accession and Facilitation for Afghanistan (TAFA) Annual report. November 2009-November 2010. Chemonics International. (2010b). Bangladesh Poverty Reduction by Increasing the Competitiveness of Enterprises (PRICE) Annual progress report October 2009-September 2010. Chemonics International. (2012). Trade Accession and Facilitation for Afghanistan (TAFA) Annual report November 2011-November 2012. Chemonics International. (2013a). Annual report: October 2011-September 2012. USAID SME Development Program. January. Chemonics International. (2013b). Business Regulatory, Investment, and Trade Environment Program. Final report. July. Chemonics International. (2013c). Year 1 annual report. Business Regulatory, Investment, and Trade Environment (BRITE) Program. June 2012-September 2013. October. Chemonics International. (2013d). The East Africa Trade Hub. Annual progress report. October 2012- September 2013. October. Chemonics International. (2013e). Bangladesh Poverty Reduction by Increasing the Competitiveness of Enterprises (PRICE) Annual progress report October 1, 2012-September 30, 2013. Chemonics International. (2013f). Trade Accession and Facilitation for Afghanistan. Final report. Chemonics International. (2014a). Second annual report: October 2012-September 2013. USAID SME Development Program. January. USAID 2017 TCB Evaluation | 126 Chemonics International. (2014b). Year 1 annual report. Business Regulatory, Investment, and Trade Environment (BRITE) Program. October 2013-September 2014. October. Chemonics International. (2014c). Investment Climate Improvement Project. Final report 2010-2014. Task order no. EEM-I-13-07-00008-00. December 15. Chemonics International. (2014d). Building skills and market linkages in Bangladesh - Poverty Reduction by Increasing the Competitiveness of Enterprises (PRICE). Final report. Chemonics International. (2015a). Afghanistan Trade and Revenue (ATAR) Project. Contract no. AID-OAA-I￾12-00035. Annual report. November 7, 2013-November 6, 2014. February 14. Chemonics International. (2015b). Third annual report: October 2013-September 2014. USAID SME Development Program. February. Chemonics International. (2015c). Business Regulatory, Investment, and Trade Environment Program.Year 3 annual report. October 2014-September 2015. October. Chemonics International. (2015d). Fourth annual report: October 2014-September 2014. USAID SME Development Program. February. Chemonics International. (2016a). Afghanistan Trade and Revenue (ATAR) Project. Contract no. AID-OAA-I￾12-00035. Quarterly report. January-March 2016. April 30. Chemonics International. (2016b). Business Regulatory, Investment and Trade Environment Program Final Report. July. Chemonics International. (2016c). Afghanistan Trade and Revenue (ATAR) Project. Contract no. AID-OAA-I￾12-00035. Quarterly report. April-June 2016. July 31. Chemonics International Inc. (Undated). Governance for Inclusive Growth (GIG). Annual report #1. December 12, 2013-September 30, 2014. Chemonics International Inc. (Undated). Governance for Inclusive Growth (GIG). Quarterly performance report (QPR) #6. (Q3 FY2015). April 1-June 30, 2015. Chemonics International Inc. (2010a). USAID Regional Trade Program for CAFTA-DR. Final report. August. Chemonics International Inc. (2010b). DR-CAFTA Implementation annual progress report. October 2009- September 2010. October 15. Chemonics International Inc. (2011). Annual report. Nepal, economic, agriculture, and trade activity. October 2010-September 2011. October. Chemonics International Inc. (2012a). Final report. Five years of CAFTA-DR implementation. September 30. Chemonics International Inc. (2012b). Annual report. Nepal, economic, agriculture, and trade activity. October 2011-September 2012. October. USAID 2017 TCB Evaluation | 127 Chemonics International Inc. (201b). Annual report. Nepal Economic, Agriculture, and Trade Activity. October 2012-August 2013. August. Chemonics International Inc. (201g). Strengthening the foundations for inclusive economic growth. Nepal Economic, Agriculture, and Trade (NEAT) Activity: Final report. August. Chemonics International Inc. (2014). Feed the Future West/Winner. Increase in mango exports due to project interventions. March. Chemonics International Inc. (2016). Governance for Inclusive Growth. December 2013-December 2018/Implementer: Chemonics International, Inc. Newsletter. March. Chemonics International Inc. (Undated). Feed the Future West/Winner. Annual report. Fiscal year 2013. October 2012-September 2013. Chemonics International Inc. (Undated). Feed the Future West/Winner. Final report. June 2009-May 2014. Chemonics International Inc. (Undated). Governance for Inclusive Growth (GIG). Quarterly performance report (QPR) #5. (Q2 FY2015). January 1-March 31, 2015. Corporate Council on Africa. (2013). Final report. South African International Business Linkages - Phase 2 (SAIBL-2). DAI. (2011). Jordan Fiscal Reform II Project. Year 1 results report. June. DAI/Nathan Group LLC. (2013). USAID/Vietnam Support for Trade Acceleration Plus (STARS PLUS). Final report. August 31. DAI, ACDI/VOCA, & Mercy Corps. (2015). Final report. Incentives Driving Economic Alternatives for the North, East, and West (IDEA-NEW). March 2, 2009-September 30, 2015. December. Deloitte Consulting LLP. (2010). USAID/Pakistan: TRADE. Annual performance report. Project year 1 (July 2009-June 2010). 31 July 2010. July. Deloitte Consulting LLP. (2013a). Economic Prosperity Initiative (EPI). Annual report. Year 3. October 1, 2012-September 30, 2013. October 30. Deloitte Consulting LLP. (2013b). Quarterly progress report: May-September 2013. Office of the Economic Development and Governance Trade Related Assisted for Development (TRADE). Contract no. AID-492-C-13-00001. October 30. Deloitte Consulting LLP. (2014a). USAID Trade Project. Final report. September. Deloitte Consulting LLP. (2014b). Annual progress report: October 2014-September 2015. Office of the Economic Development and Governance Trade Related Assisted for Development (TRADE). Contract no. AID-492-C-13-00001. October 30. USAID 2017 TCB Evaluation | 128 Deloitte Consulting LLP. (2015). Annual progress report: October 2013-September 2014. Office of the Economic Development and Governance Trade Related Assisted for Development (TRADE). Contract no. AID-492-C-13-00001. October 30. Development Alternatives Incorporated (DAI). (2008). USAID/Serbia Agribusiness Project year 1 annual report: October 1 2007- September 30 2008. Development Alternatives Incorporated (DAI). (2009). USAID/Serbia Agribusiness Project year 2 annual report: October 1 2008- September 30 2009. Development Alternatives Incorporated (DAI).(2010). USAID/Serbia Agribusiness Project year 3 annual report: October 1 2009- September 30 2010. Development Alternatives Incorporated (DAI). (2011). USAID/Serbia Agribusiness Project year 4 annual report: October 1 2010- September 30 2011. Development Alternatives Incorporated (DAI). (2012). USAID/Serbia Agribusiness Project completion report-narrative sections September 28, 2007-September 27 2012. Final report. Development Alternatives Incorporated (DAI). (2013). Annual progress report and fourth quarterly report - FY13 Feed the Future Integrating Nutrition in Value Chains Project, Malawi. Development Alternatives Incorporated (DAI). (2014a). First quarterly report - FY14 Feed the Future Integrating Nutrition in Value Chains Project, Malawi. Development Alternatives Incorporated (DAI). (2014b). Second quarterly report - FY14 Feed the Future Integrating Nutrition in Value Chains Project, Malawi. Development Alternatives Incorporated (DAI). (2014c). Third quarterly report - FY14 Feed the Future Integrating Nutrition in Value Chains Project, Malawi. Development Alternatives Incorporated (DAI). (2014d. Fourth quarterly report - FY14 Feed the Future Integrating Nutrition in Value Chains Project, Malawi. Development Alternatives Incorporated (DAI). (2015). Integrating Nutrition in Value Chains-Malawi FY15 fourth quarterly report (July-September 2015) with annual results. Development Alternatives, Inc. (2011). Agribusiness and Market Support Activity (AMARTA) Final report. June 17. Development Alternatives, Inc. (2015). Agricultural Credit Enhancement (ACE) Program. Final report - February 2015. February 15. Development & Training Services, Inc. (dTS). (2011). Evaluation of the South African International Business Linkages Program. July 21. Development & Training Services, Inc. (dTS). (2012a). Evaluation. Compendium report. Multi-stakeholder evaluation of agriculture and livestock value chain activities in Kenya. March 29. USAID 2017 TCB Evaluation | 129 Development & Training Services, Inc. (dTS). (2012b). Evaluation of West Africa Trade-Related Projects. May 8. Dod, D. & Fekete, P. (2012). Mid-term evaluation of Investment Climate and Improvement Project: Fiscal and policy components. USAID/West Bank/Gaza. October. Fintrac Inc. (2011). Kenya Horticulture Competitiveness Project. Annual report. October 2010-September 2011. October. Fintrac Inc. (2012). Kenya Horticulture Competitiveness Project. Annual report #2, 2011-2012. October. Fintrac Inc. (2013a). Kenya Agricultural Value Chain Enterprises Project (USAID-KAVES). FY 2014 annual report. 16 January-30 September 2013. September. Fintrac Inc. (2013b). Kenya Horticulture Competitiveness Project. Annual report #3, 2012-2013. October. Fintrac Inc. (2014a). USAID/Kenya. Kenya Agricultural Value Chain Enterprises. Annual performance report FY2. September. Fintrac Inc. (2014b). Kenya Horticulture Competitiveness Project (USAID-KHCP). Annual report #4, 2013- 2014. October. Fintrac Inc. (2015a). Final report. (April 2011-May 2015). USAID-ACCESO. May. Fintrac Inc. (2015b). USAID/Kenya. (Kenya Agricultural Value Chain Enterprises). Annual report FY2015. September. Fintrac Inc. (2016).Kenya Agricultural Value Chain Enterprises (KAVES). Newsletter. June. Flores, L. G. (2012). Agricultural Credit Enhancement Program. Midterm impact evaluation. Development Alternatives Inc. November. Graul, B. (2012). Pakistan Trade Project. Annual report. June 2011-June 2012. Deloitte Consulting LLP. July 22. Guatemalan Exporters Association AGEXPORT Development Division (Undated). Annual report. Rural Value Chains Project. Period from: October 2012-September 2013. Guatemalan Exporters Association Development Division (2014). Cooperative agreement no. AID-520- A-12-00003. Annual report FY2014. November. Guatemalan Exporters Association AGEXPORT Development Division (2015). Annual report FY2015. Rural Value Chains Project. Term October 2014-September 2015. Cooperative agreement no. AID-520-A-12-00003. Hecht, J. & Mitchell, A. (2014). Global Sustainable Tourism Alliance (GSTA) performance evaluation. Final report. ECODIT and International Business and Technical Consultants, Inc. (IBTCI). April. Hageboeck, M. (2010a). From aid to trade: Delivering results. A cross-country evaluation of USAID USAID 2017 TCB Evaluation | 130 trade capacity building. Management Systems International. November 10. Hageboeck, M. (2010b). From aid to trade: Delivering results. A cross-country evaluation of USAID trade capacity building. Summary. Management Systems International. November 24. Husain, T. & Scheik, S. (2014). EVALUATION. Pakistan Trade Project. Midterm performance evaluation report. Management Systems International. June 24. International Business and Technical Consultants, Inc. (IBTCI). 2013. USAID/Mali. Integrated Initiatives for Economic Growth in Mali (IICEM) performance evaluation. October. International Development Group. (2013). Tunisia ICT Competitiveness Program. Final report. International Development Group LLC. (Undated). Azerbaijan Competitiveness and Trade (ACT) Project. ACT final report. October 1, 2010-September 30, 2013. International Resources Group (IRG). (2009). Liberia Energy Assistance Program (LEAP) final report. February 28. Kenworthy, J., Birhanu, F. M., & Hailu, M. B. (2010). USAID/Ethiopia WTO Accession Plus Project Evaluation. The Mitchell Group. March 28. Kernan, B. & Rivas, C. (2012). Evaluation of the USAID/Dominican Republic biodiversity portfolio. Final evaluation report. Social Impact, Inc. (SI).. December. Khan, U.A., Kahnum, Z., Krieger, D., Rehman, Z.U., Shah, S.N.A., Tirnauer, J., & Ullah, H. (2015). Building sustainable agribusiness in Pakistan. Final evaluation of the Agribusiness Project. Management Systems International. November 4. Kormis, R. & Muir, D. (2012). Draft final report. Business Regulatory & Tax Administration (BIZTAR) project - Moldova. DAI. January. Kruze, A., Walter, B., & Strik, F. (2013). Performance evaluation of the Regulatory and Energy Assistance Program (REAP). Final report. USAID/Bosnia and Herzegovina. Social Impact. September. Management Systems International (MSI) and Centro Nacional de Consultoría (CNC). (2014). Evaluation of the More Investment in Sustainable Alternative Development (MIDAS) and Areas for Municipal Alternative Development (ADAM) programs. Final report. March. Management Sciences for Development. (2012). USAID Strengthening Labor Justice Program for CAFTA￾DR Final report. Maclean, C., Celler, B., & Zaharia, V. (2011). Business Regulation and Tax Administration Reform (BIZTAR) evaluation. Final report. Mendez England & Associates. May 2. Mendez England and Associates. (2011). Serbia Agribusiness Project (SAP) evaluation. Final report. November 8. Mendez England and Associates. (2012). Final performance evaluation of USAID/Cambodia Micro, Small, and Medium Size Enterprises II/Business Enabling Environment Project. December 27. USAID 2017 TCB Evaluation | 131 Mendez England and Associates. (2013a). EVALUATION. Performance evaluation of the USAID/Mozambique agriculture portfolio. January. Mendez England and Associates. (2013b). Mid-Term Performance Evaluation of USAID/Serbia Business Enabling Project (BEP). Final report. July. Mendez England and Associates. (2013c). Final Performance Evaluation of the Azerbaijan Competitiveness and Trade (ACT) Project. Final evaluation report. September. Mendez England and Associates & NORC. (2013a). Evaluation. Mid-term performance evaluation of the Georgia Economic Prosperity Initiative. June. Midling, M., Molina, F., Sandoval, R., & López-Tull, S. (2013). Final performance evaluation of the USAID Municipal Competitiveness Project in El Salvador. Evaluation report. January 29. The Mitchell Group. (2015). Mid-term performance evaluation of the Nigeria Expanded Trade and Transport (NEXTT) Project. Increase Nigeria’s capacity to trade agricultural products domestically, within the ECOWAS sub-region and beyond, and improve trade efficiency and inclusiveness. August. Mudge, J., Walker, J., Channell, W., Fekete, P., & Khy, D. (2011). Performance evaluation of the USAID/Vietnam Support for Trade Acceleration (STAR) Project. Final report. May. Nathan Associates Inc. (2012a). Annual Report. October 2011-September 2012. Peru and Andean Trade Capacity Building Program: USAID/Facilitando Comercio. October. Nathan Associates Inc. (2012b). Quarterly Progress Report. Peru and Andean Trade Capacity Building Program: USAID/Facilitando Comercio. October. Nathan Associates Inc. (2012b). Annual Report. October 2011-September 2012. Peru and Andean Trade Capacity Building Program: USAID/Facilitando Comercio. October-December 2011. January. Nathan Associates Inc. (2013a). Peru and Andean Trade Capacity Building Program Final Report. USAID/Facilitando Comercio. December. Nathan Associates Inc. (2013b). Quarterly progress report. Peru and Andean Trade Capacity Building Program: USAID/Facilitando Comercio. January-March 2013. National Capital Advisors LLC. (2011). The Responsible Asia Forestry and Trade (RAFT) Alliance. Final evaluation. July. The Nature Conservancy. (2011). Environmental Protection Program. Cooperative agreement no. 517-A￾00-09-00106-00. Quarterly and annual report. Oct. 2010-Sept. 2011. December 21. Neubert, D. (2011). The Agribusiness Market and Support Activity: Final evaluation. USAID: EDH-I-00-05- 00004-00 AND EDH-I-04-05-00004. Weidemann Associates, Inc. June.-00 NRECA International Ltd. (Undated). Caracol Community Electrification Program. Contract no. AID-521-A￾12-00005. Final report. USAID 2017 TCB Evaluation | 132 Office of the Inspector General. (2011a). Audit of USAID’s Liberia Energy Program activities. Audit report no. 7-660-11-007-P. June 6. Office of the Inspector General. (2011b). Audit of USAID/Rwanda’s agricultural activities.. Audit report no. 4-696-11-008-P. July 1. Office of the Inspector General. (2012a). Audit of USAID/Afghanistan’s Incentives Driving Economic Alternatives for the North, East, and West Program. Audit report no. F-306-12-004-P. June 29. Office of the Inspector General. (2012b). Audit of USAID/Georgia’s Economic Prosperity Initiative. Audit report no. 9-114-13-001-P. December 26.. Office of the Inspector General. (2013a). Audit of USAID/Vietnam’s Support for Trade Acceleration Plus Project. Audit report no. 5-440-13-005-P. May 6. Office of the Inspector General. (2013b). Audit of USAID/El Salvador’s Municipal Competitiveness Project.. Audit report no. 1-519-13-004-P. January 22. Office of the Inspector General. (2013c). Audit of USAID/Pakistan’s Agribusiness Project. Audit report no. G-391-13-004-P. June 13. Office of the Inspector General. (2013c). Audit of USAID/Jordan’s Fiscal Reform Project II. Audit report no. 6-278-14-003-P. December 11. (Revised). Office of the Inspector General. (2015a). Audit of USAID/ Audit report no. 1-522-15-003-P. January 26. Office of the Inspector General. (2015b). Audit of USAID/Honduras’s ACCESSO Project. Mozambique’s agriculture, trade, and business portfolio. Audit report no. 4-656-15-004-P. September 22. Ohio State University and Virginia Polytechnic Institute and State University. (2011). iAGRI Program performance management plan FY 2011-FY 2016. Ohio State University. (2012)a. iAGRI annual report March 1-September 30 2011. Ohio State University. (2012b). Innovative Agricultural Research Initiative (iAGRI) annual report October 1, 2011 - September 30, 2012. Ohio State University. (2013). Innovative Agricultural Research Initiative (iAGRI) annual report October 1, 2012 - September 30, 2013. Ohio State University. (2014). Innovative Agricultural Research Initiative (iAGRI) annual report October 1, 2013 - September 30, 2014. Ohio State University. (2016a). Innovative Agricultural Research Initiative (iAGRI) quarterly report January 1st to March 31st 2016. Ohio State University. (2016b). Innovative Agricultural Research Initiative (iAGRI) quarterly report April 1st to June 30th 2016. USAID 2017 TCB Evaluation | 133 Optimal Solutions Group. (2012). The mid-term performance evaluation of USAID/Macedonia’s Investment Development and Export Advancement Support (IDEAS) Project. Final report. December 12. Optimal Solutions Group. (2014a). Poverty Reduction by Increasing the Competitiveness of Enterprises (PRICE) final performance evaluation. Final report. January 13. Optimal Solutions Group, LLC. (2014b). Nepal Economic, Agriculture, and Trade (NEAT) Activity. Performance evaluation. Final report. May 30. Optimal Solutions Group, LLC. (2015). USAID Europe and Eurasia regional energy security evaluation. Final report. February 24. Paraguay Productivo implementation team. (2011). Paraguay PRODUCTIVO. Annual report. October 2010 to September 2011. October. Potter, R. (2014). Evaluation. Performance evaluation of the Program for Biosafety Systems. April. The Pragma Corporation and Management Systems International. (2014). Mid-term evaluation of the Southern Africa Trade Hub. August. The Pragma Corporation. (2015). East Africa Trade Hub (EATF). 2009-2014. Final evaluation report. August 14. Project Team. (2014). Enterprise Energy Efficiency (E3). Sixteenth quarterly progress report. January 1, 2014-March 31, 2014. Advanced Engineering Associates International, Inc. (AEAI). May 5. Rhaiem, T., Creskoff, S., Junge, N., Melikyan, L. Warman, A., & Aqrabawi, M. (2014). Management Systems International (MSI) and Integrated Services Indigenous Solutions. Performance evaluation of the USAID/Jordan Fiscal Reform Project II (FRP II). October 1. Rockeman, K., Tadesse, B., & Redda, T. (2015). Agricultural Growth Program - Livestock Market Development (AGP-LMD) Project. External mid-term evaluation performance report. Tufts University. May. Rodríguez, A., Engels, J., Mucha, N., & Malunga, C. (2015). The Feed the Future Integrating Nutrition in Value Chains Project performance evaluation. Final report. May. RTI International. (2011). USAID Municipal Competitiveness Project. Annual performance report. October 1, 2010-September 30, 2011. October 31. RTI International. (2012). USAID Municipal Competitiveness Project. Annual performance report. October 1, 2011-September 30, 2012. October 31. RTI International. (2014a). USAID Municipal Competitiveness Project. Annual performance report. October 2012-September 2013. May. RTI International. (2014b). USAID Municipal Competitiveness Project. Annual performance report. October 1, 2013-September 30, 2014. October. USAID 2017 TCB Evaluation | 134 RTI International. (2015). USAID Municipal Competitiveness Project. Final report. September 22, 2010-May 27, 2015. April 30. Rural Value Chains Project. (2014). USAID-ANACAFE cooperative agreement 520-A-12-00004. Annual report. October 1, 2013 through September 30, 2014. October 31. Rural Value Chains Project. (2015). USAID-ANACAFE cooperative agreement 520-A-12-00004. Annual report. October 1, 2014 through September 30, 2015. October 31. Schippers, V., Van Dusen, N., & Beyard, K. (2012). CARANA Corporation. Lessons learned. USAID/Rwanda Post Harvest Handling and Storage (PHHS) Project. July. Segura Partners LLC (2011). Evaluation of the regional and bilateral programs to support trade compliance under CAFTA-DR. Final report. August. Segura Partners LLC. (2013.) Knowledge Sharing and Analysis (KSA) Project. Final report , 2007-2013. June. Sibley International. (2010). USAID/Macedonia Competitiveness Project Evaluation. Midterm evaluation report. Sibley International. (2012a). Quarterly and monthly report, May-June 2012, Tunisia ICT Competitiveness Program. Sibley International. (2012b). Quarterly and monthly report, July-September 2012, Tunisia ICT Competitiveness Program. Sibley International. (2013a). Quarterly and monthly report, October-December 2012, Tunisia ICT Competitiveness Program. Sibley International. (2013b). Quarterly and monthly report, January-March 2013, Tunisia ICT Competitiveness Program. Sibley International. (2013c). Quarterly and monthly report, April-June 2013, Tunisia ICT Competitiveness Program. Snelbecker, D., Dimovski, L., Orozovic, S., & Ferrara, A. (2010). USAID Macedonia Competitiveness Project. Evaluation. May. Taylor, G. F. III, Dowhan, J. J., Rizvi, J. F., & Manembu, A. N. (2013). Mid-term performance evaluation of Asia’s regional response to endangered species trafficking program. Evaluation report. Social Impact, Inc. and Management Systems International. October. Tetelman, M. (2015a). USAID Connecting the Mekong through education and training quarterly report. First quarter - October 06, 2015 to January 06, 2015. January 15. Tetelman, M. (2015b). USAID Connecting the Mekong through education and training quarterly report. Second quarter - January 7, 2015 to April 6, 2015. April 15. USAID 2017 TCB Evaluation | 135 Tetelman, M. (2015c). USAID Connecting the Mekong through education and training quarterly report. Third quarter - April 7, 2015 to July 6, 2015. July 31. Tetelman, M. (2015d). USAID Connecting the Mekong through education and training. Year one annual report. October 6, 2014-November 5, 2015. November 6. Tetra Tech. (2014). Property Rights and Artisanal Diamond Development II (PRADD II). Quarterly progress report. April-June 2014. July. Tetra Tech. (2015). Property Rights and Artisanal Diamond Development II (PRADD II). Quarterly progress report. April-June 2015. July. Tetra Tech ARD. (2011a). Programa MIDAS. Mas Inversión Para El Desarrollo Alternativo Sostenible. Final report. Contract no. 514-C-00-06-00301-00. March. Tetra Tech ARD. (2011b). Property Rights and Artisanal Diamond Development (PRADD). Combined annual work plan for Central African Republic, Liberia and regional work. June. Tetra Tech ARD. (2013a). End-line survey results - revised. Property Rights and Artisanal Diamond Development (PRADD) in Liberia. January. Tetra Tech ARD. (2013b.) Property Rights and Artisanal Diamond Development (PRADD) Project. Quarterly progress report. (October-December 2012). January. Tetra Tech ARD. (2013c). USAID/Uganda. Feed the Future Uganda. Agriculture Inputs Activity. Year-1 annual report: January-September 2013. Year-2 annual work plan: October 2014-September 2014. October. Tetra Tech ARD. (2014a) Property Rights and Artisanal Diamond Development II (PRADD II). Quarterly progress report. July-September 2014. October. Tetra Tech ARD. (2014b). USAID Feed the Future Uganda. Agricultural Inputs Activity. Annual report: October 2014-September 2015. October. Tetra Tech ARD. (2015). Feed the Future Uganda. Agricultural Inputs Activity. Annual report: October, 2013-September, 2014. Annual work plan: October, 2014-September 2015. October. Tetra Tech ARD. (2016). Feed the Future Uganda. Agricultural Inputs Activity. Quarterly report: October￾December 2015. January. Thomas, E.S., Chiapetta, M., Limange, J., Mazotta, B., & Maraga, J. (2013). Africa Trade Hubs export promotion evaluation. Evaluation report. October. United States Government Accountability Office. (GAO, 2013). June. Haiti reconstruction. USAID infrastructure projects have had mixed results and face sustainability challenges. Report to Congressional requesters. June. United States General Accounting Office. (GAO, 2014). Foreign assistance. USAID should update its trade capacity building strategy. Report to the Ranking Member, Committee on Foreign Relations, U.S. Senate. August. USAID 2017 TCB Evaluation | 136 USAID. (Undated). Responsible Asia Forestry and Trade (RAFT). Final report (2006-2011). USAID. (2016). Trade Capacity Building Policy. September. https://www.usaid.gov/documents/1865/policy-trade-capacity-building USAID AgriFUTURO. (2010). USAID AgriFUTURO Project. Quarterly performance monitoring report. Annual progress report. October 09-September 2010. October. USAID AgriFUTURO. (2011). USAID AgriFUTURO Project. Quarterly performance monitoring report. Annual progress report. October 10-September 2011. October. USAID AgriFUTURO. (2012). USAID AgriFUTURO Project. Quarterly performance monitoring report. Annual progress report. 1 October 2011-30 September 2012. October. USAID AgriFUTURO. (2013). USAID AgriFUTURO Project. Annual performance monitoring report. 1 October 2012-30 September 2013. October. USAID/Ethiopia. (Undated). Agricultural Growth Program – Agribusiness and Market Development (AGP￾AMDe). Quarterly report October 1–December 31, 2013. Contract number AID #663-TO￾11-0000. USAID/Ethiopia. (Undated). Agricultural Growth Program – Agribusiness and Market Development (AGP￾AMDe). Quarterly Report October 1, 2014–December 31, 2014. Contract number AID #663-TO-11-0000. USAID/Ethiopia. (2015). Agricultural Growth Program – Agribusiness and Market Development, Quarterly Report October 1–December 31, 2015. Contract number AID #663-TO-11-0000. USAID Office of Inspector General (2010). Audit of USAID/Colombia’s Alternative Development Program. USAID/PARAGUAY PRODUCTIVO implementation team. (Undated). Annual report. September 2009 to September 2010. Van Dusen, N. & Beyard, K. (undated). Post-Harvest Handling and Storage (PHHS) Project. Final report. USAID/Rwanda. CARANA Corporation. Vanegas, M., Bermudez, E., & Church, P. (2011). Evaluation. USAID/Dominican Republic tri-project evaluation. The Dominican Republic Sustainable Tourism Alliance (DSTA). AMEX International and DevTech Systems, Inc. September 2. Varnalieva, J. & Johnson, B. (2012). USAID’s Competitiveness Project. Final report. July 12. Weinstock, J., Nielsen, R., & Mitchell, A. (2014). Property Rights and Resource Government Program (PRRG). Performance evaluation final report. International Business and Technical Consultants, Inc. (IBTCI). April. USAID 2017 TCB Evaluation | 137