Contract No.: GS-10F-0188V, Task Order: AID-OAA-M-14-00018 February – June 2019 This report was produced for review by the United States Agency for International Development. It was prepared by Dalberg Consulting U.S., LLC for the USAID Investment Support Program (ISP). SOUTHERN AFRICA TRADE AND FOOD SECURITY PROJECT MID-TERM PERFORMANCE EVALUATION Abstract USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 1 ABSTRACT This mid-term performance evaluation of the USAID Trade and Food Security Project (TFSP) addresses the following activities: 1. The Southern Africa Trade and Investment Hub (SATIH) – launched September 2016 2. The Seed Trade Project (STP) – launched September 2016 3. The Trade Policy Initiative (TPI) – launched December 2013 1 As the project was run as three distinct activities under a common Project Appraisal Document (PAD), project-level performance largely reflects the sum performance of the three activities. Although SATIH has met Y1 and Y2 targets for indicators such as value of exports and investment facilitated (notwithstanding the need for a significant ramp up in results to achieve LoP targets), in totality it has met just under half its indicator targets. Furthermore, the indicators provide an incomplete view of the overall performance. This reflects that they cover only a small proportion of the potential outputs and outcomes of the tasks undertaken. Where SATIH meets targets, this reflects a range of transaction￾oriented successes in areas such as investment, exports, and grain trade. These successes, while important, have had weak contributions to the broader objectives of the activity. This reflects an insufficient systems view that led to upstream and downstream challenges reducing the scale of results, ineffective translation of transactional results into wider market systems changes, and, in some cases, the hub’s decision to undertake projects with poor additionality. Management challenges also led to a lack of prioritization and exacerbated siloed working, further reducing the relevance of SATIH's activities. STP’s performance against indicator targets has been weaker, although since the project’s turnaround in 2017 STP has made steady progress in advancing a long-term Harmonized Seed Regulation System (HSRS) agenda and appears to have strong potential to reach LoP targets. Management changes enabled this improvement—including the re-location of STP from Pretoria to Lusaka, which has help deepen crucial stakeholder relationships. The sustainability of STP’s work poses a challenge, however, with additional work needed to fully embed HSRS across Southern Africa, some of which is beyond STP’s current scope. As a cooperative agreement, TPI does not have LoP targets in the same form as the other two activities; nevertheless, TPI has earned the respect of stakeholders for the quality and usefulness of its efforts, and has largely fulfilled the scope of its work plan. Stakeholders report high application rate for knowledge gained from capacity building and policy dialogue interventions, while examples suggested TPI’s research is used in highly impactful ways (although that impact often is not tracked). However, the initiative’s effectiveness is reduced by insufficient government capacity to implement trade policy improvements; TPI’s narrow audience—predominantly government officials and researchers—compounds this challenge. These findings reflect the five evaluative themes this work explores—the relevance of work delivered, management of activities and TFSP as a whole, sustainability of outcomes (equitableness and likelihood of survival past the LoP), best practices for future programming, and opportunities. To inform this work, the evaluation team conducted 128 qualitative semi-structured interviews with 154 interviewees, reviewed 77+ project documents and additional data sources gathered in meetings with project stakeholders, received 321 survey responses, and held 5 working sessions with USAID/REGO. 1 2013 reflects when USAID signed the cooperative agreement with tralac (the implementor of TPI). Contents and tables of figures USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 2 CONTENTS AND TABLES OF FIGURES TABLE OF CONTENTS Abstract .................................................................................................................................. 1 Contents and tables of figures.............................................................................................. 2 Acronyms ............................................................................................................................... 6 Executive Summary .............................................................................................................. 9 Evaluation Purpose.............................................................................................................. 25 Project Context ................................................................................................................... 26 6.1 Background of the Trade and Food Security Project (TFSP) ...................................................................... 26 6.2 Objectives and Tasks of the Southern Africa Trade and Investment Hub (SATIH).............................. 26 6.3 Objectives and Tasks of the Seed Trade Project (STP)................................................................................ 30 6.4 Objectives and Tasks of the Trade Policy Initiative (TPI)............................................................................. 33 Evaluation design ................................................................................................................. 35 7.1 Evaluation Questions............................................................................................................................................. 35 7.2 Methodology............................................................................................................................................................ 36 7.3 Limitations to the Evaluation............................................................................................................................... 37 Findings, Conclusions and Recommendations.................................................................. 39 8.1 Southern Africa Trade and Investment Hub (SATIH)................................................................................... 39 8.1.1 Relevance.................................................................................................................................................... 41 8.1.2 Management............................................................................................................................................... 64 8.1.3 Sustainability............................................................................................................................................... 70 8.1.4 Conclusions ................................................................................................................................................ 72 8.1.5 Recommendations and opportunities.................................................................................................. 72 8.2 Seed Trade Project (STP)..................................................................................................................................... 76 8.2.1 Relevance.................................................................................................................................................... 76 8.2.2 Management............................................................................................................................................... 94 8.2.3 Sustainability............................................................................................................................................... 97 8.2.4 Conclusions ............................................................................................................................................. 101 8.2.5 Recommendations and opportunities............................................................................................... 101 8.3 Trade Policy Initiative (TPI) .............................................................................................................................. 103 8.3.1 Relevance................................................................................................................................................. 103 Contents and tables of figures USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 3 8.3.2 Management............................................................................................................................................ 114 8.3.3 Sustainability............................................................................................................................................ 116 8.3.4 Conclusions ............................................................................................................................................. 118 8.3.5 Recommendations and opportunities............................................................................................... 118 8.4 Overall Trade and Food Security Project (TFSP) Findings........................................................................ 119 8.4.1 Relevance................................................................................................................................................. 120 8.4.2 Management............................................................................................................................................ 125 8.4.3 Sustainability............................................................................................................................................ 127 8.4.4 Best Practices.......................................................................................................................................... 128 8.4.5 Conclusions ............................................................................................................................................. 129 8.4.6 Recommendations and opportunities............................................................................................... 129 Annex 1: Overview of SOW and Evaluative Questions ................................................. 131 Annex 2: Detailed Objectives and Tasks by Activity...................................................... 139 10.1 Southern Africa Trade and Investment hub (SATIH)................................................................................. 139 10.2 Seed Trade Project (STP).................................................................................................................................. 146 10.3 Trade Policy Initative (TPI) ............................................................................................................................... 151 Annex 3: Evaluation Team................................................................................................ 153 Annex 4: Reports and Data Provided to the Evaluation Team..................................... 156 Additional Annex Materials .............................................................................................. 159 Contents and tables of figures USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 4 LIST OF FIGURES Figure 1: SATIH theory of change (TOC)................................................................................................................. 27 Figure 2: STP theory of change (TOC) ...................................................................................................................... 30 Figure 3: Select seed trade project outputs by country, 2016 – 2018: Number of organizations assisted / policies drafted or reviewed ......................................................................................................................................... 32 Figure 4. Seed Trade Project annual expenditure by category, 2016 – 2018 (thousands of dollars).......... 32 Figure 5: TPI theory of change (TOC) ....................................................................................................................... 33 Figure 6: Necessary factors to enable trade and investment................................................................................ 44 Figure 7: SATIH intermediate results and project objective................................................................................. 45 Figure 8: SATIH interview codification – cross-component programing........................................................... 46 Figure 9: Follow-through by component and for SATIH overall ........................................................................ 47 Figure 10: Essential pre-conditions for the success of structured trade............................................................ 50 Figure 11: Comparison of Agribusiness Trade 2018 structured trade budget against estimated programming needs in Zambia ..................................................................................................................................... 51 Figure 12: Key constraints to Southern African trade with the U.S. .................................................................. 54 Figure 13: U.S. imports of the top 10 animal/vegetable products, USD millions............................................. 55 Figure 14: F&I Follow-through ..................................................................................................................................... 59 Figure 15: SATIH interview codification – cross-component programing ........................................................ 66 Figure 16: The Ecosystem of High-Quality Seed Usage in Sub-Saharan Africa................................................. 77 Figure 17: STP interview codification – STP value-add........................................................................................... 83 Figure 18: Percentage of respondents who applied knowledge from STP training events............................ 88 Figure 19: Comparison of STP awareness event NPS ............................................................................................ 89 Figure 20: Percentage of respondents who applied knowledge from STP awareness events....................... 90 Figure 21: Interview Analysis of STP Execution ....................................................................................................... 92 Figure 22: Survey Responses To Select Statements About STP Events............................................................. 92 Figure 23: Cyclical Process of Policy Making and Policy Implementation ........................................................ 106 Figure 24: Sectors represented by tralac’s registered readerships (%)............................................................ 108 Contents and tables of figures USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 5 Figure 25: Percentage of respondents who applied knowledge from TPI training events........................... 110 Figure 26: Percentage of respondents who applied knowledge from TPI awareness events...................... 112 Figure 27: Number of users of AGOA.info website between 2013 and 2019 ............................................... 112 Figure 28: Proportion of AGOA.info new users between 2013 and 2019 as of April 2019 by country (% of top 5 countries) ........................................................................................................................................................ 112 Figure 29: Participant perception regarding the execution of TPI training events (%)................................. 113 Figure 30: SATIH theory of change (TOC)............................................................................................................. 139 Figure 31: Agribusiness Trade programming .......................................................................................................... 140 Figure 32: Export Competitiveness programming................................................................................................. 141 Figure 33: Finance and Investment programming................................................................................................... 142 Figure 34: Trade Facilitation and Enabling Environment programming ............................................................ 144 Figure 35: Breakdown of STP capacity-building efforts to improve HSRS domestication ........................... 146 Figure 36: Relationship between the relevant SADC Seed Centre institutions............................................. 147 Figure 37: Training and awareness creation events hosted by STP, 2016 – 2018 ......................................... 148 Figure 38: STP policy advocacy efforts to improve the quality of HSRS-aligned policies............................. 149 Figure 39: Categories of work for the Trade Policy Initiative ............................................................................ 151 LIST OF TABLES Table 1: Breakdown of Interviewees.......................................................................................................................... 37 Table 2: Breakdown of survey responses.................................................................................................................. 37 Table 3: SATIH indicators.............................................................................................................................................. 43 Table 4: STP indicators................................................................................................................................................... 81 Table 5: Policies or regulations supported by STP.................................................................................................. 87 Table 6: SADC policies and regulations supported by STP................................................................................... 91 Table 7: Reporting on TPI tasks from Jan 2017 – Dec 2017............................................................................... 105 Table 8: Best Practices.................................................................................................................................................. 128 Table 9. STP policy advocacy outputs, 2016 – 2018 ............................................................................................. 149 Table 10: Strategic Partnership Fund grantees, 2016 – 2018 .............................................................................. 150 Acronyms USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 6 ACRONYMS ACTESA Alliance for Commodity Trade in Eastern and Southern Africa AEO Authorized Economic Operator AfCFTA African Continental Free Trade Area AGOA African Growth and Opportunity Act AGRA Alliance for a Green Revolution in Africa APROSE Association for the Promotion of the Seed Sector (Mozambique) B2B business-to-business BITC Botswana Investment and Trade Centre BMC Botswana Meat Commission BURS Botswana Unified Revenue Service COMESA Common Market for Eastern and Southern Africa CoP Chief of Party COR Contracting Officer’s Representative CPAR Contractor Performance Assessment Reporting DAI Development Alternatives Incorporated, Inc. DCA USAID's Development Credit Authority DNSA National Seed Department. Mozambique DUS Distinctiveness, Uniformity, and Stability EAC East African Community ECOWAS Economic Community of West African States EIB European Investment Bank FANRPAN Food, Agriculture, and Natural Resource Policy Analysis Network FSP financial service provider FtF Feed the Future GUC grants under contract HSRS Harmonized Seed Regulation System ICT information and communication technologies ITC International Trade Centre ISP Investment Support Program KPI key performance indicator ISTA International Seed Testing Association LoP life of project Acronyms USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 7 MEL monitoring, evaluation, and learning MIS market and information systems MICF Malawian Innovation Challenge Fund MITC Malawian Investment and Trade Center MoU memorandum of understanding MRA Malawi Revenue Authority MT metric tons NPPO national plant protection organization NPS Net Promoter Score NSA national seed authority NTB non-tariff barrier PAD Project Appraisal Document PE private equity PMA Produce Marketing Association PMEP Performance Monitoring and Evaluation Plan PPP public-private partnership PSE private sector engagement PWS Performance Work Statement QDS Quality Declared Seed REGO USAID Southern Africa Regional Economic Growth Office SACU Southern African Customs Union SADC Southern African Development Community SADC FANR SADC Food, Agriculture, and Natural Resources SATIH Southern Africa Trade and Investment Hub SCCI Seed Control and Certification Institute (Zambia) SDC Swiss Agency for Development and Cooperation SIDA Swedish International Development Cooperation Agency SME small and medium-sized enterprise SOW statement of work SPGRC SADC Plant Genetic Resources Centre SSA sub-Saharan Africa STAM Seed Trade Association of Malawi STTA Short-Term Technical Advisor STP Feed-the-Future Seed Trade Project TA technical assistance Acronyms USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 8 TEC total estimated cost TFSP Trade and Food Security Project TOC theory of change TPI Trade Policy Initiative tralac Trade Law Centre TRAPCA Trade Policy Training Center in Africa UCT University of Cape Town UNCTAD United Nations Conference on Trade and Development USAID United States Agency for International Development USAID/SA USAID Southern Africa Regional Office USG United States Government USIDFC United States International Development Finance Corporation VCU Value for Cultivation and Use WRS warehouse receipts system WTO/TFA World Trade Organization Trade Facilitation Agreement ZAGIS Zambia Agricultural Grain Information System ZAMACE Zambian Commodity Exchange ZASTA Zambia Seed Trade Association ZNFU Zambia National Farmers Union Executive Summary USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 9 EXECUTIVE SUMMARY This report presents findings, conclusions, and recommendations from the mid-term performance evaluation of the United States Agency for International Development’s (USAID’s) Trade and Food Security Project (TFSP), delivered through the USAID Southern Africa Regional Economic Growth Office (USAID/REGO), and through implementing partners DAI and Trade Law Centre (tralac). USAID/REGO commissioned this evaluation, which is intended to inform the design and implementation of future trade and food security work and other regional programming, and to inform any necessary mid￾course adjustments for activities under TFSP. PROJECT BACKGROUND Poverty remains one of the greatest challenges in the Southern Africa region, where approximately half of the population lives on less than $1 a day. Trade and investment can potentially play a significant role in addressing a number of factors that fuel persistent poverty: 1. Stalled economic growth – The Southern African Development Community (SADC) region has consistently failed to generate an annual economic growth rate above 5%, which is considered the threshold for making a meaningful dent in poverty. 2. Inadequate agriculture / food production – The agriculture sector has also failed to generate a sustained contribution to economic growth, achieving only 2% growth per annum. This is well below the target of 6% per annum for each SADC Member State set in the Comprehensive African Agriculture Development Program. 3. Low levels of intra-regional trade – The region is characterized by small, fragmented markets and plagued by prohibitive intra-regional trade costs, leading to the lack of competitiveness of SADC Member States’ exports An incoherent policy environment compounds these challenges by limiting the extent to which economic opportunities can be exploited. In response to these challenges, the Southern Africa region needs to focus on developing the agricultural sector, fostering intra-regional trade, and improving the policy environment for trade and agricultural growth. An increase in agricultural productivity, coupled with expanding intra-regional trade, can help bring about sustained economic growth in the region—and ultimately result in improved food and nutrition security and poverty reduction. Executive Summary USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 10 OVERVIEW OF THE TRADE AND FOOD SECURITY PROJECT (TFSP) USAID Southern Africa Regional Economic Growth Office (USAID/REGO) is supporting this agenda through the implementation of TFSP—a project with a total estimated cost 2 (TEC) of $93.4 million that seeks to increase trade and improve food security through the implementation of three activities: • The South African Trade and Investment Hub (SATIH) – a $73.7 million activity focused on expanding regional and global trade and investment through a range of interventions, including agribusiness development, improving export competitiveness, enhancing access to finance and investment, and addressing barriers to trade and gaps and obstacles in the enabling environment • The Seed Trade Project (STP) – an $18.2 million initiative focused on supporting seed policy harmonization activities at SADC regional level and in member states through policy advocacy and capacity building • The Trade Policy Initiative (TPI) – a $1.5 million co-operative agreement focused on improving the quality of trade policy in the region through research, capacity building, and policy dialogue EVALUATION APPROACH The evaluation took place between February and June 2019. The evaluation team began by conducting a task-level mapping of TFSP and developing the evaluation methodology. Following this inception phase, the team embarked on extensive data gathering, interviews, and analysis, all of which form the basis of this evaluation report. The evaluation methodology draws on a broad range of source material to provide a strong fact base. The evaluation team used semi-structured interviews and surveys to gather extensive new insights. Interviews were codified to provide a quantitative base for analysis while survey results helped the team quantify what were typically more qualitative data points. The evaluation team challenged these insights through corroboration with a range of other sources—including project documentation, research and literature reviews, analyses of additional data sources gathered in country, and reviews of performance monitoring and evaluation plan (PMEP) data where provided by implementing partners. FINDINGS AND CONCLUSIONS This report presents the evaluation team’s findings across evaluative themes, set out below. The report fully explores these themes for each of the three activities, and again at the project level. RELEVANCE The assessment of relevance focused on the overall performance of each activity, how ecosystem challenges impacted that performance, the effectiveness of the tasks within the activities (including a consideration of whether or not the optimal tasks were selected), and how well those tasks were executed. 2 The total estimated cost (TEC) of an activity acts as a spending cap, and not a commitment to spend the full amount. Within the life of program of TFSP, there have been budget cuts that have impacted SATIH, which means that the full TEC is unlikely to be reached. Executive Summary USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 11 In reflecting on the common trends across activities, TFSP’s activities combine transaction-oriented approaches with longer-term policy work that contributes to activities addressing a relevant set of challenges. However, insufficient understanding of the landscape and ecosystem led to upstream and downstream challenges playing a role in hindering effectiveness. This reflects that TFSP took on activities without fully considering the other prerequisite conditions that need to be in place, which reduced TFSP’s potential to deliver outcomes, as the assessment for each activity will illustrate. The overall relevance of TFSP benefits from its use of Private Sector Engagement approaches, while its regional nature has allowed it to add value in ways that the bilateral missions alone cannot—e.g., by acting as a single face to multi-country stakeholders and creating regional linkages, advancing strategies that require both regional and national components such as market information systems or seed policy harmonization, and providing expertise on region-wide issues. That said, there were missed opportunities to exploit the regional model, for example to have served as a single face to regional actors working across multiple countries (in contrast to working with national actors, or with regional actors in the context of a single country only). Moving to consider each activity and its relevance, SATIH’s scope is the broadest of the three activities. Although it has met 10 of 22 Y1 and Y2 indicator targets, particularly those around exports and investments, there are a range of indicators SATIH has missed, while the indicators themselves provide an insufficient view of performance. These successes reflect the transaction-oriented approach mentioned above and, to a lesser degree, focused policy work under the Trade Facilitation component—which together helped SATIH address a set of challenges that businesses identified as relevant. That said, the indicators provide an insufficiently clear view of performance because they only cover a small proportion of the potential outputs and outcomes of the tasks undertaken, and so do not measure the broader performance against SATIH’s overall objectives. This reflects an issue in the design and set-up of SATIH, which became a greater issue as a result of execution challenges. An insufficient systems view led to unaddressed ecosystem challenges hindering the effectiveness of results (e.g., trade linkage work that has been less successful than it could have been owing to lack of access to trade finance or inability of target businesses to scale to meet the demand). SATIH has often delivered transactional results without driving more systemic change (e.g., driving exports for trade-ready businesses, while wider competitiveness challenges remain unaddressed). At the same time, some aspects of the program do not provide significant additionality (e.g., large investment deals considered a result of SATIH’s efforts that would likely have taken place in the absence of SATIH’s support). It is also important to note that the need for significant scale-up in the final years was implicit in these targets from the start. For example, SATIH is currently meeting 128% of its combined 2017/2018 export facilitation target, but is at 16% of the LOP target for this indicator. Given limited data available (as outlined in the limitations to this evaluation), the evaluation has been unable to fully assess whether this scale up is likely to be delivered. STP, meanwhile, addresses a more focused challenge: Harmonized Seed Regulation System (HSRS) implementation – consisting of both policy change and capacity building work. Although STP performed poorly in its initial years (that is, it did not meet indicator targets), performance has improved markedly following the reset of STP in 2017. The project now seems to be on target to achieve LoP goals; Performance has, however, been stronger at the country level than at SADC / regional level. The work is by all indications highly relevant. Seed policy harmonization plays a crucial role in improving access to quality seed, and is an issue that USAID/REGO is well positioned to advance given the need for policy change at both national and regional levels. STP has pursued the appropriate strategies and tasks to drive HSRS implementation, which have largely been well executed. The challenge faced by STP is that although it is performing well in its focus areas, there are gaps in coverage that limit its effectiveness—for example, Executive Summary USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 12 importing countries will need support in advancing the adoption of HSRS, and governments will need greater levels of financial support in order to fully implement harmonized policies. TPI’s relevance is reduced by other factors in the policy ecosystem. Although TPI focuses on agenda setting at the earlier stages of the policy cycle, its effectiveness is limited without there being sufficient implementation capacity in the system to drive policy change. Although this is not the remit of TPI, stakeholders frequently identified the lack of implementation capacity as an issue. This, combined with an audience comprising mostly of government officials and researchers, limits the potential results TPI can achieve. That said, the quality of its work is well regarded—participants report high utilization of knowledge gained at capacity building and policy dialogue events, while research has been used in highly impactful (if not rigorously tracked) ways by the users with whom the evaluation team engaged. The following pages provide a breakdown of the three activities performance against indicator targets. Executive Summary USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 13 Table 1: SATIH indicators3 Indicator Output/ outcome Absolute value achieved Progress against 2017/ 2018 targets Progress against LoP Target Value of exports in targeted non-agricultural and agricultural commodities (ACTE DO1.1) (EG 3.2-23) Outcome $ 25,541,719 128% 16% Value of new private sector investment in agricultural sector (ACTE DO I.2); (FTF 4.5.2-38); (EG 3.2-22) Outcome $ 83, 074, 147 346% 65% Value of new private sector investment in non￾agricultural targeted sectors leveraged by the Hub (ACTE DO I.3) Outcome $ 32, 734, 760 205% 36% Number of metric tons of grain traded through commodity exchanges / certified warehouses supported with Trade Hub assistance Outcome 77,174 1286% 107% Number of certified warehouses participating in commodity exchanges and/or warehouse receipt systems with trade hub assistance Outcome 85 2125% 100% Metric tons of installed storage capacity Outcome 82,642 55% 17% Number of for-profit non-agricultural sector private enterprises that applied improved organizational-level technologies or management practices as a result of USG assistance (ACTE IR 2.1) (FTF 4.5.2-42); (EG 3.2- 20) Outcome 58 153% 3% Number of buyer / seller linkages established in targeted non-agricultural sectors as a result of Trade Hub assistance (ACTE IR 2.2) Output 623 88% 6% Number of assisted non-agricultural sector firms / associations meeting international grades and standards to export (ACTE IR 2.2) Outcome 2 67% 40% Number of for-profit private enterprises that applied improved organizational-level technologies or management practices as a result of USG assistance (ACTE IR 1.1) (FTF 4.5.2-42); (EG3.2-20) Outcome 12 19% 4% Number of buyer / seller linkages established in targeted agricultural sectors as a result of Trade Hub assistance. (ACTE IR 1.2) Output 122 51% 7% Number of assisted agricultural sector firms / associations meeting international grades and standards to export (ACTE IR 1.3) Outcome 6 120% 20% Value of loans to target enterprises / groups due to Trade Hub assistance. (EG 3.2-6) Outcome $ 16,810,533 168% 13% Number of food security private enterprises, POs, WUAs, women’s groups, trade and business associations & CBOs receiving USG assistance (EG 3.2- 4) Output 650 445% 67% Number of PPPs formed with hub assistance (EG 3.2-5) Output 17 89% 38% Number of enabling environment policies analyzed, consulted on, drafted or revised, approved, and implemented with USG assistance (ACTE IR 1.3) (FTF 4.5.1-24); (EG 3.1-12) Outcome 23 460% 47% Number of organizations assisted to reduce non-tariff barrier (NTB), which are relevant to the SADC region Output 4 36% 0% Person hours of USG-supported training completed in trade and investment (EG.2-1) Output 799 14% 6% 3 Note: this reflects 18 of the 22 non-context top-level indicators. Difficulties have been encountered in reporting against the remaining four (time to trade, cost to trade, firms experiencing increased profitability, jobs created) Executive Summary USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 14 4 STP documentation did not classify the indicators as output vs. outcome indicators in the way SATIH did; given this, the classification here reflects the view of the evaluation team. Table 2: STP indicators Indicator Output/ outcome 4 Absolute value achieved Progress against 2016/ 2017/ 2018 targets Progress against LoP Target EG.3.2-20: Number of for-profit private enterprises, producer organizations, water user associations, women’s groups, trade and business associations and community based organizations (CBOs) that applied improved organization-level technologies or management practices with USG assistance Output 54 600% 1080% EG.3.1-12: Number of agricultural and nutrition enabling environment policies analyzed, consulted on, drafted or revised, approved and implemented with USG assistance Outcome 11 550% 367% 4.5-Z03: Number of seed varieties in distribution from the regional HSR systems. Outcome 20 1000% 100% EG.3.2-4 Number of for-profit private enterprises, producer organizations, water user associations, women’s groups, trade and business associations, and community based organizations (CBOs) receiving USG food security related organizational development assistance Output 26 100% 65% EG.3.2-1: Number of individuals who have received USG supported short-term agricultural sector productivity or food security training Output 1,047 98% 63% 4.5-Z04: Number of seed companies engaged in regional seed varietal release Outcome 6 150% 60% EG.5.2-1: Number of firms receiving USG funded technical assistance for improving business performance Output 54 95% 45% 4.5-Z07: Value of grants distributed by the Seed Trade Project Output $415,071 45% 28% EG.3.2-22: Value of new private sector investment in the agriculture sector or food chain leveraged by Feed the Future implementation Outcome $100,000 40% 27% 4.5-Z05: Number of technologies or management practices transferred as a result of USG assistance Output 2 29% 20% EG.3.2-5: Number of public-private partnerships formed as a result of Feed the Future assistance Output 2 22% 14% 4.5-Z06: Quantity of high quality seed traded under HSR by crop Outcome 0 0% 0% EG.3.2-23: Value of targeted agricultural commodities exported with USG assistance Outcome $0 0% 0% 4.5-Z01: Sustainable business model for regional seed variety release; certification and quality assurance operational Output 0 Targeted 0 to date 0% 4.5-Z02: Percent of regional HSR administrative system costs covered by revenues generated Outcome 0% 0% 0% Executive Summary USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 15 Table 3: Reporting on TPI tasks from Jan 2017 – Dec 2017 Activity 2017 Workplan Target Reported in 2017 M&E matrix 2017 Quarterly report tracking Jan - Mar Apr - Jun Jul - Sep Oct - Dec 2017 Total Trade Policy Monitoring and Analysis (Research) Newsletters/ daily news updates 40 Not reported in M&E matrix Not reported Not reported 72 67 134 Discussion notes 25 Not reported in M&E matrix Not reported 19 16 20 22 Data updates 40 Not reported in M&E matrix Not reported Not reported 5 Not reported 4 Trade briefs 10 8 2 5 2 1 10 Working papers 8 4 1 2 0 6 9 Training (Capacity Building) Masters degree scholarships 10 Not reported in M&E matrix 0 0 0 0 0 Short courses - Trade remedies and safeguards, Standards (SPS and TBT), Rules of origin, Trade policy development for the 21st century; Regional community law 1 Not reported in M&E matrix 0 0 0 0 0 Other workshops/trainings (Intellectual property and SACU workshop) 0 Not reported in M&E matrix 0 0 1 1 2 Interns 2 Not reported in M&E matrix 0 0 3 0 3 Policy dialogue Annual conference participants (co￾funded event) 40 Not reported in M&E matrix 0 94 0 0 94 AGAO workshops (reported under capacity building in this evaluation) 4 Not reported in M&E matrix 0 0 0 1 1 Interns 2 Not reported in M&E matrix 0 0 2 0 2 Executive Summary USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 16 MANAGEMENT The management evaluative theme looked at the role of contract structures and management models (including organizational structures and processes, systems, interfaces, and key personnel) in helping or hindering the performance and relevance of the project. The evaluation team considered these factors within both USAID/REGO and the implementing partners (DAI for SATIH and STP and tralac for TPI). TFSP at project level has played a limited management role; the three activities have largely been run independently, with efforts to seek out synergies limited mostly to the development of a joint PAD during the inception of the project. The evaluation team did identify missed opportunities for USAID/REGO to add value at the project level—for example, by leveraging common relationships or building off the programming of other activities (e.g., TPI’s agenda setting work aligning with the priorities of SATIH and STP, and vice versa). That said, the evaluation team did not perceive these synergies to be of significant value, but USAID/REGO or its implementers could realize these without significant effort. At the activity level, both SATIH and STP have faced management challenges within the first half of the LoP. STP took steps earlier to address these challenges; SATIH currently is taking corrective action. In the case of SATIH, these challenges consisted of poor prioritization processes (with strong corresponding impacts on relevance), significant leadership overhaul, and slow reactions to major changes in SATIH’s context (including budget cuts on the part of the implementer). On the USAID/REGO side, the breadth of SATIH’s scope led to competing priorities within USAID/REGO that added complexity to management. Challenges on both the implementer and USAID/REGO sides led to diseconomies of scale predominately at the senior level; with the four components programming largely independently, this generated a greater volume of operational approvals at the COR and COP level, whereas an approach where the components built on each other’s successes and relationships could have alternatively created greater synergies. Furthermore, the contract structure (including the Performance Work Statement) provided a high level of flexibility to the implementer and a broad set of target outcomes, without creating strong management tools for USAID to hold the implementer to account beyond the Contractor Performance Assessment Reporting (CPAR) system (with challenges including the absence of fees at risk for the implementer, and gaps in the quality of reporting and tracking mechanisms). STP faced similar management issues in the early part of the LoP—although decisive action on the part of USAID/REGO, including calling for a performance review outside of the usual cadence for the project, led the implementer to take early corrective action. In the wake of this, management of STP has shown significant improvement, which rapid progress in 2018 toward indicator targets reflects. A key move was the relocation of the activity from Pretoria to Lusaka. This, coupled with strong country advisors and the CoP working closely with SADC institutions, has led to significantly stronger stakeholder relationships. These relationships, given the policy-oriented nature of the work, have been critical to achieving better results. The management of TPI has been more consistent throughout the LoP. TPI is structured in a distinct manner to SATIH and STP under a cooperative agreement. With this, the work plan approval process allows sufficient control on the part of USAID/REGO to influence TPI’s activities, while the cooperative agreement also allows TPI a sufficient measure of independence to pursue a rigorous research and policy agenda without introducing U.S. bias. TPI’s management elicits strong positive feedback from stakeholders, earning high praise for the credibility and expertise of the TPI team. Executive Summary USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 17 SUSTAINABILITY Common themes have emerged at the project level around sustainability – both around the ability of results to be sustained beyond the LoP, and the equity of results. The use of private sector approaches has, in some cases, produced results that appear to be sustainable—where firm-level support to businesses will continue to have an impact beyond the LoP providing the firm continues to operate. From an inclusiveness perspective, across the project there was a systematic lack of a deliberate inclusion or gender strategy – with only pockets of gender-focused activities identified. On the activity level, SATIH’s repertoire of interventions varies considerably in its degree of sustainability. Transaction-oriented successes have played the role of a “proof of concept,” encouraging self-sufficient investment once the business case is proven (e.g., as was the strategy for sending businesses to trade fairs, with the view that if they proved successful it would encourage businesses to self-fund their attendance in future). That said, frequently SATIH has pulled back from larger-scale interventions too soon, before they are sufficiently developed for local actors to take them on as sustainable models (e.g., as was the case with Zambia Agricultural Grain Information System (ZAGIS), where a sustainable business model had been identified, but more support was needed but not provided to move the project to a point local stakeholders would continue to advance it). In addition, SATIH pursued large-scale interventions that had weak prospects for sustainability, often owing to other ecosystem challenges that limit the potential for these to continue without donor funding (e.g., the high transaction costs associated with smaller size investments means that many of these have no path to viability in the absence of donor subsidies). STP’s sustainability hinges on the sustainability of both regional and national-level policy and institutional change. In the case of SADC, there continues to be a lengthy path to a sustainable SADC Seed Centre. Although a business model has been defined, it will take time to scale this up; meanwhile, the SADC Seed Centre is not yet embedded into the SADC structure. While country-level policy changes are by their nature likely to be permanent and sustained, many countries struggle to find and sustain funding for the new capacity that is needed to execute on-the-ground changes laid out in new policy. With TPI, the fact that USAID is a minority funder of tralac means that tralac would continue to exist, and therefore a similar set of activities would continue to operate in the absence of USAID’s support— albeit at a smaller scale, particularly in relation to African Growth and Opportunity Act (AGOA) activities (for which USAID is the sole funder). From an inclusiveness perspective, interventions and their audience are biased toward male participants, although TPI is currently taking steps to correct that imbalance. BEST PRACTICES The evaluation team identified several best practices within the project that can inform future programming: • Be deliberate and focused – the broad mandate of SATIH resulted in a diverse set of programs that did not build on each other, whereas a more focused mandate helped both STP and TPI to deliver results and gave USAID/REGO a clearer view into progress • Prioritize geographic proximity – in-country presence played a critical role in the case of STP, allowing the project to form strong local relationships and provide consistent follow-up, whereas a mostly Pretoria-based team within SATIH struggled to cover a broad geographic range • Co-create solutions – STP noted that close collaboration with the missions provided local context that was crucial to task design Executive Summary USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 18 • Draw on a balanced portfolio – of both long-term and transaction-oriented work. This has enabled SATIH to deliver transaction-oriented successes and build credibility while also pursuing longer-term policy changes • Align with country priorities – work that aligned with country priorities tended to benefit from a multiplier effect, wherein local actors provided additional capacity; meanwhile, where TFSP pursued work that was not prioritized by local actors it often suffered from a lack of implementation capacity RECOMMENDATIONS AND OPPORTUNITIES Based on the evaluative findings outlined here, there are a number of key opportunities that the project as a whole should consider (activity-specific recommendations appear within the full report). These recommendations apply to more than one of the three activities, while more detailed recommendations (which often contextualize these top level themes) are provided for each of the activities. 1. Conduct further systems analysis and re-align programming accordingly – with a clear understanding of how upstream / downstream challenges impact effectiveness and the extent to which particular challenges in the system can be resolved 2. Develop a clear partnership strategy – aligning with and building off the work of other development partners, and ensuring greater alignment with the priorities of local actors that can act as a force multiplier for TFSP 3. Refine contract structures and contract management set-up – for example, by clarifying desired outcomes, strengthening monitoring mechanisms, and putting in place tools to hold the implementer to account (including increased emphasis on tying performance-based contracts to fees at risk) 4. Re-align indicator frameworks – to ensure both that outcome indicators provide a sufficient perspective on the success of the activities, and that output indicators tie fully to outcomes and serve as a useful management tool / ‘early warning system’ 5. Create greater alignment across activities – either through increasing USAID resources to fund coordination at the TFSP level, or through incentivizing coordination across activities by embedding relevant metrics into the incentive framework 6. Build a clear inclusion strategy – ensuring that all elements of the project consistently embed USAID priorities around gender and inclusion into programming 7. Build on future advances in the USG ecosystem – bringing the tools of organizations such as the United States International Development Finance Corporation to bear (e.g., as part of SATIH’s Finance and Investment component, or within STP to meet countries’ funding needs) Further recommendations have also been provided by component area, which are outlined below. Southern Africa Trade and Investment Hub (SATIH) recommendations Within the remaining Life of Project (LoP): Cross-cutting Relevance • Continue to build a matrixed way of working within SATIH and more connected programming across components – Enable interventions that bring to bear the tools of the various component teams • Improve relationship and knowledge management / stakeholder mapping – Ensure that SATIH has a clear ‘rolodex’ of its stakeholders, re-prioritize these as the project enters its second Executive Summary USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 19 half, and use this insight to inform prioritization of activities (connecting to the previous opportunity around bringing the full set of tools to more a more focused set of beneficiaries) • Continue re-prioritization of tasks, considering both the criteria identified in the 2019 work plan and ecosystem and landscape mapping – De-prioritize tasks that are not feasible due to wider systemic challenges that SATIH cannot resolve with available resources, or explore alternative approaches (e.g., partnering with other donors that are working in adjacent spaces) where other gaps need to be addressed to enable SATIH’s activities • Refine prioritization processes / tools – Ensure that activities align with USAID’s desired outcomes; for example, consider scaling the use of “fit check” criteria across the activity (e.g., criteria on additionality or gender inclusivity), and introduce “deal breaker” criteria into these tools where necessary to improve their effectiveness • Drive proactive donor and bilateral coordination – Ensure that this effort is informed by deeper stakeholder and landscape mapping, particularly where donors are working in interdependent areas that SATIH’s work does not fully address • Revise the theory of change to ensure strategic focus – Re-develop intermediate results and objectives to provide clear strategic guidance (e.g., what relative focus is desired within a specific component where various tools are at SATIH’s disposal) • Refine the indicator framework – Ensure that outcomes are defined in a way that provides clarity on what SATIH is expected to deliver; develop a more robust framework that ties output indicators (used predominately for management rather than outward reporting) to outcomes so that they serve as an improved “early warning system”. Ensure that the indicator framework places a tighter burden of proof on additionality • Be more deliberate about gender – Develop explicit gender goals and targets and track gender disaggregated data to ensure that goals are being achieved; look to integrate a gender lens in transaction support through—for example, by taking a broader view of gender lens investing using 2X challenge criteria, 5 or by focusing on value chains and sectors that have a disproportionate representation of women (while acknowledging that interventions with higher additionality are often harder to implement; with this, there may be a trade off on ‘quantity’ of results while delivering greater ‘quality’) • Elevate the role of MEL in work-planning – Ensure that the MEL team contributes to technical design, particularly in ensuring deliberate discussion on whether activities are consistent with the broader theory of change Agribusiness Trade • Develop a clear “where to play” vision for Agribusiness Trade – Conduct further scoping to provide much needed focus given the breadth of agribusiness activities in the region, the current budget, and the remaining LoP duration; conduct landscaping and stakeholder mapping as required to inform this exercise • Review structured trade engagement – In light of extensive upstream and downstream challenges, including endemic political economy challenges that require long-term solutions beyond the LoP, review the extent to which SATIH can address these challenges, and how it may fit into the structured trade ecosystem in this context • Coordinate effectively – Coordinating strongly with local actors to identify specific parts of an issue that USAID can play a unique role in advancing—particularly when addressing challenges such as structured trade, which one actor alone cannot resolve; this will ensure that programming is not duplicative with the work of local actors, and that USAID/REGO funding is focused on areas that it is uniquely positioned to address effectively 5 www.2xchallenge.org Executive Summary USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 20 Export Competitiveness AGOA work to date has largely focused on developing AGOA strategies. In light of the structural competitiveness constraints identified in the relevance section of this report, there is a need to consider how much resource SATIH is prepared to invest in continuing to drive AGOA success. Ultimately, any approach for pursuing AGOA trade will likely be costly in the face of these competitiveness challenges. Should AGOA remain a key priority, this report sets out two options for consideration: • Advance the implementation of AGOA national strategies as a private sector strengthening initiative (high investment) – Invest in using the AGOA national strategies as a broader private sector strengthening initiative; this approach likely requires industrial development investments, strengthening of business associations, courting foreign direct investment, and lending capacity to governments to support the ongoing implementation of national strategies. It is important to note that should SATIH attempt to pursue this option, significant investment in implementation support would likely be needed – meaning either more funding or a reprioritization from other activities. • Pursue a focused AGOA trade campaign with AGOA-ready businesses (low investment) – Drive AGOA trade by identifying and picking a smaller set of businesses that are already well set up to trade with the U.S. (likely a relatively small segment of businesses in the region, which may be difficult to source). Doing this effectively likely requires a strong sourcing effort, likely using a “fit-check” type set of criteria in the manner that Finance and Investment do at present to prioritize provision of support to the businesses most likely to deliver AGOA trade Finance and Investment • Re-evaluate desired outcomes of transactions and corresponding fit-check criteria – Although large transactions deliver strong market signaling benefits, their size will often yield lower additionality as investors are more likely to be able to cover transaction costs; define the relative desired focus and adjust fit-check criteria and targets accordingly (e.g., introducing a deal￾breaker criteria on additionality if desired, which would improve additionality but reduce potential value. Place greater emphasis on the complexity of the business itself, which also determines whether investors could conduct due diligence internally.) • Explore use of co-funding mechanisms, particularly for larger deals – For example, where large deals bring a benefit to SATIH (particularly in signaling and driving market systems change) but the costs of transaction support could be sustained by the investor, consider a co￾funding mechanism by which costs are shared between SATIH and the investor • Evaluate relative programming focus based on ecosystem needs / priorities – Evaluate whether the current programming focus areas make sense against broader regional challenges in finance and investment, and whether the current tools are best suited to addressing them (e.g., is the allocation of effort toward supporting private equity and commodity lending in line with both the market needs and USAID appetite?) • Use the Development Credit Authority (DCA) facility as one tool, alongside good product design – Recognizing the limited faith stakeholders had in the DCA as a tool on its own, consider a more focused pursuit of DCA opportunities—combining DCA with deeper product design work that alters the risk profile for banks; although this may yield fewer FSP relationships, given the perception of the DCA as a standalone tool it may have potential to improve total lending under SATIH Executive Summary USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 21 Trade Facilitation and Enabling Environment • Define a clear set of priority areas where SATIH resources can make a significant difference in reaching time and cost to trade targets: In the context of current project timelines and budget, define specific areas where there is potential to improve time and cost to trade—recognizing that SATIH cannot address full region dynamics; prioritization could take place against a sub-set of corridors, border posts, value chains, or specific issues (e.g., standards or types of technological solutions) • Draw on the expertise of the private sector to inform prioritization – Understand the issues that are truly holding them back from trade and impacting their profitability (e.g., building closer relationships with anchor firms or industry associations that can provide a broad perspective on private sector challenges) • Use adaptive programming approaches and build an evidence base – Particularly when working in specific value chains, combine more adaptive programming approaches with desk research; in the absence, to date, of rich SATIH-specific data on which interventions are most effective, this approach could enable rapid progress and inform future programming beyond the LoP Beyond the LoP: • Align on the future focus and targets of SATIH – Create a more focused set of priorities within the TOC that aligns with the budget at SATIH’s disposal • Draw on USAID’s full suite of tools – For example, explore opportunities to work with new USG initiatives such as the United States Development Finance Corporation to bring a fuller set of tools to programming • Link fees to performance if continuing with performance-based contract structures – Performance-based contracts are powerful tools—putting a proportion of fees at risk can create joint accountability for the implementer and additional measures for USAID to ensure strong performance. To effectively leverage this model, outcomes need to be well designed, indicators need to accurately reflect these outcomes, and indicator targets must be well contextualized (which will likely require the input of the selected implementer, along with USAID design capacity) • Integrate data collection and management into project design – Ensure better transparency and improve MEL efficiencies by developing and using data collection tools that gather MEL data on an ongoing basis throughout task implementation. Seed Trade Project (STP) recommendations During the remaining LoP STP strategy • Conduct a systems analysis of the regional seed sector – Systematically identify the most critical issues in the ecosystem that impact upon STP’s effectiveness, as well as areas where STP might collaborate with others. Such collaboration will also support STP’s sustainability (e.g., it may ensure that programs focusing on provision of quality inputs are minimizing the impact of nutrient mining). • In upcoming events, explore means of expanding the geographic reach of the project’s beneficiation to include more import-dominant countries (e.g., Angola) – For example, have more regional trainings that allow analysts and inspectors from more regional Executive Summary USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 22 countries to attend, STP could also support improved capacity to implement HSRS in those countries. • Support National Seed Authorities (NSAs) and national plant protection organizations (NPPOs) in meeting structural infrastructure funding needs – Prioritize supporting NSAs and NPPOs in acquiring “hard” infrastructure such as laboratories, testing equipment, and vehicles . Where STP cannot finance this itself, consider partnering with DFIs and other organizations (e.g., the African Development Bank in Malawi) that are potentially willing and able to fund the infrastructure investment that STP cannot. STP Execution • Consider expedited protocols for rules related to expenditure, travel, and communications – Potentially work with the implementer to devise expedited processes for certain protocols given the regularity and extent of stakeholder interactions involved with implementing STP and frequent need for short-notice travel. Such expedited processes could still ensure the protection of USAID/REGO against financial and reputational risk while giving STP’s implementers more flexibility, reducing stakeholder complaints. For example, consider setting out clear guidelines for where USAID/REGO will attempt to accommodate short-notice travel requests (e.g., when the meeting is with an individual of a minimum level of seniority) Policy advocacy: • Provide needs-driven support to NSAs and NPPOs to enable implementation of STP￾supported policies – Ensure that more of the policies STP has supported are implemented by supporting NSAs and NPPOs in addressing specific constraints they have to implementing this (e.g., helping Mozambique NPPO to conduct sensitization of the country’s revised pest list). Training events: • Ensure that training includes sufficient coverage and time for demonstrations and participant exercises – Increase the likelihood that participants retain what they have been taught, and that they will apply the material when they return to their laboratories, testing fields, and border posts. Awareness events: • Put more care into the facilitation of STP awareness events - Work closely with seed trade associations as the grantees tasked with hosting STP awareness events to carefully select facilitators, coordinate their presentations, and consider additional presentation tools such as interpreters. • Provide regular updates to awareness event participants regarding HSRS progress in their countries and the region – Stay connected to past participants to advance building awareness about HSRS and give them information with which to lobby for HSRS implementation Executive Summary USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 23 Beyond the LoP • Include other significant seed import markets within the scope for future HSRS implementation support – Potentially include countries like Tanzania, the DRC, and Angola to drive regional harmonization further and promote project sustainability • Strengthen commitment to operationalizing the SADC Seed Centre through deeper technical assistance – Either fund a consultant with a limited scope of work (centered on implementation of the regional variety release system – the core of the Centre’s business model) or give the SADC Seed Centre coordinator (once appointed) an additional grant that can only be used toward the regional variety release system. Trade Policy Initiative (TPI) recommendations • Take steps to reinforce TPI’s efforts by addressing implementation capacity gaps – There are multiple options for doing so, depending on the level of ambition to scale TPI’s efforts: o Drive alignment across TFSP activities – e.g., ensure that SATIH builds off the agenda-setting work of TPI to improve implementation, and vice versa o Partner and coordinate with other implementing partners – ensure TPI’s efforts align with the efforts of other donors supporting trade facilitation implementation o Increase funding of implementation activities – At the same time, potentially bring in additional implementers to provide expertise in these areas in a manner that complements the agenda-setting work of TPI • Expand audience reach to ensure that TPI interventions benefit more stakeholders – To do this, TPI and/or USAID could: o Re-focus activities to include other high-potential audiences that can tangibly drive policy outcomes and are less susceptible to the implementation capacity challenges of government ministries; for example, build on the successes of work with journalists and COMESA, or consider catering to private sector audiences o Offer trainings and research materials in French and Portuguese – interviewees expressed although TPI’s work is valuable for the continent, the use of English makes the content inaccessible to franco- and lusophone stakeholders • Build greater awareness of TPI within USAID missions – in service of greater engagement with TPI content, both directly within USAID and the ability of bilateral missions to make others aware of it. This could be achieved through roadshows (as SATIH and STP have used), or including USAID bilateral in key events such as the annual tralac conference • Improve indicator design and tracking mechanisms to capture the effectiveness and value of TPI interventions – with a particular focus on proving the outcomes, and not just outputs. With this, provide greater clarity on the specific outcomes USAID/REGO desires within the project’s goal of improving the quality of trade policy in the region • Identify opportunities to introduce co-funding mechanisms where participants contribute to costs – enabling TPI funds to support more stakeholders or interventions. This recognizes that a considerable number of event participants expressed a willingness to pay to attend TPI events (although not all may be able to financially) • Continue to support and expand initiatives such as the tralac women’s network – Improve the equity of TPI’s audience, which is currently weighted towards a male audience, by encouraging greater female participation Executive Summary USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 24 READERS GUIDE TO THE FULL REPORT The full report includes significant detail around the activities and performance of the three initiatives. This reflects the diverse range of audiences for this report, each likely to focus on specific elements of it. There has therefore been an effort to ensure that the report contains standalone sections that can be understood in isolation. Indeed, each section is designed to be read as a self-contained activity-level evaluation, recognizing that many of the stakeholders involved in the project have only been involved in one of the activities, and in many cases only in a specific sub-component of that activity. Particular elements of the report can be found at the following pages: • Summary findings: beyond the overall executive summary, which reflects the whole of PAD evaluation, summaries are provided at the start of each section covering the findings for: o SATIH – pages 39 – 40 o STP – page 76 o TPI – page 103 o Project-level (TFSP) findings – pages 119 – 128 • Recommendations: are provided with greater specificity for each activity, and for TFSP at project-level, covering: o SATIH – pages 72 – 75 o STP – pages 101 – 102 o TPI – page 118 o TFSP as a whole – pages 129 – 130 Evaluation Purpose USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 25 EVALUATION PURPOSE USAID-REGO procured the services of the Investment Support Program (ISP) to conduct an evaluation into the Trade and Food Security Project (TFSP). This evaluation aims to provide USAID and its partners with an independent assessment of the project’s performance, highlighting successes and areas for improvement, that can be applied to further regional programming and programming around trade and investment. In conducting the evaluation, the aim is not just to describe what works and what does not, but also to probe more deeply into root causes. The evaluative questions are shaped accordingly. This ensures that insights and recommendations are action-oriented, ultimately informing ongoing project activities, and the future Project Appraisal Document (PAD) and project setup. Where possible, the evaluation team identified common patterns of success or failure; these types of insights can shape guardrails of future design efforts. The evaluation team conducted this evaluation by considering these five overarching themes: 1. Relevance: To what extent has the Project been successful in achieving results that meet its stated objectives? This looks specifically at: o The outcomes that the interventions delivered o How these interventions fit into the wider ecosystem and the impact that ecosystem factors have on the potential of the activity to deliver intended results o The effectiveness of the interventions, in terms of their additionality (would they have occurred in the absence of USAID support), and how well-considered the strategies were in their potential to deliver intended results o The quality of execution associated with the intervention 2. Management: To what extent has the current management structure (across the USAID Southern Africa Regional Economic Growth Office (USAID/REGO) and the implementer) supported or hindered performance of project activities? 3. Sustainability: What mechanisms have USAID/REGO or the implementing partners put in place to ensure the sustainability of the project’s achievements? This covers: o The ability of the project to deliver long-term impact beyond the life of the project (LoP) o The extent to which the impact delivered is equitable in its nature - which reflects the ability to deliver long-term optimal outcomes that achieve USAID’s desired goals, while avoiding exacerbation of existing inequalities 4. Best Practices: Based on the findings and analysis of the preceding questions, what are the key strategic, programmatic, technical, and managerial features of the Project that USAID/REGO and implementers should take into account when implementing further trade and food security programming in the region? 5. Opportunities: What current opportunities does the project face? This evaluative theme outlines the recommendations from the evaluation team, which are reflective of the conclusions drawn across the remaining four evaluative themes The evaluation team identified best practices at the project level (i.e., identified best practices for TFSP as a whole, informed by the evaluation of the three activities). The evaluation team explored the other four evaluative themes for each activity independently, as well as at project level. Project Context USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 26 PROJECT CONTEXT 6.1 BACKGROUND OF THE TRADE AND FOOD SECURITY PROJECT (TFSP) USAID/REGO is implementing the Trade and Food Security Project to reduce poverty in Southern Africa by increasing agricultural productivity and regional trade. To achieve this, USAID/REGO has implemented three activities under TFSP: 1. Southern Africa Trade and Investment Hub (SATIH) 2. Feed-the-Future Seed Trade Project (STP) 3. Trade Policy Initiative (TPI) As part of the inception phase of this project, the evaluation team conducted a detailed activity mapping to inform the evaluation design. The section below sets out the descriptions developed in the inception phase. A summary level view is provided in this section of the report (Section 6), while ‘Annex 2: Detailed Objectives and Tasks by Activity’ sets out more detail by component. Note that the TOCs the report presents below are based on existing project documents, hence their use of different nomenclature to refer to different levels of the TOC. 6.2 OBJECTIVES AND TASKS OF THE SOUTHERN AFRICA TRADE AND INVESTMENT HUB (SATIH) Historically REGO’s flagship activity, SATIH was launched in October 2016 at a total estimated cost of $73 million. 6 REGO awarded the fourth-generation iteration of trade and investment programming to a new implementing partner, DAI Inc (DAI), through a competitive bidding process. Over the two years since its launch, the fourth generation of SATIH has undergone several changes. Central to this shift is a model that focuses on the private sector. This private sector model can be seen in the inclusion of and focus on a Finance and Investment component, as well as the decreased prioritization of regional Trade Facilitation and Enabling Environment efforts in partnership with the Southern African Development Community (SADC). As outlined in Figure 1, SATIH aims to create a more integrated Southern African region with improved quality of life, driven by expanding regional and global trade and investment. 6 The total estimated cost (TEC) of the activity acts as a spending cap. It is not guaranteed that this will be spent over the activity lifetime. Project Context USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 27 Figure 1: SATIH theory of change (TOC) Source: DAI, SATIH Performance and Monitoring and Evaluation Plan, 2017 SATIH has two key objectives over the course of its five-year programming period. These feed into a much higher-level objective of expanding regional and global trade and investment. The two objectives are increasing the competitiveness of target value chains (intermediate objective 1, or IO 1) and improving the enabling environment for regional trade and investment (intermediate objective 2, or IO 2). Two intermediate results (IRs) feed each objective, as follows: • Increasing the competitiveness of target value chains (IO 1): o Intermediate result 1: Increase intra-regional trade in staple foods, agricultural inputs, and primary agribusiness products o Intermediate result 2: Increase global and intra-regional trade in higher-value products • Improving the enabling environment for regional trade and investment (IO 2): o Intermediate result 3: Implement regional and multilateral trade and investment agreements and support institutional strengthening o Intermediate result 4: Implement trade and transportation facilitation measures in the region To achieve SATIH’s goal, the activity undertakes a range of tasks that sit within four ‘components’. Programming varies widely across and within the components, from investment transaction support in South Africa to SADC institutional building in Botswana to structured trade seminars in Zambia. This range speaks to the wide array of objectives under SATIH’s mandate, and the associated complexity of its interventions. The Annex to this report outlines these in further detail. The scope of the four components are as follows: Project Objective Intermediate Results IR1: Intra-regional trade in staple foods, agricultural inputs and primary agribusiness products increased USAID Southern Africa RDCS GOAL: MORE INTEGRATED REGION WITH IMPROVED QUALITY OF LIFE FOR SOUTHERN AFRICANS Activity Goal Component 2: Export Competitiveness • Company-level technical assistance • Trade shows & B2B linkages • Marketing and promotion of Southern African products • Improve knowledge of market conditions • Association and export agency capacity building • Build capacity to increase AGOA utilization Component 3: Finance and Investment • Warehouse receipt system strengthening • Financial sector strengthening • Increasing investment into the region • Development of funding platforms & vehicles • Investment enabling environment Component 4: Trade Facilitation and Enabling Environment • Implement trade facilitation agreements • Address non-tariff barriers • Facilitate & increase trade • Support logistics sector improvements • Empower professional associations’ advocacy in support of trade policy reforms Gender Integration • Capacity building for advocacy by women’s business associations • Link and strengthen women’ business associations in the region • Increase lending to women-owned businesses • Support women-owned businesses through BDS support • Promote gender-sensitive job creation in agricultural enterprises IR 2: Global and intra-regional trade in higher value products increased IR 4: Trade and transportation facilitation measures implemented Intermediate Objective 2: An improved regional trade and investment enabling environment Intermediate Objective 1: Increased competitiveness of target value chains USAID Southern Africa RDCS Development Objective I: Increased Sustainable Economic Growth IR 3: Regional and multi-lateral trade & investment agreements & support institutions strengthened Southern Africa Trade and Investment Hub Project Objective: Expand Regional & Global Trade Investment Intermediate Objectives Components Component 1: Agribusiness Trade • Improve trade in inputs and services • Facilitate use of new agricultural technologies • Support trade in key non￾staple commodities • Strengthen trade associations & cooperatives Project Context USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 28  Component 1: Agribusiness Trade – aims to increase the competitiveness of selected regional agricultural value chains for export and regional trade. This includes: o Support of structured trading institutions and agriculture market information systems o Value chain development work in various export-focused value chains – including beef in Botswana, and mohair in South Africa and Lesotho  Component 2: Export Competitiveness – seeks to increase exports from Southern Africa countries to world markets, leveraging the African Growth and Opportunity Act (AGOA) where possible. This includes: o Information sharing and technical assistance to enable businesses to meet relevant export standards (e.g. Food and Drug Administration (FDA) regulations) o Funding businesses to attend international trade fairs and facilitation of inwards buyer missions o Supporting Ministries of Trade to develop and socialize AGOA strategies  Component 3: Finance and Investment- facilitates and accelerates investment in small and medium-sized enterprises (SMEs), and seeks to develop strategic value chains that strengthen economic growth and trade with the U.S., the Southern Africa region, and across sub-Saharan Africa. This includes: o Direct facilitation of investments by providing transaction support o Improving lending, both in combination with the Warehouse Receipt System (in close alignment with Agribusiness Trade) and using USAID's Development Credit Authority (DCA)  Component 4: Trade Facilitation and Enabling Environment – supports regional and national partners in transparent, reliable, efficient, and cost-effective trade via both a top-down, policy-level approach and bottom-up activities supporting traders and associations. This includes: o Technical assistance to government bodies to implement trade facilitation measures (e.g. the National Single Window in Botswana, SADC’s trade facilitation program) o Implementation support for a Trade Information Portal in Zambia o Provision of training on sanitary and phytosanitary standards and non-tariff barriers Geographic scope SATIH works in eight Southern African countries (Botswana, eSwatini, Lesotho, Malawi, Mozambique, Namibia, South Africa, and Zambia). The bulk of programming has occurred in Zambia, Malawi, and South Africa. Additional context for the evaluation Several factors outside of the immediate control of USAID/REGO and DAI played an important role in shaping programming. The shift in administration in the USA brought with it a new set of programming priorities, from working closely with governments and regional bodies to focusing on private sector engagement. This meant that the interventions that involved working with national governments (such as Project Context USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 29 those under the Trade Facilitation and Enabling Environment component) were deemed not to be a priority. In a separate move, reprioritization of budget meant that SATIH had to significantly reduce the rate of activity implementation. There is value in gathering insights on managing and designing for these types of contextual factors; however, the evaluation team will undertake the assessment cognizant of the challenges these factors brought. Project Context USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 30 6.3 OBJECTIVES AND TASKS OF THE SEED TRADE PROJECT (STP) USAID/REGO launched Feed-the-Future Seed Trade Project (STP) in October 2015 at a total estimated cost of $18.2 million. USAID/REGO awarded the programming to DAI Inc (DAI), through a competitive bidding process. STP seeks to increase the trade of quality seed in SADC by assimilating the SADC Harmonized Seed Regulation System (HSRS) principles into national laws. The project is over halfway through its implementation and is expected to end in September 2020. As outlined in Figure 2, STP aims to expand the regional trade in seed, in order to improve agricultural productivity. Figure 2: STP theory of change (TOC) Source: Seed Trade Project, Written Response to ISP Team Information Request, 2019; USAID, Feed-the-Future Seed Trade Project Annual Performance Report, 2018; USAID, Feed-the-Future Seed Trade Project Annual Performance Report, 2017 Activity Objective Intermediate Results IR1: Increased Availability of Improved Seeds in the Region 1.1. Increased seed that is pest and disease free 1.2. Improved seed quality assurance 1.3. Increased production of varieties registered in the catalogue 1.4. Increased registration on the regional catalogue of improved varieties of a diversity of key SADC crops Increased Seed Trade Across SADC Region Activity Goal IR2: Increased Availability of Technologies Management Practices and Innovation 2.1. Improved use of ICT in seed quality assurance 2.2. Increased human capacity of national seed authority staff 2.3. Equipped national seed laboratories to HSRS standards 2.4. Harmonized standards across countries IR3: Improved Regional Policies to Support Agriculture Growth 3.1. SADC HSR domesticated by Member States 3.2. Aligned policies and laws that support agriculture growth in SADC member states 3.3. Increased awareness of SADC HSRS created among Member States 3.4. Established collaborative partnerships with other USAID projects and organizations Capacity building Policy Advocacy Support to SADC institutions Technical trainings and awareness creation events Acquisitions of new physical and digital infrastructure for seed sector stakeholders Updating existing policies to align with HSRS Drafting new seed sector policies where gaps have been identified Assessments of seed regulatory environments Tasks Project Context USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 31 Over the course of its five-year programming period, STP has three objectives, which feed into a much higher-level objective of expanding regional trade in seed. The three objectives and their associated intermediate results are as follows: • Activity Objective 1 (AO 1): Increased availability of improved seed in the region o 1.1. Increased seed that is pest and disease free o 1.2. Improved seed quality assurance o 1.3. Increased production of varieties registered in the catalogue o 1.4. Increased registration in the regional catalogue of improved varieties of a diversity of key SADC crops • Activity Objective 2 (AO2): Increased availability of technologies, management practices, and innovation o 2.1. Improved use of information and communication technology (ICT) in seed quality assurance o 2.2. Increased human capacity of national seed agencies’ staff o 2.3. Equipped national seed laboratories to HSRS standards o 2.4. Harmonized standards across countries • Activity Objective 3 (AO3): Improved regional policies to support agriculture growth o 3.1. SADC HSR domesticated by member states To achieve STP’s goal, DAI undertakes two main activities which will be the focus of this evaluation: • Capacity building – seeking to improve HSRS assimilation by providing support to the institutional capacities of SADC institutions and the authorities within focus countries to implement • Policy advocacy - providing financial or technical support on the review or creation of HSRS￾aligned policies To support capacity building work, STP also manages grants under contract (GUC) of up to $1.5 million. DAI uses the GUC facility to issue grants to seed trade associations, national seed authorities, private sector companies, and other relevant actors that can advance STP’s capacity building work. Geographic Scope As a Feed-the-Future (FtF) project, STP’s geographic scope covers FtF’s focus or aligned countries— namely, Zambia, Malawi, Mozambique, and Zimbabwe. The majority of STP initiatives focus on Zambia and Malawi. Zambia’s focus reflects its role as a significant player in the broader regional seed industry, and as the host of the SADC Seed Centre. Malawi’s focus reflects the strong relationships STP has built with the Ministry of Agriculture and national seed sector representative authorities. Due to challenges with the national political climate in Zimbabwe, and STP not having a Zimbabwe country advisor, STP’s work in Zimbabwe is limited to supporting a handful of private sector institutions. Project Context USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 32 While Mozambique has proved to be a more welcoming policy environment, STP has been far less active in Mozambique than in Zambia and Malawi. This varying level of activity across the four focus countries can be seen in the number of organizations from each country that STP has supported, as well as in the number of policies STP participated in drafting or reviewing across the four countries (see Figure 3). Additional context for the evaluation Following concerns about poor performance in 2017, the Bureau for Food Security and USAID/SA conducted a project assessment of STP. This assessment recommended that STP deepen stakeholder engagement and collaboration, especially with focus country governments and SADC FANR, and explore additional means of increasing trade of high-quality seed in addition to the harmonization of quality seed. STP began implementing the recommendations of the assessment in late 2017 / early 2018. starting with relocating from Pretoria to Lusaka and restructuring its staffing model through changes in both personnel (notably, the Chief of Party) and role designs. Following these adjustments, 2018 was by far the project’s most active year, during which STP made significant progress toward many life-of￾project Targets. Four of the five grants in issue, 11 of the 12 national policy reviews, and all 20 varieties registered in the SADC Variety Catalogue by the project were undertaken in 2018. This increased activity is reflected in higher project spending (see Figure 4); STP spent $2.1 million in 2018—30% more than in 2017. 7 7 There was also a 23% increase in year-on-year expenditure between 2016 and 2017. This is attributable to the fact that 2016 was the project’s first year and was largely spent forging new stakeholder relationships and consulting on the state of HSR assimilation in the region. Source: Seed Trade Project, Written Response to ISP Information Request, 2019 2016 2017 2,140 1,654 1,342 2018 Salaries & Wages Grants Other Direct Costs Travel, Transport and Per Diems Consultants Figure 4. Seed Trade Project annual expenditure by category, 2016 – 2018 (thousands of dollars) Figure 3: Select seed trade project outputs by country, 2016 – 2018: Number of organizations assisted / policies drafted or reviewed Project Context USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 33 6.4 OBJECTIVES AND TASKS OF THE TRADE POLICY INITIATIVE (TPI) The Trade Policy Initiative (TPI) is a $1.5 million activity implemented via a cooperative agreement with the Trade Law Centre (tralac), a public benefit organization based in South Africa funded by several donors that are interested in supporting the advancement of trade policy in Africa 8. TPI aims to enhance the quality of trade policymaking in Southern Africa. This entails equipping policy stakeholders with knowledge they can use to ensure that trade policy supports inclusive growth and sustainable development (see: Figure 5). To achieve TPI’s goal, the activity undertakes three main categories of work, which will be the focus of this evaluation: • Applied policy research – TPI research aims to provide quality, accessible trade policy and trade law analyses to enable the advancement of regional integration issues (Strategic Objective 1) • Capacity building – TPI capacity-building efforts aim to transfer trade policy and trade law knowledge to government officials and other stakeholders (Strategic Objective 2) 8 SIDA (core donor), USAID, and other development partners with ad hoc requests. Activity objectives Immediate results Outputs Tasks Provide quality, accessible trade policy and trade law analysis to policymakers and other stakeholders Transfer knowledge on trade policy and trade law to government officials and other stakeholders through partnerships that focus on embedding capacity in institutions Empower, especially marginalized stakeholders to participate more effectively in trade policy and governance debates and processes IR1: Greater availability of trade and integration data • Monitoring trade negotiations • Collecting and analysing data, policy, and legal instruments • Disseminating information through trade briefs, working papers, and daily news digests Research Capacity building Policy dialogue IR:2: More capacitated and empowered stakeholders to make better decisions on trade and integration IR3: More active and transparent dialogue about good trade governance and integration • Providing short courses; • Running Internship programs • Developing and providing e￾learning programs • Facilitating dialogue events • Providing commentary on trade and integration issues to contribute to public dialogue Filling gaps in access to information about international trade and integration through dissemination of data and analysis Building capacity for good trade governance and integration, through innovative training solutions Supporting and facilitating active and transparent dialogue, which underpins good trade and integration governance Trade Policy Initiative goal: Enhance the quality of trade policy in Southern and Eastern Africa Activity Goal Figure 5: TPI theory of change (TOC) Source: USAID/REGO, Trade and Food Security Project Appraisal Document, 2014; Tralac website, 2019 Project Context USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 34 • Policy dialogues – Policy dialogue activities seek to empower stakeholders to participate more effectively in trade policy and governance debates and processes (Strategic Objective 3) Geographic Scope tralac, the implementer of TPI, is based in South Africa. Many of the short courses and training programs are hosted in Cape Town, South Africa. A large part of tralac’s work under TPI nonetheless reaches other parts of Southern Africa and East Africa. Attendees at those trainings (including officials, trade law professionals, and journalists) are drawn from across Southern and East Africa, as is tralac’s network of trade and law professionals / researchers that contribute to the production of research and policy dialogue. Furthermore, over the last five years the annual conference (tralac’s flagship event) took place in Namibia, Rwanda, Zambia, and South Africa. Evaluation design USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 35 EVALUATION DESIGN 7.1 EVALUATION QUESTIONS For each activity, the evaluation team tailored the evaluative themes to each activity by developing more detailed evaluative questions. This ensured that the evaluation approach reflected the significant difference in scope of the three activities within the program, and addressed a relevant set of corresponding questions. These questions then formed the basis for building the evaluation approach—with evaluative design matrices in the inception report detailing analyses the team set out to conduct, along with proposed data sources and tools. The evaluation team developed a detailed list of evaluative questions as part of the inception phase, which are clustered into higher-level questions to aid the readability of this report; the annex includes the full list of evaluative questions considered. The scope of each activity warranted a different set of focus areas, as summarized below: SATIH • Relevance – given the breadth of tasks under evaluation, questions focused on degree to which specific tasks achieved their intended outcomes and delivered additional benefits (beyond what would have happened without USAID interventions). With this, the evaluation team sought to identify common trends that informed why some elements of SATIH’s work were better placed and made stronger contributions its objectives than others • Management – in light of performance challenges identified during the inception phase, questions focused on the effect that contract structures, along with DAI and USAID/REGO management practices, had on the performance of SATIH. Given the breadth of activities and wide scope of SATIH, questions also centered on the role of prioritization processes in informing the relevance of tasks undertaken • Sustainability – in light of the significant volume of transactional work, questions focused on whether or not these activities would be sustained in the absence of USAID support, as well as the extent to which the private-sector-led and transaction-oriented approaches of SATIH are delivering results in an equitable manner STP • Relevance – relevance questions seek to explore whether harmonizing seed regulations is the right objective (i.e. both necessary and sufficient) to increase agricultural productivity, along with whether the right tasks have been selected and effectively executed to realize this objective (looking at the relative effectiveness of different types of support, e.g., grants, training, awareness creation events) • Management – management questions recognize both the previous performance issues that STP faced, along with efforts by activity management within both DAI and USAID/REGO to correct performance issues. With this, questions focus on how management of the activity is currently perceived, and the impact that the revised country coverage model has had on implementation of STP Evaluation design USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 36 • Sustainability – in light of the long-term timelines for policy change, sustainability questions look at the extent to which STP’s achievements will be sustained and furthered beyond the life of the project, particularly looking at the extent to which institutions strengthened on both national and regional levels have an enduring strength that is not dependent on ongoing USAID support TPI • Relevance – focus is given to the extent to which other ecosystem prerequisites are in place that enable TPI to enhance the quality of trade policy in the region through its relatively focused set of activities. From this, relevance questions then go on to look in further detail at the outcomes generated by each activity, addressing a gap created by the relatively output-focused nature of data available to the evaluation team at the start of this evaluation. In addition, the evaluation team explored how well executed these tasks were, which included elements such as the quality of facilitators, and organization of events • Management – from a management perspective, questions focus on how the use of a cooperative agreement as a contract structure in the case of TPI, and the TPI management model and Associate network, helps or hinders delivery of the activity • Sustainability – sustainability questions reflect on the extent to which TPI activities would continue in the absence of USAID support, along with the extent to which the TPI audience is diverse and equitable 7.2 METHODOLOGY The evaluation approach blends several data sources and tools to ensure a grounded evidence base. This includes: • 150+ semi-structured interviews – conducted both by phone and during visits to Botswana, Malawi, Mozambique, and Zambia • 333 survey responses – targeting beneficiaries of training and awareness events (a response rate of 32.3% out of the 1,032 survey links distributed) 9 • 77+ program documentation – including the PAD, vision and strategy documents, work plans, and budget data • Sourcing of data in country – including publicly available and privately held datasets and reports on the issues the project was addressing (e.g., covering Malawi’s Warehouse Receipt System (WRS), trade flow data, time and cost to trade data) • PMER data from the implementing partners – where it was possible to make this available to the evaluation team (subject to limitations to protect confidentiality and observe data protection requirements) • Supplementary research – to stress-test insights generated through other data sources 9 Number of survey links distributed excludes “bounce back” emails received, where the email addresses provided by the implementer were invalid. This also excludes illegible email addresses, given that distribution lists were often transcribed from hand written attendance sheets that the implementers had not previously digitized. Evaluation design USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 37 Interview breakdown Interviews were conducted in person in Botswana, Malawi, Mozambique, South Africa, and Zambia, as well as over the phone in other countries (see Table 1). With this, the evaluation team engaged with 154 interviewees during 128 distinct interviews. Table 4: Breakdown of Interviewees Category Bots. Malawi Moz. SA Zambia Zim. Other Grand Total Implementers: DAI / tralac 1 1 5 6 1 14 Government officials 7 7 3 3 10 1 31 Investors / CrossBoundary 8 2 10 NGOs / donors 10 4 6 2 2 24 Other 3 1 1 5 Private sector 4 21 3 3 27 2 1 63 USAID / bilateral 3 1 1 1 6 USAID / REGO 1 1 Total 11 45 12 21 50 5 8 154 Survey responses The evaluation team distributed surveys to participants and beneficiaries of training and awareness creation events across the three activities of TFSP. Table 2 below breaks down the responses received by activity: Table 5: Breakdown of survey responses Activity Links Distributed 10 Responses Response Rate Southern Africa Trade and Investment Hub 238 87 36.6% Seed Trade Project 255 112 43.9% Trade Policy Initiative 539 134 24.9% Total 1,032 333 32.3% 7.3 LIMITATIONS TO THE EVALUATION The evaluation design put in place significant mitigations to address risks identified during the evaluation’s inception phase. Those risks included sample bias, stakeholder availability, low survey response rates, the sufficiency of activity mapping, lack of availability of quantitative data, and scope of travel only covering a sub-set of countries. Full details of relevant mitigations are contained in the inception report, and the 10 Excludes “bounce back” emails where the project had provided invalid email addresses to the evaluation team. Evaluation design USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 38 evaluation team is confident that these mitigations were largely successful in addressing the main limitations. That said, there are limitations that remain, in part driven by factors out of the control of the evaluation team that arose in program execution to date: 1) Availability of robust outcome data – during a mid-term performance evaluation the evaluation team would typically have access to strong outcome data with moderately high coverage across the project. A combination of factors hindered access in this case: - Weaknesses in indicator frameworks and data gathering – which the evaluation report will outline in further detail. Specifically in the case of SATIH, this reduced the ability to identify the totality of tasks SATIH worked upon, made it difficult to source disaggregated data to break down the results, while spend data is difficult to assign to specific activities in order to make value for money assessments - A highly fragmented set of activities undertaken to date, particularly in SATIH – which reduced availability of evidence around the extent to whether a particular approach had worked 2) “Survivor bias” – data naturally reflected the activities that the projects had pursued. The evaluation team did however engage with implementing partners and USAID/REGO to identify tasks that had been deselected, and have surfaced missed opportunities where possible 3) Selection bias of participants – given implementing partners provided initial contacts and access to project stakeholders, there is a risk of gathering partial views. This largely did not arise, both with the evaluation team hearing a diversity of balanced perspectives (both positive and negative), and with the evaluation team identifying further leads beyond those identified by implementers. That said, within the timeframe available for the evaluation this report reflects what the evaluation team consider to be a representative, but ultimately non-exhaustive set of findings 4) Recall bias – stakeholders may have been engaged on activities early in the life of the hub, and therefore may not have a perfect recall of their involvement. This has been mitigated where possible through the breadth of stakeholders engaged with, and through triangulating interview findings with other data sources where feasible 5) Absence of the counter-factual – given the size and broad scope of the project, there is no ‘control group’ to understand what would have happened in the absence of TFSP. Given this, assessments of additionality have been based on the best available data 6) Difficulty associated with disaggregating stakeholder interview codification and survey response data – although respondents primarily responded to surveys against a single intervention, and interviewees often provided an average response across the areas they have been involved in, this does mean that more disaggregated survey responses are directional in nature having been affected by other factors beyond the specific task in focus Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 39 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS The report now presents the evaluative findings for each of the activities in turn, before turning to the TFSP project-level evaluative findings. This reflects the fact that project-level findings primarily serve to surface common learnings and missed opportunities across the three activities, and are heavily informed by the activity-level evaluations. 8.1 SOUTHERN AFRICA TRADE AND INVESTMENT HUB (SATIH) The report now presents the evaluative findings for each of the activities in turn, before turning to the TFSP level evaluative findings. This reflects the fact that project-level findings are heavily informed by the activity-level evaluations, with project-level findings primarily serving to surface common learnings and missed opportunities across the three activities. Summary findings: SATIH SATIH is meeting 10 of the 22 Y1 and Y2 top-level indicator targets, including outcome-level targets around value of exports and investments created. That said, SATIH is failing to meet a number of its targets. Furthermore, there are challenges in the design of the indicator framework that mean indicators alone do not tell a full story of SATIH’s performance; these issues became increasingly relevant in the face of execution challenges. Where SATIH has had successes against its indicator framework, they are largely driven by a transaction￾oriented approach, and to a lesser degree, policy-oriented work (e.g., under the trade facilitation component). Many of the indicators where Y1 and Y2 targets have been met will still require significant ramp up of results to achieve LOP targets however, with this ramp up implicit in the targets from the start. For example, although SATIH met 128% of its combined 2017/2018 export target, it remains at 16% of the indicator’s LoP target (despite being two years into the five-year life of project timeframe). SATIH is falling short on some outcome indicators, which include installation of storage capacity, and businesses applying improved technologies or management practices. The majority of missed targets are at the output level—e.g., buyer/seller linkages created, public-private partnerships (PPPs) created, person hours of training completed, and number of firms meeting standards to export.11 Furthermore, there have been difficulties in reporting comprehensively against certain indicators, including time and cost to trade (a crucial outcome indicator for SATIH). The insufficiency of the indicators in providing a clear view of performance relates to how they only cover a small proportion of the potential outputs and outcomes of the tasks undertaken. The issues with the indicator design then became increasingly relevant in the face of execution challenges; the outputs and outcomes of the activity have had weak contributions to SATIH’s overall objectives, which the indicator 11 The indicator on number of organizations assisted to reduce non-tariff barriers (NTBs) is also not meeting targets, but latest DAI reporting suggests this indicator has been discontinued. Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 40 reporting does not fully reflect. For example, some of SATIH’s activities have lacked additionality, despite their performance against indicator targets. Transaction-oriented approaches have not always fed into more systemic changes—for example, although Finance and Investment efforts have resulted in more loans dispersed, driven particularly by financing for construction of commodity warehouses, the underlying challenges that limit banks from lending remain largely unaddressed. In parts of the program, particularly Trade Facilitation and Enabling Environment, and Agribusiness Trade, relatively small budgets are insufficient for addressing what are considerable challenges (e.g., reducing time and cost to trade, establishing functioning structured trading environments). SATIH has faced significant management issues, although at the time of writing both USAID/REGO and the implementer appear increasingly confident that the right leadership is in place for the remainder of project. Both USAID/REGO and the implementer have played a role in these issues. On the USAID/ REGO side, leadership changes and competing priorities (driven in part by multiple sources of project funding) added complexity to the management environment. Meanwhile high levels of CoP turnover on the implementer side have resulted in a loss of continuity that has limited the clarity of SATIH’s vision, while poor prioritization processes have contributed to SATIH taking on activities of low relevance. Meanwhile, there were missed opportunities for the implementer to take a strong leadership role in steering through budget cuts and shaping the technical direction of the program. The nature of the contract structure has led to some of these issues. The contract is set up with a Performance Work Statement, however USAID has few tools at its disposal to monitor and steer performance. This extends largely to the CPAR mechanism, as opposed to a fees at risk structure that is typical of a performance based contract. From a sustainability standpoint, there is significant variance across SATIH. Many of the transactional successes by their nature deliver sustainable benefits beyond the LoP. For example, investments facilitated by SATIH will continue to enable those businesses to deliver future benefits (e.g., job creation) beyond the LoP providing the businesses stay in operation. Similarly, the relationships established via linkage events will continue to be valuable as the business scales. In other instances, however, transaction-oriented approaches focused on making a business case possible that would not exist without SATIH’s support – e.g., by reducing the transaction costs that make small investment sizes difficult to execute, or covering the cost of trade fairs that may have a high risk / reward profile for a specific business. In addition, many of these interventions address market failures through a wider signaling effect (i.e., improving the profile of Southern African businesses for trade and investment). Ultimately, the weak underlying business cases at the firm- or investor- level means some of this work would cease in the absence of USAID support, with low potential to create sustainability. Interventions targeting more systemic changes have also faced sustainability challenges – primarily because SATIH’s efforts did not move these interventions to a point where they could become self-sufficient, owing in part to a lack of capacity and follow through. As an example, stakeholders had agreed on a business model for an independent grain market information systems operator in Zambia (ZAGIS) whereby users paid a small fee per ton in the system. However, funding support needed to set up the institution and system itself was required to operationalize this business model. SATIH support stopped prior to this stage. More broadly, the extent to which results have been equitable is limited. There is no evidence of a clear gender strategy (despite this appearing in the TOC). Transaction support work was one of the few areas of the project where SATIH incorporated a gender-lens (using fit check criteria), while other gender activities (e.g., under export competitiveness) appeared more ad hoc in nature. The report now goes on to detail and evidence these findings in depth. Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 41 8.1.1 RELEVANCE Scope of evaluative theme SATIH’s tasks reflect a substantial breadth of work, both within and across the four components. The evaluation of SATIH’s relevance looks first at the overall performance of SATIH as an activity, before then reviewing how the relevance of SATIH’s activities shape its performance (considering how they fit into the ecosystem, how effectively designed they are, and how well-executed they have been). This is considered firstly at the activity level, before moving into a more technical set of evaluative findings that look at how these findings manifest in a particular component – with a view to informing future programming in each specific component area. Synthesized findings SATIH has met many key indicator targets, including overall value of trade and investment facilitated. While positive, these successes do not reflect the program’s performance in its entirety, with indicator successes driven in a large part by transactional programming providing direct support to businesses or investors. This fails to capture harder to measure, but crucially important, progress against the overall objectives of SATIH in delivering more systemic level change. It also does not reflect whether activities were as effective or efficient as possible. The analysis suggests there are gaps in this regard, with challenges in overall program design and prioritization resulting in tasks that failed to build on each other in a coherent manner, and to achieve broader systemic change. There were also low levels of additionality in parts of SATIH, which meant some of the results delivered would have been likely to occur in SATIH’s absence. Overall execution varied broadly across the program, with stakeholders recognizing teams that had stronger in-country presence as executing to a higher quality. It is important to note that recent changes in leadership suggest a significant strategic and operational shift in SATIH moving forward, although it is too soon to evaluate the efficacy of strategic shifts, which also fell outside of the evaluation period. 8.1.1.1 Overall Activity-Level Relevance Findings Outcomes: How has SATIH performed against Life of Project (LoP) indicators? SATIH is meeting 10 of 22 top-level indicator targets in Y1 and Y2 (as shown in Table 3). These include key top-line outcomes such as value of investment facilitated, value of trade facilitated, number of (non-agricultural) private sector enterprises applying improved technologies and management, and number of metric tons traded through commodity exchanges. Transaction-oriented firm-level support has, in part, driven the successes seen within the indicator data. Contributions to SATIH’s outcome indicators often links back to direct support to individual businesses. The work of the Finance and Investment component providing transaction support to investors has driven significant results around new private sector investment. Similarly, this component’s facilitation of loans for warehouse construction played a pivotal role in ensuring that targets were met for indicators such as number of metric tons of grain traded and value of loans to the private sector. The technical assistance and linkage support from the Export Competitiveness team, and to a lesser degree Agribusiness Trade team, has increased exports and the number of linkages established. Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 42 In addition to this transaction-oriented work, the output indicators reflect the policy￾oriented work undertaken by SATIH. The number of enabling environment policies supported exceeded targets over the period, in part due to the efforts of the Trade Facilitation component, which was able to use its global expertise to support technical policy development. That said, although this is classified as an outcome indicator, the indicator data do not quantify whether there were on-the-ground results of this work, in terms of its impact on the time and expense of trade as an outcome. Meeting LoP targets will, in most cases, require a significant scale-up of outputs and outcomes over the remaining half of the program. In 5 of 10 cases, where performance in Y1 and Y2 has exceeded 100% of target progress against LOP targets remains below 40% (despite being two￾fifths of the way through the LOP). In the case of many indicators, this trajectory is natural—proving early success often generates more interest and makes it easier for a project to bring others on board. Similarly, much of the work has significant lags between initial activity and outcome – e.g. it may take months or even years for an introduction at a trade show to result in an export transaction. This means that there is an anticipated compounding effect for businesses supported in the early days of the project that continue to yield results for the remainder of the LoP. This was implicit in the targets from the start. Given the significant changes taking place within SATIH, including an ongoing activity re-prioritization exercise, the evaluation team has a limited basis on which to assess whether SATIH will be able to sufficiently accelerate results delivery to meet its LoP indicator targets. It is also worth noting that LoP targets remain unchanged, despite the significant budget cuts SATIH has undergone— at certain points, the monthly budget was nearly halved. Other sections of the report assess how these budget cuts were handled by management; regardless, any consideration of SATIH’s ability to achieve LoP targets must consider this change in available resources. SATIH is behind target on 8 of 22 top-level indicators. It is falling short on indicators that include installation of storage capacity, buyer/seller linkages created, and businesses applying improved technologies or management practices. The remainder of missed targets are at the output level—e.g., public-private partnerships (PPPs) created, person hours of training completed, and number of firms meeting standards to export.12 Furthermore, there have been difficulties in reporting comprehensively against certain indicators, including time and cost to trade as a crucial outcome indicator. Challenges in the baselining and reporting have also limited the indicator framework as a tool to monitor and evaluate programming. SATIH has faced challenges in establishing effective reporting against 4 of the 22 non-context indicators. This limited the ability of the evaluation team to measure progress fully against these. Reducing time and cost to trade was originally positioned as a central goal of the program, and a data collection and baselining exercise was envisioned to track and program against this. Early attempts at establishing a baseline were unable to meet the precise USAID data gathering requirements (e.g., the need to track the time and cost for a specific container size to cross the border). Similarly, indicators capturing FTE jobs created and firms reporting an increase in profitability were only reported on at the end of 2018 – two years into the program. When reported on, these indicators were in a different unit to the target (e.g. number jobs as opposed to FTE). Overall the indicators themselves do not tell the full story in that it is difficult to assess whether transaction-oriented successes feed wider systemic changes, or if interventions have been as efficient and effective as they could have been. The remainder of the relevance section will explore this dynamic in more detail. 12 The indicator on number of organizations assisted to reduce non-tariff barriers (NTBs) is also not meeting targets, but latest DAI reporting suggests this indicator has been discontinued. Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 43 Table 6: SATIH indicators13 Indicator Output/ outcome Absolute value achieved Progress against 2017/ 2018 targets Progress against LoP Target Value of exports in targeted non-agricultural and agricultural commodities (ACTE DO1.1) (EG 3.2-23) Outcome $ 25,541,719 128% 16% Value of new private sector investment in agricultural sector (ACTE DO I.2); (FTF 4.5.2-38); (EG 3.2-22) Outcome $ 83, 074, 147 346% 65% Value of new private sector investment in non￾agricultural targeted sectors leveraged by the Hub (ACTE DO I.3) Outcome $ 32, 734, 760 205% 36% Number of metric tons of grain traded through commodity exchanges / certified warehouses supported with Trade Hub assistance Outcome 77,174 1286% 107% Number of certified warehouses participating in commodity exchanges and/or warehouse receipt systems with trade hub assistance Outcome 85 2125% 100% Metric tons of installed storage capacity Outcome 82,642 55% 17% Number of for-profit non-agricultural sector private enterprises that applied improved organizational-level technologies or management practices as a result of USG assistance (ACTE IR 2.1) (FTF 4.5.2-42); (EG 3.2- 20) Outcome 58 153% 3% Number of buyer / seller linkages established in targeted non-agricultural sectors as a result of Trade Hub assistance (ACTE IR 2.2) Output 623 88% 6% Number of assisted non-agricultural sector firms / associations meeting international grades and standards to export (ACTE IR 2.2) Outcome 2 67% 40% Number of for-profit private enterprises that applied improved organizational-level technologies or management practices as a result of USG assistance (ACTE IR 1.1) (FTF 4.5.2-42); (EG3.2-20) Outcome 12 19% 4% Number of buyer / seller linkages established in targeted agricultural sectors as a result of Trade Hub assistance. (ACTE IR 1.2) Output 122 51% 7% Number of assisted agricultural sector firms / associations meeting international grades and standards to export (ACTE IR 1.3) Outcome 6 120% 20% Value of loans to target enterprises / groups due to Trade Hub assistance. (EG 3.2-6) Outcome $ 16,810,533 168% 13% Number of food security private enterprises, POs, WUAs, women’s groups, trade and business associations & CBOs receiving USG assistance (EG 3.2- 4) Output 650 445% 67% Number of PPPs formed with hub assistance (EG 3.2-5) Output 17 89% 38% Number of enabling environment policies analyzed, consulted on, drafted or revised, approved, and implemented with USG assistance (ACTE IR 1.3) (FTF 4.5.1-24); (EG 3.1-12) Outcome 23 460% 47% Number of organizations assisted to reduce non-tariff barrier (NTB), which are relevant to the SADC region Output 4 36% 0% Person hours of USG-supported training completed in trade and investment (EG.2-1) Output 799 14% 6% 13 Note: this reflects 18 of the 22 non-context top-level indicators. Difficulties have been encountered in reporting against the remaining four (time to trade, cost to trade, firms experiencing increased profitability, jobs created) Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 44 Ecosystem: To what extent is SATIH’s work set up for success when reviewed against the wider ecosystem and its challenges? Stakeholders identified a broad set of factors that need to be in place to enable trade and investment (outlined in Figure 6); SATIH is working in almost all of these spaces. These factors manifest regionally and at country level, with value-chain-specific permutations. In this context, addressing any one of these factors in a single value chain risks accomplishing little in the face of other constraints. Numerous examples within SATIH illustrate the interdependent nature of these challenges. For example, the evaluation team engaged with a business in Botswana that SATIH had supported by providing trade linkages and sponsoring its attendance at a trade fair. Although there was international interest in the product, after a long lead time the trade had not yet materialized in part because of difficulties scaling production to cater to that demand (including sourcing inputs and gaining access to capital). The evaluation team saw other examples where other crucial elements were missing that hindered success—including AGOA policies that are not implemented due to weak government capacity and warehouses that, post￾training on the warehouse receipt system, are unable to get certified because the commodity exchange lacks the capacity to do so. In the context of these challenges, the breadth of SATIH’s scope hinders its effectiveness, particularly given that this breadth exceeds SATIH’s budget. SATIH has an exceptionally broad mandate. Many of the wide range of issues falling under its remit are problems that could warrant programs nearing the size of SATIH in and of themselves. In Zambia alone, for example, the size of the structured trade funding need amounts to $25 million, while the overall 2018 SATIH budget for structured trade is $104,000 (and includes programming in both Zambia and Malawi). 14, 15 Similarly, in trade facilitation the 14 Indaba Agricultural Policy Research Institute (IAPRI), Draft Report Feasibility for Investment Enhancing the Zambian Warehouse Receipt System and Aligning the Food Reserve Agency Strategic Plans to the System, 2019 15 SATIH 2017/ 2018 Annual Workplan; These numbers were presented prior to further budget cuts in 2018, which means that actual spend was likely lower. Source: interview insights Figure 6: Necessary factors to enable trade and investment Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 45 OECD notes that budgeted or estimated capital expenditure to introduce trade facilitation measures ranges from $3.9 million to $21.3 million in a single country alone; SATIH’s 2018 budget for trade facilitation implementation was $329,000. 16 Even prior to the budget cuts, but particularly after them, SATIH was at risk of spreading itself too thinly in trying to address the wide range of challenges assigned to it—a risk that materialized in several areas of the program. Examples of this lack of focus include the wide range of activities undertaken in each component that focus on different, loosely connected issues in different geographies and in different value chains. Effectiveness: How effectively have SATIH’s interventions contributed to the outcomes delivered? SATIH’s breadth of activities reflects both the theory of change itself and the way SATIH has structured and managed programming around it—both of which have reduced SATIH’s effectiveness. As outlined Figure 7, Intermediate Results 1 and 2 (IRs 1 and 2) effectively restate the overall project goal of increasing regional and global trade, and are therefore extremely broad in nature. Intermediate Results 3 and 4, on the other hand, articulate conditions that need to be put in place to enable trade. Management around intermediate results means that the components related to IR 1 and 2 (which restate the overall project objective) have broad remits with no inherent prioritization. This lack of direction in turn contributed to piecemeal programming, particularly under one of these components - Agribusiness Trade, whose remit could cover a significant volume of issues given the dominant role of agriculture in the Southern Africa region (the other was Export Competitiveness, which had its mandate narrowed by the need to focus on AGOA). Management around intermediate results also contributed to overlaps in components – requiring strong coordination that did not always materialize. The components related to IR 3 and 4 (which articulate conditions to improve trade) should feed into achieving IRs 1 and 2. This overlap would not have been an issue if there was effective coordination between component to ensure shared programming. However, as the report elaborates on in management, components were for the most part managed in siloes – leaving opportunities on the table and creating overlapping component objectives. 16 OECD, The Costs and Challenges of Trade Facilitation, 2012 Project Objective Intermediate Results IR1: Intra-regional trade in staple foods, agricultural inputs and primary agribusiness products increased IR 2: Global and intra-regional trade in higher value products increased IR 4: Trade and transportation facilitation measures implemented IR 3: Regional and multi-lateral trade and investment agreements and support institutions strengthened Southern Africa Trade and Investment Hub Project Objective: Expand Regional and Global Trade Investment Intermediate results one and two effectively restate the project objective of increasing regional and global trade Intermediate results three and four state important conditions for enabling trade and investment Figure 7: SATIH intermediate results and project objective Source: SATIH Performance Monitoring and Evaluation Plan Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 46 Early signaling from new leadership suggests that it is looking at SATIH programming through a more strategic lens to address some of these issues. While it is too early to evaluate how planning translates into activities, the 2019 work plan suggests a much more interconnected approach to programming. The work plan outlines a new set of three strategic objectives (SOs) that all components feed into: 1) firm-level support, 2) market systems transformation, and 3) U.S. engagement. For example, SO 1 regarding firm-level assistance now places more explicit emphasis on the role that trade facilitation and better policies play in enabling the businesses receiving firm-level support to trade. In comparison, the previous TOC would have viewed export competitiveness interventions as feeding into a distinct objective (increasing trade) from trade facilitation work (improving the enabling environment). SATIH’s programming has not fully maximized the opportunity for mutually reinforcing interventions. On a wide range of individual issues it could have been beneficial to bring multiple SATIH tools to bear. For example, when it comes to enabling a business to become more competitive, support with linkages could be useful, as could support with meeting export requirements and accessing capital. However, there was little evidence to suggest that SATIH attempted to meet multifaceted challenges by programming across components. The evaluation team identified only eight examples among the stakeholders it engaged with of organizations becoming involved with multiple components of SATIH. The evaluation team also identified instances where businesses had a broader set of challenges that could have benefited from interventions across other components. As outlined in Figure 8, there were three times as many missed opportunities to program across multiple components as there were instances of multiple component engagement. There are also instances of indicator data demonstrating the achievement of strong results—but for which the additionality of SATIH programming is questionable. The assessment of relevance by component will explore these instances of questionable additionality in more depth, but specific issues identified included: • Transaction support – several large deals contributed significantly to overall reported investment in the indicators; however, interviews revealed that these larger deals would likely have occurred in the absence of USAID support (either by conducting due diligence in house or the investor paying for transaction support services akin to those provided through SATIH) • Warehouse investments – a big success claimed by SATIH was facilitating loans to enable warehouse investments in Malawi. However, later struggles to verify these investments are in part due to local stakeholders not recognizing SATIH as having contributed significantly to the investments – raising questions on the centrality of its role in the process • Trade facilitation technical assistance – although the project has not yet reported its impact on time and cost to trade, some government ministries prioritized some of these interventions 8 24 Multiple component engagement Unmet opportunity for multi-component programming Number of interviewees that interacted with more than one component vs number where there were clear examples of complementarities that were not leveraged (n=32) Figure 8: SATIH interview codification – cross-component programing Source: SATIH stakeholder interviews; n=32 Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 47 anyway (e.g., the Authorized Economic Operator [AEO] program in Malawi), and therefore would have had some beneficial impact on these trade indicators in the absence of USAID support (although they do cite performance improvement related to USAID support) On the policy side, there have been gaps in effectiveness due to lack of capacity to absorb SATIH’s interventions. Although there were some cases in which initial SATIH technical assistance yielded strong results (e.g., SATIH supported the design and scoping of an AEO program in Malawi, which the Malawian Revenue Authority went onto execute successfully), there were cases where lack of implementation capacity rendered the work ineffective. For example, while support to SADC on its trade facilitation program yielded a good program design, the support has so far been ineffective because SADC is still seeking the funding to implement the design. The evaluation team also identified at least one ministry of trade that received support for the design of AGOA strategies but did not include implementing AGOA strategy on its ministerial priority list – resulting in limited implementation capacity. Similarly, some businesses that engaged with SATIH’s workshops around standards have claimed that they lack the capacity to implement what they learned in workshops—suggesting that those workshop spaces could have been more fruitfully allocated to businesses with a demonstrated and proven capacity to implement. Execution: How well executed were SATIH’s interventions? Overall, SATIH earned mixed reactions on its execution, with a significant body of stakeholders expressing frustration at SATIH’s lack of follow-through. ‘Execution’ in this case concerns the various factors required to deliver successful outputs, including the quality of SATIH’s expertise and staff, and the extent to which SATIH advanced progress by following up on interactions and performing expected next steps (which at times included confirming SATIH was not going to pursue a particular intervention or opportunity). Only 40% of stakeholders engaged during the evaluation reported high levels of follow-through, although performance varied by component. As shown in Figure 9, Export Competitiveness outperformed the other components, followed by Finance and Investment – both of these components had low turnover in staff, and were noted by stakeholders as being highly communicative and having good technical expertise. Trade Facilitation suffered in part from its relocation from Gaborone to Pretoria, which led to stakeholders in Gaborone expressing concerns about follow-through, and from instances in which stakeholders requested follow-on funding and did not receive a reply. Agribusiness Trades’ follow-through metrics were in part a result of staff turnover meaning that conversations were dropped. Further, activities in structured trade were often left without additional Source: SATIH stakeholder interviews;n=52 Figure 9: Follow-through by component and for SATIH overall Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 48 follow-up. (this included unsuccessful applications that did not culminate in feedback or even notice that the application had been rejected). The two program areas conducted by sub-contractors were routinely commended; stakeholders noted the depth of expertise and quality of teams involved in the work. This applied to both CrossBoundary for transaction support under Finance and Investment and Nathan Associates under Trade Facilitation. As an example: all investors engaged by the evaluation team viewed the work of CrossBoundary as excellent, with only two of nine investors identifying minor areas for improvement (related to the desire for more senior staff time, and some process issues that slowed the start time). 8.1.1.2 Component-Level Relevance Findings Many of the themes the evaluation team identified at the activity level for SATIH manifest in specific ways within each of the components. With this, the evaluation team has prepared a set of technical findings per component area – with a view to informing future programming in that area. 8.1.1.2.1. Agribusiness Trade Agribusiness Trade worked across two broad sets of activities. The first focused on advancing structured trading in staple commodities. This work tackled challenges across the ecosystem, including building market information systems, raising awareness of warehouse receipt systems (WRS), and building commodity exchange capacity. The second focused on advancing trade in non-staple commodities and, to a lesser degree, in inputs. This included a wide range of activities, from sector assessments, to investment facilitation, to private sector advocacy support. Outcomes: What contributions to SATIH’s outcomes did the Agribusiness Trade component make? Agribusiness Trade indicator results flowed primarily from a single B2B grain linkage event 17. Per the FY2018 annual report, 18 most PMEP indicators that the Agribusiness Trade team contributed to reflect a single activity – the B2B grain linkage event hosted in 2017. The event aimed to connect surpluses of grain in Zambia countries with deficits in East African countries. Ultimately this enabled $10.2 million in trade, while also contributing to progress on linkage indicators and the number of metric tons of grain being traded through warehouses. In addition to the linkage event, grain traded through Africa Commodity Exchange (Malawi) was also included in number of metric tons traded through warehouses/ commodity exchanges indicator. Outside of the linkage event, the ad hoc nature of the activities limits the extent to which the evaluation team could isolate what these activities achieved. Many of the tasks undertaken by the Agribusiness Trade team are not captured by indicators. Examples such as the beef sector assessment or market information system work do not directly contribute to indicators measured, and 17 An even co-hosted with the East African Trade and Investment Hub (EATIH) where Zambian grain traders were introduced to East African buyers and a series of commitments to purchase were made. 18 The FY2017 Annual Report indicates little progress on indicators. This is like the result of the late start to programming by the Agribusiness Trade team due to delays in hiring. Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 49 the evaluation team have not been provided with disaggregated data under the ‘exports’ indicator that may include successful trade from these interventions. That said, stakeholders suggest that these other interventions made little progress, as the remainder of this section will explore in more detail. Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 50 Ecosystem: To what extent is the work of Agribusiness Trade Component set up for success when reviewed against the wider ecosystem and its challenges? Most of the Agribusiness Trade programming focused on structured trade, which stakeholders and secondary research note to be of significant potential value. Functioning commodity exchanges sit at the center of structured trade. They ensure that prices reflect underlying market dynamics, improve market transparency, and lower the transaction costs of linking buyers to sellers. 19 A functioning WRS, in turn, enables smallholders to access quality storage and smooth out price changes during the season (therefore maximizing profit), while accessing cheaper warehouse receipt backed loans. 20 A second aspect of structured trade is market information systems. Where these function effectively, they ensure more predictable, evidence-based government staple food export policy and improve market efficiency and transparency by reducing information asymmetries. 21 Interventions across the structured trade system should enable regional trade—creating a more predictable policy environment and creating access to trade finance, which ultimately makes it possible to link regional buyers and sellers on commodity exchanges. However, making structured trade work in a developing market context requires addressing issues across the value chain; Agribusiness Trade and the wider ecosystem of actors failed to do so, hindering effectiveness of SATIH programming. Enabling structured trade requires several key conditions to be in place—among them, sufficient market scale, credible contract enforcement, recognized grades and standards, and freedom from interference in the political economy—many of which require significant effort in Southern Africa markets (see Figure 10)22 19 Fintrac Inc., FtF Enabling Environment for Food Security Commodity Exchange, 2017 20 World Bank, A guide to Warehouse Receipt Financing, 2016 21 IAPRI, Draft Report Feasibility for Investment Enhancing the Zambian Warehouse Receipt System and Aligning the Food Reserve Agency Strategic Plans to the System, 2019 22 IPFRI discussion paper, Purpose and Potential for Commodity Exchanges in African Economies, 2010 Government ability to change in the short run Enabling Conditions for Commodity Exchange Sufficient market scale and activity Absence of contracts for the same commodity in offshore exchange (non￾redundancy) Freedom of cash market from political interference Integrated and active cash markets Credible contract enforcement, useful grades and standards, and consistent regulation Very limited Very limited Limited, given political sensitivity of food prices Can be affected through investment in infrastructure and appropriate regulation Potentially high, with credible governance Source: IPFRI discussion paper, Purpose and Potential for Commodity Exchanges in African Economies, 2010 Figure 10: Essential pre-conditions for the success of structured trade Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 51 Even once these are in place, significant investment is needed to develop and link relevant market institutions. Agribusiness Trade did not program at a scale that could markedly shift structured trade outcomes, did not strategically select specific interventions that would be effective in the face of other obstacles, and did not maximize coordination with others. As outlined in Figure 11 SATIH’s entire structured trade 2018 budget was a small fraction of the estimated need in Zambia alone. 23 The rest of the Agribusiness Trade programming focused on addressing issues in trade￾oriented value chains; much of this work was independently relevant but did not appear to feed into a wider agenda. Activities included grain B2B market linkages, beef sector assessments in Botswana to leverage AGOA, fresh produce market investment facilitation, and promoting a Zambian feed manufacturers advocacy group. These examples addressed very relevant needs and held potential for positive outcomes. The fresh produce market, for example, is a proven concept in South Africa and was being undertaken alongside leading local non-profits. Similarly, the grain B2B market linkage event helped enable $10.2 million in trade. And given that Botswana is already exporting 9,000 tons of beef a year to the EU, with targeted support it may have potential to tap into the U.S. market. 24 However, although each of these value chains has a regional or global trade element, they are largely independent of one another; 23 Estimated programming needs come from IAPRI, Draft Report Feasibility for Investment Enhancing the Zambian Warehouse Receipt System and Aligning the Food Reserve Agency Strategic Plans to the System, 2019; it is important to note both that this is a draft report and that it is notoriously difficult to accurately estimate the costs of, for example, policy development and implementation. These figures should therefore be taken as directional. 24 European Union External Action website Figure 11: Comparison of Agribusiness Trade 2018 structured trade budget against estimated programming needs in Zambia 0 500 000 1 000 000 1 500 000 2 000 000 19 500 000 Farmer group TA 2018 Structured trade budget Credit guarantees FRA storage certification Establishment of ZAGIS Local aggregation support Agri. marketing review Standards harmonisation Exchange VC assessment Support ZAMACE/LuSE MoU WRS sensitization WRS government committee development USD Commodity Fund Review FRA activities ZAMACE TA WRS policy review WRS legal review IAPRI budget estimates SATIH Agribusiness 2018 structured trade budget Source: IAPRI, Draft Report Feasibility for Investment Enhancing the Zambian Warehouse Receipt System and Aligning the Food Reserve Agency Strategic Plans to the System, 2019 Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 52 there is little sense across these interventions of the pursuit of a systemic regional agenda. Given the data available to the evaluation team, it is difficult to assess the independent results that these interventions create or catalyze. Those involved in implementation suggested that identification of these activities was often ad hoc and opportunistic, and therefore there is no evidence to confirm that these specific value chains, or the specific issues tackled therein, represented the most effective use of SATIH’s resources. Effectiveness: How effectively have Agribusiness Trade interventions contributed to the outcomes delivered? As outlined above, Agribusiness Trade programming was under-resourced relative to the size of the challenges it addressed, and did not sufficiently coordinate with other local development actors. As outlined in Figure 11, the component’s budget was small in comparison to the challenges it targeted. Particularly in the context of insufficient resources, programming required significant coordination with the development community to ensure that the limited resources SATIH could provide would be efficiently integrated into the wider programming of other stakeholders. Based on the evidence available, the Agribusiness Trade team does not appear to have systematically collaborated strongly or effectively with local development actors—and likely was not well positioned to do so, given the multiple countries within the team’s purview and the need for frequent travel to allow for such engagement. Another stakeholder emphasized that SATIH needed to communicate priorities better to enable others to plan around SATIH’s work. This is reflected in the fact that only half of the Agribusiness Trade stakeholders the evaluation team engaged with had a good understanding of what SATIH was and what it did. Agribusiness Trade activities tended to undertake ‘once-off’ interventions that did not build on one another. The component often focused on relevant issues. However, these activities typically failed to achieve significant outcomes due to capacity or legislative constraints elsewhere in the system. The WRS workshops, for example, generated momentum in that a major processor expressed interest in using the system and needed its warehouses certified. However, a lack of capacity at the Zambian Commodity Exchange (ZAMACE) meant follow-up was weak, and the processor did not obtain certification. Similarly, grain value chain actors reported that a SATIH market information workshop equipped them with a clear business model and plan for an independent market information service provider. However, capacity constraints among the local actors needed to drive implementation mean that very little progress was made after the meeting. Both instances were opportunities for SATIH to step in and provide support; this did not happen. Half of Agribusiness Trade interviewees called out a lack of follow-through as an issue – the highest of any component (see Figure 9). Ultimately, by choosing such a disparate set of activities the Agribusiness Trade component both invested little in each activity and did not build any momentum or competencies around a focused set of issues. Execution: How well-executed were interventions under the Agribusiness Trade component? The quality of Agribusiness Trade’s execution contributed to fewer than expected outcomes. Agribusiness Trade’s execution had to overcome several challenges. The Agribusiness Trade team was hired roughly six months after SATIH started, which meant that planning and execution in the first year was delayed. Thereafter, staff turnover in year two meant Agribusiness Trade did not follow up effectively on many of the interventions they initiated. Furthermore, the team’s structure meant that there was a lack of on-the-ground presence; interviewees highlighted this a contributing factor in the Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 53 component’s poor communication, insufficient understanding of local political economy factors, and weak local networks. 8.1.1.2.2. Export Competitiveness To increase exports from countries within Southern Africa to global markets, SATIH undertook two main sets of tasks. One set focused on information sharing and technical assistance to help businesses meet export requirements, along with direct facilitation of trade linkages, while the other focused specifically on increasing AGOA utilization in Botswana, Malawi, Mozambique, and Zambia, typically supporting ministries of trade in developing and promoting AGOA strategies. Outcomes: What contributions to SATIH’s outcomes did the Export Competitiveness component make? Trade linkage efforts have enabled new transactions, contributing to SATIH’s exceeding export value targets. The value of exports in targeted non-agricultural and agricultural commodities resulted in $15.4 million worth of exports over FY2017 and FY2018, exceeding Y1 and Y2 targets (although the LoP targets assume a significant ramp-up in exports over the remaining duration). However, reporting shows trade shows and inwards buyer missions are falling short of their targets in creating linkages between buyers and sellers 25. In FY 2017 and 2018, SATIH organized a total of six trade shows and six inwards missions, which gave rise to 623 buyer and seller linkages in targeted non-agricultural sectors. This number falls short of the target number of linkages over that period of time (705 in FY2017 and 2018 combined). The lag time associated with AGOA support makes it difficult to assess outcomes at this stage. From an output perspective, SATIH provided technical assistance to Botswana, Malawi, Mozambique, and Zambia. AGOA trade volumes remain low, but lag time between the policy being created and this yielding trade under AGO means that it is too soon to assess whether the national AGOA strategies that SATIH contributed to will shape this. That said, the evaluation team did assess the effectiveness of the support, which will offer indications of how likely these outputs are to yield strong outcomes. Ecosystem: To what extent is the work of Export Competitiveness Component set up for success when reviewed against the wider ecosystem and its challenges? The component has placed significant focus on trade under AGOA and with the U.S., in addition to driving trade within the Africa region. With this, the evaluation team analyzed the extent to which there is potential for trade – setting out here a set of barriers that impact both trade intra-regionally and with the U.S., although are exacerbated in the case of U.S. trade. With the exclusion of South Africa, less than one percent of the total trade from the target countries is destined to the U.S.. Analyzing data from 2013-2018, exports to the U.S. (both AGOA and non-AGOA exports) only account for six percent of total trade from Botswana, Lesotho, Malawi, Mozambique, Namibia, eSwatini, Zambia and South Africa combined. 25 Trade shows host buyers and are attended by potential exporters. They are often hosted in buyer countries (e.g., the U.S.). Inwards buyer missions host potential exporters and are attended by buyers. Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 54 This reflects underlying competitiveness issues the region, with wider structural challenges limiting the ability of Southern African countries to cater to international markets, particularly in higher value products.26 This is particularly true of the U.S. – with all interviewees that had a perspective on the topic citing competitiveness concerns as a key barrier to trade...27 Specific examples raised by interviewees include the high cost of transport, high costs of production, and costs associated with meeting standards – these mirror constraints raised in a previous ISP team assessment of U.S. trade, presented in Figure 12. 26 N. Verter, International Trade: The Position of Africa in Global Merchandise Trade,2017 27 12 interviewees Figure 12: Key constraints to Southern African trade with the U.S. Source: ISP team analysis Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 55 Further, SATIH target value chains are often already served by closer countries, weakening the case for this type of value chain support. Previous ISP team analysis revealed that FtF Africa countries are not competitive in U.S. markets compared to current trading partners in Latin America and South East Asia – with these countries accounting for the majority of U.S. imports in target value chains (as outlined in Figure 13). These trade differences are, at least in part a function of Southern Africa’s greater distance and the associated higher cost of trading. Effectiveness: How effectively have Export Competitiveness component interventions contributed to the outcomes delivered? Information sharing, technical assistance (TA), and trade linkages Linkages and awareness support are still of value in stimulating trade where businesses have a competitive offering. Interviewees from national and provincial trade and investment promotion authorities in Lesotho and South Africa, for example, cited the lack of access to information (export standards, trade fairs list, etc.) as a limiting factor in the ability of SMEs in their areas to take full advantage of available trading options. Similarly, these organizations noted that linkage work connects businesses to buyers that they would otherwise not have had access to. This firm-level assistance also has a market systems effect of raising the profile of Southern African SMEs in the U.S. International literature supports this view, with studies making the case for the value of trade shows as a means to gather information, market, and develop networks. 28 Across the board however, a pre-requisite for businesses taking advantage of these events is that they are in a position to competitively serve the market – this is at times challenged by the underlying competitiveness issues described above. The lack of disaggregated data reduced the ability of the evaluation team to assess value for money of events, although businesses reported inwards missions to be more effective than trade shows at facilitating linkages. Interviewees, both from the private sector and particularly senior officials from trade and investment promotion authorities, consistently described inwards missions as more effective at facilitating linkages than trade shows from the standpoint of producers. Inwards buyer missions allow buyers to visit more businesses in their area of focus, to have more time dedicated to sellers to understand their products, and to engage in more detailed conversations and networking. 28 Evers and Knight, Role of international trade shows in small firm internationalization: a network perspective, 2008; Tafesse and Korneliussen, The dimensionality of trade show performance in an emerging market, 2009 Source: ISP team analysis Figure 13: U.S. imports of the top 10 animal/vegetable products, USD millions Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 56 Interviewees also highlighted the need for more permanent representation in the U.S.. One area for SATIH to explore could be having permanent staff based in the U.S. representing and marketing for Southern African businesses, with multiple interviewees calling out that they would benefit from this type of arrangement, as the cost of travel to the US to forge relationships can be prohibitive. This again will only be effective for businesses that can overcome the competitiveness hurdle outlined above. AGOA support Interviewees noted several barriers that limit the potential of AGOA national plans to drive Southern African exports to the U.S. Besides the underlying lack of competitiveness of Southern African products, as outlined above, there are other constraints that limit the potential of AGOA national plan’s effectiveness as a tool to drive trade. These include: • Lack of government capacity to implement national AGOA strategies – with one ministry of trade noting that the AGOA strategy was not a key priority within the ministry, and therefore few resources were available for implementation. More broadly, ministries consistently noted having challenges finding the capacity needed to implement the AGOA strategies • Limited remaining life of AGOA – given AGOA is in effect until only 2025, this represents a short time frame for major industrial investments—a barrier identified by one ministry of trade Execution: How well executed were interventions under the Export Competitiveness component? Businesses and associated stakeholders credited SATIH with well-executed events and the provision of high-quality technical assistance for both AGOA and linkage support. Participants of information sharing, and technical assistance events reported that the presenters were very knowledgeable about export standards and pedagogical in their approach to knowledge transfer. They were able to distill complex information on standards and certification processes and answer questions with practical advice for companies. Similarly, companies that participated in trade shows and buyers’ missions praised SATIH for its professional organization of events. Stakeholders recognized AGOA strategy support as valuable but offered constructive feedback on ways in which SATIH could increase momentum around and improve uptake of AGOA strategies. Stakeholders gave broadly positive feedback on the quality of analysis and strategic recommendations in AGOA national strategies. Moving forward, SATIH could consider how to position itself to ensure better momentum behind the strategy within relevant institutions. In one interview, a senior government official suggested that SATIH did not consult ministries of trade to agree on the best working approach to promote transfer of skills and build internal momentum as part of the AGOA national strategy development. Similarly, SATIH’s consultants mostly worked remotely, with limited time in country. Moreover, the USG’s slow turnaround with U.S. approvals and validation of data sets to include in the national AGOA strategy (comparing statistics of different platforms) led to further delays. Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 57 8.1.1.2.3. Finance and Investment The Finance and Investment component covered two broad types of activities: 1) providing transaction support services to investors through a sub-contractor, CrossBoundary, in order to facilitate investment in the region and 2) providing technical assistance and support to financial service providers (FSPs) to encourage lending. The second form of support included facilitating the use of a European Investment Bank (EIB) lending facility for commodity warehouse construction, and work with FSPs to use the DCA facility as a means of encourage lending. Many, but not all, of the DCA conversations oriented around lending within the warehouse receipt system (WRS). Outcomes: What contributions to SATIH’s outcomes did the Finance and Investment component make? Transaction support and warehouse financing work made significant contributions to SATIH’s overall progress on indicators. Transaction support work enabled $113 million in facilitated transactions, while uptake of the EIB lending facility contributed 82,642 metric tons under the indicator covering metric tons of installed storage capacity. 29 Efforts to boost lending proved less successful. SATIH teams engaged with a range of FSPs in the region to enable lending. There are examples of success in this regard. In Malawi, SATIH support contributed to FMB lending to through the WRS. Similarly, technical assistance to Absa in Botswana for SME lending was viewed as successful by SATIH. Beyond these isolated instances, however, there has been little progress despite a significant volume of conversations between SATIH and potential partners – particularly in utilizing the DCA to enable WRS lending. Ecosystem: To what extent is the work of the Finance and Investment component set up for success when reviewed against the wider ecosystem and its challenges? The work of the Finance and Investment component addressed real needs expressed by stakeholders, where access to finance was critical yet missing. In facilitating investments, all seven investors interviewed recognized the need for strategic due diligence support, noting the difficulty of making small-deal-size investments without it. In facilitating lending, several banks noted that warehouses represent poor collateral as they have poor re-sale value, limiting lending in this space. Malawi-specific data shows that there is a significant need for further storage, particularly in rural areas – with the vast majority of existing storage space existing in the Lilongwe and Blantyre areas. 30 Furthermore, lack of both storage capacity and access to finance across the WRS as a whole are factors that the World Bank notes are inhibiting the efficiency of structured trade for farmers. 31 Although the outcome data reflects the transactional successes, many of the bigger barriers to investment and lending remain unaddressed by the work of SATIH. In line with the broader challenge outlined in the overall SATIH relevance findings, this limits the potential of Finance and Investment to deliver greater results in a more systemic and sustainable manner. 29 DAI indicator data 30 ACE, Warehouse Receipt System in Malawi: a strategy - a solution, 2012, p4 31 World Bank, Malawi Agricultural Commercialization Project, Project Information Document/ Integrated Safeguards Data Sheet (PID/ISDS), 2017 Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 58 For example, on the investment side, investors note that although the subsidized nature of the transaction support does enable them to invest smaller amounts than are typically viable given transaction costs, there are systemic challenges that limit greater investment flows. Multiple investors highlighted examples such as:32 • The need to deliver USD denominated returns (and a lack of hedging products) • Funding availability (low institutional investor allocations towards private equity in the region, as well as information asymmetries that deter international investors from considering investing in Africa) • Costs of other due diligence (financial, legal, and tax), and the quality of investment pipelines (a scarcity of investor-ready businesses) • The quality of investment pipelines (a scarcity of investor-ready businesses) SATIH has undertaken few interventions to address these challenges. This is also true in the case of support provided to FSPs to enable lending. In discussing this support, banks noted that structured markets improved lending in non-staples (e.g., tobacco and sugar), but the political interference that remains in staples (around which most structured trade programming focused) means the residual risk remains high and therefore lending is difficult. Effectiveness: How effectively have Finance and Investment interventions contributed to the outcomes delivered? The three distinct types of intervention have had differing levels of success—although many of the challenges identified for SATIH as a whole (e.g., weak additionality in places, as well as limited effectiveness due to unresolved upstream and downstream challenges) also apply to Finance and Investment. Transaction support has been instrumental in enabling smaller deal-size investments, although the additionality of this support appears to lower in the case of larger deals (albeit they do contribute to a market signaling effect). All deals had a strong development narrative. This reflects the use of a fit check criteria that SATIH assessed deals against aspects such as size, feasibility, gender, and additionality. As the report elaborates on below, the one gap in the fit-check could be the lack of a deal breaker criteria, which fails to de-select large transactions with a high likelihood of taking place in the absence of SATIH’s support. The six deals reviewed under $3 million were all highly additional—particularly for deals in which the technical or operational complexity of the business made internal due diligence processes difficult given lack of expertise. Of the four deals reviewed over $8 million, there was a moderately high or high likelihood the investor could have absorbed the transaction costs. CrossBoundary notes that these larger deals play a market signaling effect, building precedent and therefore encouraging other investors into the continent (addressing one of the systemic challenges identified above). Given the likelihood that investors would pursue these larger deals in the absence of SATIH support, it remains unlikely that SATIH subsidizing this cost yields additional outcomes. Warehouse financing under the EIB facility played a crucial and additional role, given that the lack of a strong lending business case for warehouse construction made blended capital essential. Banks noted that because their resale value was low, warehouses typically made for poor collateral and commercial lending for their construction was difficult. Warehouse financing thus 32 Each point was brought up in at least two of the seven interviews with investors Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 59 represented a desired tool. It was ultimately well received—interviewees who had received these loans gave mostly positive feedback, recognizing the both how warehouse financing benefitted their businesses in terms of improving supply chain security throughout the year and the role this support played in providing farmers with more certain demand for their crops and the ability to access better prices. However, despite the promotion of the DCA facility the underlying risk profile remaining unresolved, which resulted in limited progress on warehouse receipt financing. Only half of FSPs believed that the DCA had the potential to increase lending, and all noted that it would not deliver a sustainable increase in lending beyond the life of the DCA facility. The DCA is an important tool in improving the integrity of the WRS process, with one bank noting it encouraged them to consider WRS lending. That said, SATIH’s promotion of WRS alone proved insufficient to significantly improve lending conditions for smallholder farmers – reflecting the broader need to tackle structured trade in a holistic, coordinated manner. SATIH has attempted to combine the DCA with more innovative de-risking products to encourage lending. FSPs noted that these ideas have potential, although they are yet to gain traction— in part, as suggested by USAID’s DCA team, due to the fact that there has not been sufficient time within the LoP to date to fully design and launch a DCA-underpinned product (which is a significant undertaking). Execution: How well executed were interventions under the Finance and Investment component? Execution of the transaction support work delivered through CrossBoundary (a DAI sub￾contractor) was highly praised. All seven investors interviewed were extremely complimentary of CrossBoundary’s work, praising the team, the smoothness of the process, and the quality of the final output, which informed decision making by confirming existing assumptions and uncovering risks and opportunities. Only two investors raised minor concerns. One pointed out a need for more senior input (other investors commented that senior input had elevated work from ‘good’ to ‘exceptional’). The other remarked on the relatively slow pace of starting the work compared to other providers—which, this investor speculated, was likely related to USAID’s internal approval processes (although other investors commenting on the approval process felt it was pragmatic and fair given the support USAID was providing). In the case of wider Finance and Investment work, stakeholders were complimentary of technical skills, although they noted a lack of follow-up and formalized engagement structures in the case of support to FSPs. This work was predominately executed through a single short-term technical advisor (STTA) that was generally well regarded for their deep technical expertise. However, concerns were raised around the sufficiency of formal structures around this support – noting it was difficult to plan around the support without a formal terms of reference in place. This also led to low levels of awareness of SATIH as a whole beyond the STTA stakeholders were engaging with (70 percent of the 26 individuals interviewed under this component had a low or moderate understanding of the wider hub), while, as indicated in Figure 14, actors noted a lack of follow-up in this component (with 50 percent reporting low levels of follow up). Figure 14: F&I Follow-through Source: SATIH stakeholder interviews; n=20 Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 60 8.1.1.2.4. Trade Facilitation and Enabling Environment Over 20 STTA arrangements executed SATIH’s work under Trade Facilitation, the large majority of which focused on providing technical support to governments or regional bodies, while the remaining quarter focused on wider landscaping efforts or baselining work around the time and cost to trade. In evaluating this work, the evaluation team engaged with BURS, MRA, SADC, and SATIH employees working on the Zambia buy-in providing good coverage across the STTA assignments. Examples of this work include SATIH providing the following support: • The Malawian Revenue Authority over three rounds of support to help design the AEO set-up, including a review of an internally developed proposed design and work plan, and further reviews partway through the design and implementation processes • SADC to design its Trade Facilitation Program—including scoping of various activities intended to support trade across the region • Botswana Unified Revenue Service—to support the development of legislation underpinning Botswana’s National Single Window program SATIH has conducted additional Trade Facilitation work for other similar actors, including in Namibia, South Africa, and eSwatini. Further, several workshops also focused on building awareness of specific trade facilitation measures and complemented this TA support. Outcomes: What contributions did the Trade Facilitation and Enabling Environment component make to SATIH’s outcomes? SATIH’s indicator data reflect positive outputs from the work of Trade Facilitation, but do not tie these to outcomes. FY17 and FY18 indicator data demonstrate that SATIH analyzed, consulted on, drafted, or revised and then approved and implemented with USG assistance 23 enabling environment policies. However, these data track what should be considered an ‘output’ indicator that drives the ‘outcome’ of reduced time and cost to export. The delays on the baselining study and the fact that outcome monitoring is still ongoing mean that these indicator data do not give a full picture of the effectiveness of these interventions.33 Given this gap in the data, the evaluation team set out to assess these efforts by reviewing a sub-set of interventions. Ecosystem: To what extent is the work of the Trade Facilitation and Enabling Environment component set up for success when reviewed against the wider ecosystem and its challenges? The size of the trade facilitation challenge, combined with relatively small SATIH budget, limits the ability of SATIH’s work to drive a systemic impact. SATIH spent $2.8 million over two years on the Trade Facilitation component overall. However, the costs of implementing trade facilitation measures can be orders of magnitude larger 34. COMESA, as a parallel example, is spending $48 million on 33 It is noteworthy that the delays in the baselining data relates to the difficulty sourcing time and cost to trade data in a manner that meets USAID’s requirements, which are based around being able to identify both size and contents related factors. This has yielded otherwise technologically sound solutions that utilize GPS data insufficient in that they cannot provide this level of disaggregation in their current form. 34 Trade facilitation is the simplification, modernization and harmonization of export and import processes and can include a broad range of activities – including harmonizing transit charges and vehicle dimensions across countries, one stop border posts, and harmonizing standards. UNECE, Trade Facilitation Implementation Guide, 2019 Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 61 an ongoing trade facilitation program funded in part by the EU. 35 For a single country, the OECD notes that budgeted or estimated capital expenditure to introduce trade facilitation measures ranges from $3.9 million to $21.3 million. 36 Stakeholders have been skeptical of the extent to which the funding required to see this policy change to fruition is available beyond SATIH. Twenty-nine stakeholders interviewed explicitly referenced policy implementation challenges. Meanwhile, SADC has faced challenges funding its trade facilitation program, and also notes that donors are typically focusing on ‘soft instruments’ (technical assistance and capacity building), while there is much unmet need for support on acquisition of ICT or physical infrastructure. This sentiment concerning the type of support donors are prepared to offer was echoed by an official in a Ministry of Trade, who noted that SATIH’s work in the past has been “remote controlled TA,” whereas what is really needed is “boots-on-the-ground support” to enable things like business process re-engineering. These gaps in funding further underscore the need for SATIH to be selective in only providing trade facilitation technical support where it can be certain that sufficient implementation capacity exists in the system. Effectiveness: How effectively have Trade Facilitation and Enabling Environment interventions contributed to the outcomes delivered? Most stakeholders noted that the type of support provided was instrumental, although insufficient, in improving time and cost to trade. This reflects the fact that the nature of the support focused typically on high-level design, legislative drafting, and program scoping. Following this initial work, there is still significant additional resources required to see it through to implementation and achieve the full benefits (ultimately, reduced time and cost to export). Although it was insufficient alone, stakeholders noted that the deep technical expertise provided resulted in more robust designs. For example, in the case of the Malawi Revenue Authority, the TA ensured a more robust design and scope, along with a more realistic implementation plan for the pilot of the Authorized Economic Operator program. Botswana Unified Revenue Service noted how difficult it would have been to access the quality of legal expertise provided by the STTA without the support of SATIH, without which progress would have been slower. Stakeholders reported tangible benefits associated with some of these activities. The Malawi Revenue Authority reported that the Authorized Economic Operator program, currently in pilot stage, has been well received by the 5–8 private sector participants and that it has significantly improved the ability of those companies to move goods across borders. Botswana Unified Revenue Service also confirmed that traders can now trade online through the national single window. These examples illustrate the potential for this type of trade facilitation to work when there are sufficient internal resources available on the recipient’s side to fund implementation, or alternatively there are other development partners prepared to provide further funding. The lack of outcome-related data limits the extent to which the evaluation team could validate the impact these initiatives have had, beyond what stakeholders were able to self-report. The lack of implementation capacity has been an issue in other cases, however. For example, SADC continues to court funding for its Trade Facilitation Program, limiting the extent to which SATIH’s work in this space has had an impact. SATIH must be selective in the trade facilitation projects it takes 35 tralac, “COMESA, EU sign €48 million trade facilitation programme agreement”, 2018 36 OECD, The Costs and Challenges of Trade Facilitation, 2012 Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 62 on—the insufficiency of the support it is able to provide means that it must ensure that the agencies it supports have the capacity and/ or funding to see through the technical support that it has been offering. It is worth noting that the Zambia’s buy-in is more focused on implementation than design, with greater reliance on SATIH to support the implementation of policy. This work has also been dependent on the alignment and buy-in of a wider group of stakeholders, however, and therefore does not fully avoid the challenges associated with implementation capacity in the wider ecosystem. There have been further suggestions that, in light of these challenges, there may be alternative private-sector-led approaches that may yield more effective and efficient results. Specifically, USAID/Malawi suggested that trade facilitation issues could be addressed in the context of specific exporters and their problems. This was echoed by a Malawian private sector company, which noted specific examples of private-sector-led trade facilitation it has engaged in, including collaborations with: • Malawian government ministries to reduce NTBs in the case of dairy, where Zambia requested phytosanitary requirement adherence to be proven on an animal-by-animal basis Outcome: Requirements were eventually loosened following representations around the issue between the Malawian and Zambian ministries • The Malawian Investment and Trade Center (MITC) to lobby for improved processing times for health and veterinary permits Outcome: This resulted in reductions from six days to one to two days processing time • The Malawian Innovation Challenge Fund (MICF), a DfID and UNDP co-funded program, to bring together private sector companies to lobby against VAT increases. Outcome: The program reporting notes that this intervention involved leading a “team of representatives of milk processors who lobbied for the removal of VAT on locally processed milk which had negatively affected the sales of milk. The reduced sales subsequently forced the companies to reduce their milk buying prices, thus negatively affecting farmers as well.” 37 In these cases, work at the firm level (although not conducted through SATIH) has shown the potential to feed into a wider market systems approach, and resulted in tangible benefits, with smaller effort and resource required compared to working directly at the systems level. The willingness of businesses to contribute to these efforts also provides a level of assurance that they are indeed issues that are significantly hindering businesses success and will therefore contribute to strong outcomes. Execution: How well executed were interventions under the Trade Facilitation and Enabling Environment component? The quality of execution for trade facilitation delivered through Nathan Associates (a DAI sub-contractor) has been high. Recipients of support noted the strong technical expertise of the consultants, which in turn either enabled either stronger designs or allowed work to be completed at a faster pace owing to greater technical expertise than would be available in house. Workshops have also been well received— 82 percent of participants find the events useful or extremely useful, 72 percent of participants attest to performing their work better following the events, and 100 percent of participants 37 Nathan and Associates / Malawi Innovation Challenge Fund – Annual Report 2017: Manufacturing and Logistics Window, February 2018 Southern Africa Trade and Investment Hub (SATIH) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 63 agree that the events were well structured, the event venues were well located, and that they benefited from the presence of other participants. Some concerns were raised in the case of the Zambia buy-in, where stakeholders noted a need to fully align the activities of a broad spectrum of stakeholders involved in this work— an effort that had been slowed by staffing turnover. That said, this is an issue that the implementer is aware of. The sub-contractor has stated their intention to address this by ensuring they have the right mix of skills in country to advance the work. Southern Africa Trade and Investment Hub (SATIH) – Management USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 64 8.1.2 MANAGEMENT Scope of evaluative theme From the early stages of this evaluation, both USAID/REGO and the implementer identified management challenges as a major theme. The evaluation team’s engagement with in-country stakeholders reflects the fact that management challenges are a major theme, with deteriorating relationships between implementers and local stakeholders, low levels of follow-up, and weak coordination between implementation teams all noted at various points (as outlined under “execution” in 8.1.1 – Relevance). This section looks at the how the overall management model (e.g., contract structures, people, processes, systems) within both USAID/REGO and the implementer contributed to these challenges. Synthesized findings The evaluation team found that the contract structure itself played a crucial role and was not effectively set up to ensure excellent performance (owing both to a lack of management tools, weaknesses in measurement indicators, and the breadth of outcomes specified in the contract). The management set-up, including high CoP turnover combined with competing interests within USAID/REGO, reduced the ability of SATIH to correct earlier performance issues. Furthermore, many of these issues identified under relevance stem from shortcomings in the internal operations of SATIH— including weak prioritization processes, a siloed organizational structure and poor interfaces with bilateral missions. This drove diseconomies of scale, where SATIH’s components programming in isolation added operational complexity (e.g., an increase in operational management decisions and approvals), without maximizing the impact of its work as outlined in the relevance section. The use of monitoring, evaluation, and learning (MEL) data was also insufficient, with little collaboration between technical teams and MEL teams (particularly in the early phases of the project), resulting in limited use of MEL data and learnings to shape the workplan, and discrepancies in reporting given that technical teams were at times using different datasets to the core PMEP data to inform reporting. It is important to note that USAID/REGO and the implementer have already recognized many of the challenges described below, and are in the process of taking steps to address them (as outlined at the end of this section). These efforts took place predominately outside of the evaluation period, and are also too recent to be able to reach a conclusion on whether they are likely to fully resolve the issues identified here. To what extent did the SATIH contract structure and set-up enable success? SATIH and USAID/REGO agreed to a performance-based contract, which, if well designed, could have helped USAID leverage implementer expertise and incentivize results. TFSP’s contract structure—which used a Performance Work Statement (PWS) to summarize the outcomes to be achieved—provided high levels of flexibility on how those outcomes could be achieved. This flexibility is designed to allow the implementer to draw on its extensive expertise to shape the program. However, the evaluation team noted that three conditions need to be in place to enable an effective performance￾based contract: Southern Africa Trade and Investment Hub (SATIH) – Management USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 65 1. Outcomes are clearly defined and achievable 2. A measurement mechanism is in place that provides good insight to the contract issuer on the extent to which outcomes are being achieved 3. The contract issuer has a set of tools that can effectively hold the contractor to account for outcome delivery In this case, there were gaps in relation to all three prerequisite conditions. The nature of the contract provided a very broad set of outcomes with insufficient specificity in terms of what USAID was looking to achieve. For example, the IRs reflected a broad set of long￾term outcomes, while the indicators could be achieved in many ways. Several stakeholders, including the implementer, noted that the breadth of a single outcome indicator could lead to many strategies—for example, increasing investment could be achieved through equity investment, debt investment, or foreign direct investment, each of which would lead to a very distinct set of activities with trade-offs around each option. Although USAID may have desired the input of the implementer in making these trade-offs, the challenge was that many of these trade-offs were never explicitly articulated, by USAID making it difficult to hold the implementer to account for a specific set of desired outcomes. Both the outcomes defined and the indicator framework reduced the extent to which USAID had the appropriate information for monitoring and appraising performance. As outlined further below, annual reporting has often been inconsistent (with component-level summaries written in an isolated manner from Performance Monitoring and Evaluation Plan [PMEP] data) and the poor logical linkages between outputs and outcomes limit the extent to which metrics provide a good ‘early warning system’ that can help guide USAID management actions. USAID also felt they had few tools at their disposal to manage the contract effectively. Although the contract was directionally performance-based, fees were not tied to outcomes. In addition, USAID’s interventions were often limited to inputs around key personnel selection and workplan sign￾offs. Members of USAID/REGO expressed that they felt they had few tools at their disposal. Although not sufficient to cause management issues, in light of wider challenges (outlined in relevance and below in management) the contract did not put in place effective structures that provided the right accountability and contract management structures. To what extent did SATIH’s management model support effective programming? This section of the report will look at the key elements of SATIH’s management model: organizational structure and processes, key personnel, interfaces, KPIs and indicators and MEL implementation. Processes and structure The limitations of prioritization processes used in SATIH’s early days contributed to many of the challenges identified in Section 8.1.1 – Relevance. Interviewees involved in implementation reported an opportunistic prioritization process, particularly in the early days of SATIH; according to these accounts, decisions were made based more on staff’s past experience than on any rigorous “fit check” or prioritization criteria. This led to the implementation of tasks with no prior landscape mapping or anticipation of challenges, a lack of a “systems thinking” perspective, and weak identification of synergies. Southern Africa Trade and Investment Hub (SATIH) – Management USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 66 SATIH’s internal ways of working contributed to siloed programming and diseconomies of scale, against what is a significant potential set of synergies. Organizing the teams by component, combined with siloed ways of working, weakened SATIH’s ability to realize synergies between components. Interviewees involved in implementation also identified communication challenges across components in certain focus areas in the early days of SATIH, which was not conducive to information sharing and cross￾component programming. Teams sought to optimize their component alone. As a result, although there were significant opportunities identified, teams did not use tools from other components to address the various upstream, downstream, or parallel challenges that occurred alongside their focus areas – weakening the overall efficacy of their programming. As outlined in Figure 15 – interviews presented a wide range of missed opportunities for cross-component programming. This also led to greater levels of operational complexity and approvals in the hub; each component was focused on a different set of priorities, each of which generated a volume of operational approvals and decisions likely greater than had the components shared a common set of stakeholders and built on the work of other components. This culminated in diseconomies of scale at the COR and COP level, where the lack of complementary programing requires these roles to work across a greater breadth of tasks and relationships than was necessary. The lack of in-country coverage and resource constraints contributed to a lack of follow-up. As outlined in 8.1.1 – Relevance, this is a recurring theme across SATIH. Only 13 interviewees rated SATIH’s follow-through as high, against 30 interviewees rating it as low or medium. (This is particularly notable in contrast to the implementation of STP—where the in-country advisors have enabled close and regular follow-up with stakeholders.) Interviews with those involved in implementation noted the need for greater in-country resources to enable better local relationship management. Further, as outlined in, the SATIH relevance section of this report, where the Export Competitiveness component had more localized presence, it delivered markedly better follow-through compared to other teams. Key personnel High CoP turnover added complexity to relationship management and hindered the ability of SATIH to establish a long-term vision. High CoP turnover has contributed to losses in momentum, and multiple different visions for the program. It also means relationships consistently needed to be redeveloped, both with USAID and external stakeholders. The importance of a strong CoP was elevated when budget cuts were made and changes to program needed to be implemented - leading the program through structural changes and re-prioritizing program activities in the face of budget cuts adds 8 24 Multiple component engagement Unmet opportunity for multi-component programming Number of interviewees that interacted with more than one component vs number where there were clear examples of complementarities that were not leveraged (n=32) Figure 15: SATIH interview codification – cross￾component programing Source: SATIH stakeholder interviews; n=32 Southern Africa Trade and Investment Hub (SATIH) – Management USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 67 a further degree of complexity to program management. The implementer acknowledges that they did not act sufficiently expediently nor provide sufficient leadership when budget challenges first arose. Competing USAID/REGO priorities, as well as ad hoc management interventions by USAID/REGO, hindered management of the contract. The fact that funding come from different sources meant that competing priorities within SATIH were almost inevitable. These conflicting priorities came to the fore during budget cuts—at which point USAID/REGO offered insufficient direction on trade￾offs or focus areas, making it difficult for SATIH to prioritize. Interviews revealed that, post-budget cut, Activity Managers often had differing views on what should be a priority. Similarly, interviewees also noted instances in which informal direction was given outside the Contracting Officer’s Representative (COR), further complicating prioritization. Systemic knowledge management was also lacking, which was increasingly crucial in the face of turnover, siloed ways of working, and low levels of in-country presence. Several of those involved in implementation noted this challenge, noting poor handovers between incoming and outgoing staff and a failure of SATIH to build on successes and relationships from previous iterations. Much of this could have been avoided had effective processes and systems been in place (e.g., clear relationship owners) to track relationships and interactions. Interfaces The implementer and USAID/REGO noted that SATIH had engaged the bilateral missions; however, interviewees at both organizations deemed that engagement to have been insufficient. This insufficient engagement manifested in limited synergies between SATIH’s work and that of the bilateral missions. Missions reported insufficient visibility into SATIH’s work and planning, which reduced their ability to incorporate SATIH activities into their own planning cycles. Furthermore, they pointed to missed opportunities for complementary activities to take place—one mission noted the need for a complementary SME development program, in which engagement of participating SMEs with SATIH serves almost as a “graduation point” from the program. Overall, this insufficient engagement points to an opportunity for planning to become a co-creation process, rather than socialization of SATIH priorities. KPIs and indicators Earlier sections of the report identified challenges associated with the indicator design that both impacted the effectiveness of the contract and had implications on the relevance of interventions themselves. These challenges are outlined below in further detail. Because there are few safeguards in place that ensure the additionality of claimed results, it is possible that indicator targets can be met without delivering additional development benefits. As noted in 8.1.1 – Relevance, additionality was a challenge in places (e.g., larger transaction support deals that likely would have occurred in the absence of USAID support). Across the program, prioritization tools / criteria / “fit check” scoring mechanisms were not sufficient to counter these challenges—in part due to the lack of “deal-breaker” criteria 38 to prevent, for example, non-additional successes, or successes with lower associated development benefits. 38 IE criteria that, at a certain point on a continuum give a go/ no-go option on an activity (e.g. on transactions past a certain value, no support is provided) Southern Africa Trade and Investment Hub (SATIH) – Management USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 68 The indicator framework does not explicitly create a logical linkage for how specific outputs feed specific outcomes. This has limited the ability of USAID/REGO to use output indicators as a way of predicting ultimate success against specific outcomes—hindering their effectiveness as a management tool. Giving all indicators equal weight risks creating perverse incentives. Over the evaluation period, the PMEP used a set of 22 outcome and output indicators to track SATIH’s performance (e.g., number hours of training administered, value of investment facilitated). 39 Although output indicators serve a useful role when it is clear how they feed specific outcomes, and therefore become a leading indicator or “early warning system,” the PMEP established output indicators as metrics in their own right, without a clear linkage to a specific outcome. In this context, indicators such as “Number of food security private enterprises… receiving USG assistance” risked encouraging activity without it necessarily feeding an outcome, and risked creating unnecessary limitations (e.g., the requirement to engage broadly, limiting SATIH’s ability to create deeper engagement with a more focused set of organizations). Although the target for this indicator was not met, this breadth of engagement was reflected in SATIH’s interventions. Implementation of Monitoring, Evaluation, and Learning MEL did not appear to play a central role in technical design and program implementation. Those involved in implementation noted that, particularly under previous CoPs, the MEL team was siloed from technical teams and not sufficiently empowered. One comment noted that the role of MEL was limited to generation of content on indicators for reports. Technical teams did not incorporate MEL and data gathering considerations at the point of task design. Through the efforts of the MEL team, data were gathered to inform USAID reporting, but frequently were not used for learning purposes. Several interviewees contrasted this limited role for the MEL team to the East Africa Trade and Investment Hub— which had MEL associates embedded into the technical teams. The secondary role played by MEL is reflected in reporting, where technical teams often report different data points to those reported in PMEP indicator data. This results in a range of data inconsistencies in quarterly and annual reports and work plans that make it difficult for USAID to have a clear picture of SATIH’s achievements. Disaggregated numbers under the components (e.g., number of linkages created) at times do not sum to their totals. Proportioning SATIH-level indicators to specific components adds further complexity, in that these appear to change across work plans and reports and there are no easily accessible component level targets for each indicator. Moreover, the way that activities are labeled and reported on differs significantly from year to year. This inconsistency makes it extremely difficult to track progress on specific items over time—weakening the potential for the reports to serve both as a management and communications tool. To what extent were the budget cuts effectively managed? USAID/REGO did not appear to the implementer to align internally on relative priorities in the face of budget cuts. USAID/REGO staff often cited competing interests at play, which was in part 39 27 indicators were tracked overall, of which 5 were context indicators not used to monitor or evaluate performance in anyway Southern Africa Trade and Investment Hub (SATIH) – Management USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 69 driven by the different sources of funding that SATIH draws from. This reduced the extent to which USAID/REGO and the COR could provide clear direction to the implementer on relative priorities. The implementer put in place significant cost reductions but acknowledged its role in not taking the initiative to lead the discussion sooner. The implementer reported that earlier rounds of budget cuts could be met through fixed cost reductions (e.g., reducing travel, downsizing office space), particularly as the full expenditure cap was not being met. However, the original LoP targets remained in place, despite a significantly reduced budget. The implementer noted that, given that its contract was to deliver against a set of outcomes, it believed it should have addressed these challenges earlier. Additional considerations beyond the evaluation timeframe The evaluation team also noted that, although outside of the evaluation timeframe (i.e., after December 2018), new leadership has taken steps to address many of the management concerns outlined here. These changes include: • Introduction of a new CoP – credited with improving relationships between the implementer, USAID/ REGO, and the bilateral missions • A revised vision and strategy – placing greater focus on integrated programming across the components • Changes in ways of working – including more matrixed ways of working, and encouraging SATIH staff to spend increasing time in the office to drive more collaboration across components • Hiring of country advisors – to improve the ability to follow up and implement more localized tasks. • Improved prioritization processes – which link in with an ongoing review of activities that SATIH is undertaking • Additional ‘roadshows’ to bilateral missions – to strengthen relationships and buy-in Southern Africa Trade and Investment Hub (SATIH) – Sustainability USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 70 8.1.3 SUSTAINABILITY Scope of evaluative theme Sustainability in the case of SATIH sets out to explore the extent to which the benefits of these interventions would be sustained in the absence of USAID support, and the equity of these interventions. Synthesized findings Many of the observations at the component level within 8.1.1 – Relevance reflect on the underlying sustainability of the specific interventions. At the activity-level, sustainability for SATIH varies significantly. There were examples of interventions that serve as proof of concepts that lead to self-sufficient future investments, and cases where initial support led to self-sustaining results. However, frequently there were challenges with either interventions ceasing too soon to become self-sufficient, or else not addressing the underlying challenges that would have enabled more self-sufficient solutions. Furthermore, beyond isolated examples within specific components, there is a limited evidence that SATIH has taken systematic efforts to deliberately target disadvantaged and underserved groups, including the lack of a strong gender lens. Would the results of SATIH be sustained in the absence of USAID support? In some cases, initial support was designed to play a catalytic role, enabling further self￾sufficient investment. For example, trade show costs covered by SATIH serve as a pilot, proving to businesses that there is a business case for self-sufficient investment. Similarly, transaction support for deals in the region has played a market signaling effect that attracts other investors into Southern Africa once the initial deal proves the concept. Similarly, where SATIH has invested in a collective group of private sector companies, this creates awareness and advocacy that can deliver long-term gains. The evaluation team saw this play out in Zambia, where an awareness and training trip to the U.S. for leading grain traders as part of an earlier iteration of SATIH was an important contributor to the development of the influential Grain Trade Association. Although the lead time for these interventions to pay off can be lengthy, in this case it proves that they are capable of delivering sustainable change beyond the LoP. However, other interventions either were stopped before they were able to achieve self￾sufficiency or were too ad hoc to deliver sustained, systemic change. For example, efforts to establish the Zambia Agricultural Grain Information System (ZAGIS) may have ultimately resulted in a sustainable system, but USAID support ceased before ZAGIS had the opportunity to reach this threshold. In the case of significant amounts of SATIH’s work that did not resolve the underlying upstream and downstream challenges—for example, use of the DCA to encourage WRS lending—the unresolved issues limit these intervention’s potential for sustainability. To what extent are the resulted delivered equitable? Anecdotally, stakeholders noted that channeling development support through the private sector yields equitable outcomes—although the limited evidence base available through SATIH restricts the ability to assess this. One example of is the work done with COMACO, a social enterprise that supports wildlife conservation and smallholder farmers. Channeling development support through COMACO ultimately provides a longer-term relationship with the farming communities than an individual program would. In addition, creating greater linkages and exports ultimately allows the Southern Africa Trade and Investment Hub (SATIH) – Sustainability USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 71 communities to realize higher sale prices for their good. Ultimately, however, it is difficult to determine the extent of the inclusion of smallholders, given that SATIH’s data only extend to firm-level attributes. With the exception of standalone examples, there is little evidence of a systematic and deliberate focus on gender or minority groups. Where the evaluation team identified examples of gender-focused work, these tended to be Finance and Investment transaction support work and Export Competitiveness business support. Within transaction support, the investment “fit-check” includes a gender criterion. This contributed to a portion of deals that included a gender lens – specifically around women-owned businesses, women-owned investment firms, and businesses with a significant gender focus. Under Export Competitiveness there were also examples of programming targeting women. These include ad hoc support for the African Women’s Entrepreneurship Program, ensuring a representative sample of women attended overseas trade shows, and bringing a specific gender angle into Botswana’s AGOA strategy. That said, there was limited evidence of a holistic gender strategy or clear ownership of this agenda across SATIH as a whole, despite the PAD identifying gender as a critical priority. Southern Africa Trade and Investment Hub (SATIH) – Conclusions USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 72 8.1.4 CONCLUSIONS The opportunities this report details are a response to the performance observed through the end of 2018, in line with the evaluation period. This discussion takes into consideration SATIH’s relatively strong performance against indicator targets, as well as the gaps identified across the themes of relevance, management and sustainability—including low levels of additionality in some activities, disconnected programs that do not build on each other, unaddressed ecosystem challenges that hinder effectiveness, and management challenges experienced to date. Reflecting on the lessons learned from SATIH, it is worth noting that the implementer and USAID/REGO have already put in place corrective actions to improve performance as part of the 2019 work plan. It is possible, therefore, that some of these recommendations are already under consideration or are being implemented within the program. Ultimately however, the impact of course corrections to date remains unknown, given the lag time between the work planning process, delivery of activities, and outcomes being observed on the ground. In identifying lessons learned and opportunities for SATIH, it is important to note that the variety of programming undertaken under the activity, combined with MEL weaknesses identified, often means that SATIH does not have an extensive evidence base from which to assess the effects of its work or, ultimately, what has or has not worked. This has limited the extent to which the evaluation team can draw clear conclusions. 8.1.5 RECOMMENDATIONS AND OPPORTUNITIES Within the remaining Life of Project (LoP): Cross-cutting Relevance • Continue to build a matrixed way of working within SATIH and more connected programming across components – Enable interventions that bring to bear the tools of the various component teams • Improve relationship and knowledge management / stakeholder mapping – Ensure that SATIH has a clear ‘rolodex’ of its stakeholders, re-prioritize these as the project enters its second half, and use this insight to inform prioritization of activities (connecting to the previous opportunity around bringing the full set of tools to more a more focused set of beneficiaries) • Continue re-prioritization of tasks, considering both the criteria identified in the 2019 work plan and ecosystem and landscape mapping – De-prioritize tasks that are not feasible due to wider systemic challenges that SATIH cannot resolve with available resources, or explore alternative approaches (e.g., partnering with other donors that are working in adjacent spaces) where other gaps need to be addressed to enable SATIH’s activities • Refine prioritization processes / tools – Ensure that activities align with USAID’s desired outcomes; for example, consider scaling the use of “fit check” criteria across the activity (e.g., criteria on additionality or gender inclusivity), and introduce “deal breaker” criteria into these tools where necessary to improve their effectiveness • Drive proactive donor and bilateral coordination – Ensure that this effort is informed by deeper stakeholder and landscape mapping, particularly where donors are working in interdependent areas that SATIH’s work does not fully address • Revise the theory of change to ensure strategic focus – Re-develop intermediate results and objectives to provide clear strategic guidance (e.g., what relative focus is desired within a specific component where various tools are at SATIH’s disposal) Southern Africa Trade and Investment Hub (SATIH) – Recommendations and opportunities USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 73 • Refine the indicator framework – Ensure that outcomes are defined in a way that provides clarity on what SATIH is expected to deliver; develop a more robust framework that ties output indicators (used predominately for management rather than outward reporting) to outcomes so that they serve as an improved “early warning system”. Ensure that the indicator framework places a tighter burden of proof on additionality • Be more deliberate about gender – Develop explicit gender goals and targets and track gender disaggregated data to ensure that goals are being achieved; look to integrate a gender lens in transaction support through—for example, by taking a broader view of gender lens investing using 2X challenge criteria, 40 or by focusing on value chains and sectors that have a disproportionate representation of women (while acknowledging that interventions with higher additionality are often harder to implement; with this, there may be a trade off on ‘quantity’ of results while delivering greater ‘quality’) • Elevate the role of MEL in work-planning – Ensure that the MEL team contributes to technical design, particularly in ensuring deliberate discussion on whether activities are consistent with the broader theory of change Agribusiness Trade • Develop a clear “where to play” vision for Agribusiness Trade – Conduct further scoping to provide much needed focus given the breadth of agribusiness activities in the region, the current budget, and the remaining LoP duration; conduct landscaping and stakeholder mapping as required to inform this exercise • Review structured trade engagement – In light of extensive upstream and downstream challenges, including endemic political economy challenges that require long-term solutions beyond the LoP, review the extent to which SATIH can address these challenges, and how it may fit into the structured trade ecosystem in this context • Coordinate effectively – Coordinating strongly with local actors to identify specific parts of an issue that USAID can play a unique role in advancing—particularly when addressing challenges such as structured trade, which one actor alone cannot resolve; this will ensure that programming is not duplicative with the work of local actors, and that USAID/REGO funding is focused on areas that it is uniquely positioned to address effectively Export Competitiveness AGOA work to date has largely focused on developing AGOA strategies. In light of the structural competitiveness constraints identified in the relevance section of this report, there is a need to consider how much resource SATIH is prepared to invest in continuing to drive AGOA success. Ultimately, any approach for pursuing AGOA trade will likely be costly in the face of these competitiveness challenges. Should AGOA remain a key priority, this report sets out two options for consideration: • Advance the implementation of AGOA national strategies as a private sector strengthening initiative (high investment) – Invest in using the AGOA national strategies as a broader private sector strengthening initiative; this approach likely requires industrial development investments, strengthening of business associations, courting foreign direct investment, and lending capacity to governments to support the ongoing implementation of national strategies. It is important to note that should SATIH attempt to pursue this option, 40 www.2xchallenge.org Southern Africa Trade and Investment Hub (SATIH) – Recommendations and opportunities USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 74 significant investment in implementation support would likely be needed – meaning either more funding or a reprioritization from other activities. • Pursue a focused AGOA trade campaign with AGOA-ready businesses (low investment) – Drive AGOA trade by identifying and picking a smaller set of businesses that are already well set up to trade with the U.S. (likely a relatively small segment of businesses in the region, which may be difficult to source). Doing this effectively likely requires a strong sourcing effort, likely using a “fit-check” type set of criteria in the manner that Finance and Investment do at present to prioritize provision of support to the businesses most likely to deliver AGOA trade Finance and Investment • Re-evaluate desired outcomes of transactions and corresponding fit-check criteria – Although large transactions deliver strong market signaling benefits, their size will often yield lower additionality as investors are more likely to be able to cover transaction costs; define the relative desired focus and adjust fit-check criteria and targets accordingly (e.g., introducing a deal￾breaker criteria on additionality if desired, which would improve additionality but reduce potential value. Place greater emphasis on the complexity of the business itself, which also determines whether investors could conduct due diligence internally.) • Explore use of co-funding mechanisms, particularly for larger deals – For example, where large deals bring a benefit to SATIH (particularly in signaling and driving market systems change) but the costs of transaction support could be sustained by the investor, consider a co￾funding mechanism by which costs are shared between SATIH and the investor • Evaluate relative programming focus based on ecosystem needs / priorities – Evaluate whether the current programming focus areas make sense against broader regional challenges in finance and investment, and whether the current tools are best suited to addressing them (e.g., is the allocation of effort toward supporting private equity and commodity lending in line with both the market needs and USAID appetite?) • Use the Development Credit Authority (DCA) facility as one tool, alongside good product design – Recognizing the limited faith stakeholders had in the DCA as a tool on its own, consider a more focused pursuit of DCA opportunities—combining DCA with deeper product design work that alters the risk profile for banks; although this may yield fewer FSP relationships, given the perception of the DCA as a standalone tool it may have potential to improve total lending under SATIH Trade Facilitation and Enabling Environment • Define a clear set of priority areas where SATIH resources can make a significant difference in reaching time and cost to trade targets: In the context of current project timelines and budget, define specific areas where there is potential to improve time and cost to trade—recognizing that SATIH cannot address full region dynamics; prioritization could take place against a sub-set of corridors, border posts, value chains, or specific issues (e.g., standards or types of technological solutions) • Draw on the expertise of the private sector to inform prioritization – Understand the issues that are truly holding them back from trade and impacting their profitability (e.g., building closer relationships with anchor firms or industry associations that can provide a broad perspective on private sector challenges) • Use adaptive programming approaches and build an evidence base – Particularly when working in specific value chains, combine more adaptive programming approaches with desk research; in the absence, to date, of rich SATIH-specific data on which interventions are most Southern Africa Trade and Investment Hub (SATIH) – Recommendations and opportunities USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 75 effective, this approach could enable rapid progress and inform future programming beyond the LoP Beyond the LoP: • Align on the future focus and targets of SATIH – Create a more focused set of priorities within the TOC that aligns with the budget at SATIH’s disposal • Draw on USAID’s full suite of tools – For example, explore opportunities to work with new USG initiatives such as the United States Development Finance Corporation to bring a fuller set of tools to programming • Link fees to performance if continuing with performance-based contract structures – Performance-based contracts are powerful tools—putting a proportion of fees at risk can create joint accountability for the implementer and additional measures for USAID to ensure strong performance. To effectively leverage this model, outcomes need to be well designed, indicators need to accurately reflect these outcomes, and indicator targets must be well contextualized (which will likely require the input of the selected implementer, along with USAID design capacity) • Integrate data collection and management into project design – Ensure better transparency and improve MEL efficiencies by developing and using data collection tools that gather MEL data on an ongoing basis throughout task implementation. Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 76 8.2 SEED TRADE PROJECT (STP) Summary findings: STP The evaluation team found STP to have been highly effective in driving seed policy harmonization. It pursued a relevant strategy and executed a set of tasks well. This has translated into steady changes in national policies, although there is more work to be done during the remaining life of project (LoP) to translate these policies into enforced regulations and stronger national institutions. Progress on the regional level has been slower—there is no evidence that a sustainable set of SADC institutions is in place, despite STP providing support in this area. From both a relevance and sustainability standpoint, the biggest challenge relates to what STP has not focused on — with gaps in coverage limiting its effectiveness. Further activities are needed beyond the current scope and LoP to fully deliver on HSRS implementation in the Southern Africa region. This includes greater financing support to reflect the scale of resources required – including ‘hard’ infrastructure improvements – and the need to advance HSRS implementation in importing countries. STP’s sustainability is further challenged by the prospects for creating sustainable operational capacity within SADC seed institutions. These institutions, particularly the SADC Seed Centre, are central to cementing and building on STP’s national and regional work beyond the LoP. However, STP and SADC’s current plan to operationalize these institutions will take time to fully realize, and the evaluation team has not seen evidence of a plan for how SADC will fund any shortfalls during this period. Management of STP has shown significant improvement during the LoP, following changes in staff and the relocation of the HQ back to Lusaka. This yielded significantly strengthened stakeholder relationships, both with crucial stakeholders in Lusaka and in focus countries. Given the policy-oriented nature of the program, and the importance of strong relationships in building consensus for and advancing government policy changes, these management improvements have enabled STP to deliver positive results. One can see these positive results in the country-level policy changes, and the capacity improvements that STP has built. The following sections provide more detail and evidence on these overall findings. 8.2.1 RELEVANCE Scope of evaluative theme This section of the evaluation sets out to review the relevance of STP with respective to the ultimate objective of improved agricultural productivity. Does STP focus on an optimal set of challenges within the wider quality seed ecosystem? What is the overall performance of the activity compared with LoP targets? How effectively has it provided support? This section will address each of these topics in turn, beginning with the ecosystem question. This order differs slightly to the one taken in the other activities because the team considered the question of STP’s relevance as an essential first-order question to explore before assessing its outcomes, effectiveness and execution. Synthesized findings Overall, STP appears to deal with a real problem that USAID/REGO is well positioned to address. It is on track to meet LoP targets in a manner that is highly additional. To do this, it has pursued the appropriate strategies and tasks, although performance has been stronger at the country level than at SADC regional Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 77 level. That said, it is important to note that although STP has performed strongly in the areas it has focused on, further work will required beyond the scope and life of the project in order to cement these gains— for example, importing countries will need partners in driving the adoption of HSRS, and governments will need greater levels of financial support in order to fully implement harmonized policies. Ecosystem: To what extent will harmonizing seed policy result in improved agricultural productivity? As an initial step, the evaluation team assessed whether USAID is addressing a real challenge that has significant potential to improve productivity. This assessment entailed reviewing how STP interventions fit into a wider ecosystem of quality seed use. The team found that STP plays a relevant role in improving seed availability. The work of other actors to address a range of gaps in the ecosystem supports the translation of STP’s efforts into improved usage and productivity gains. High quality seeds play a significant role in improving agricultural yields. Seed carries the “genetic potential of crop species and their varieties”.41 As a result, the Alliance for a Green Revolution in Africa (AGRA) and the Food, Agriculture, and Natural Resource Policy Analysis Network (FANRPAN) believe that seed is the most important input to determining agricultural productivity. AGRA states that half of total crop yield is determined by the genetic potential of the crop,42 while FANRPAN states that “seed gives the highest ‘bang for the buck’ of any agricultural investment”.43 Indeed, the impact of quality seed on improved agricultural yields has been proven in a number of studies.44 41 FAO, Seeds – FAO topic page, 2019 42 AGRA, Africa Must Increase Local Production of Improved Seeds, 2016 43 FANRPAN, Seeds without Borders – Policy Brief Series Issue no.1; Volume XI, 2011 44 See for instance IFAD, The impact of the adoption of CGIAR’s improved varieties on poverty and welfare outcomes: A systematic review, 2018; Kassie et al., Agricultural Technology, Crop Income, and Poverty Alleviation Figure 16: The Ecosystem of High-Quality Seed Usage in Sub-Saharan Africa Sources: McGuire & and Sperling, ‘“Seed systems smallholder farmers use,”’, 2016; ISP analysis; Stakeholder interviews; Icons from The Noun Project. R&D Governments can certify seed quality and phytosanitary purity Transparent and consistent quality standards Good agronomic practices Market information Good agronomic practices Quality inputs Good policy Trained personnel e.g. inspectors, border officials Infrastructure e.g. labs, vehicles for field tests Accessible roads Storage facilities Enabling factors Appreciation of value Appropriate input finance Affordable prices Trust in authenticity Farmers access markets for surplus yields Farmer has quality seeds Farmer uses quality seed to have higher yields Seed companies can produce quality seed Quality seeds satisfy border control standards Seed companies can distribute to agro￾dealers and local markets Farmers purchase certified quality seed (~7% of all seed acquired)1 Production Standards and borders Distribution Access Usage STP area of intervention NGOs / govts. buy and distribute seed (~10% of seed) Access to markets Variable quality seed from various sources – esp. own stocks, local markets, friends/neighbors/relatives (~83% of all seed acquired) Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 78 Although high-quality seed is more effective when other factors—such as good farming practices—are in place, it can still improve agricultural outcomes in the absence of those other factors. In a recent intervention in Tanzania, the One Acre Fund found that the use of hybrid maize seed resulted in a 45% average increase in yield even in communities where only 30% of farmers also used fertilizer.45 High-quality seed can also have a multiplier effect on good farming practices because it tends to be more responsive to these practices than lesser-quality seed. An often-touted advantage of high-quality seed is that it is more “responsive to fertilizer and irrigation”46 than poor quality seed. Several studies have also illustrated this, showing that farmers who use improved seed and fertilizer achieve higher yields than farmers who use traditional seed and fertilizer.47 STP’s focus on harmonizing seed policy is a necessary improvement for the ecosystem of quality seed use, but is insufficient given the breadth of other needs in the ecosystem. As illustrated in Figure 16, from seed production through to market access, a range of other conditions must be in place for farmers to plant high-quality seed. The aspects of the ecosystem that STP seeks to enable are government seed certification and the movement of seed across borders. Seed policy harmonization plays a major role in improving the seed production environment by strengthening the business case for producing high-quality seed. Seven of eight seed companies interviewed agreed that harmonized quality seed policies increase the accessible market for their companies’ varieties. A representative of one Zambian seed company claimed that “the protocol [HSRS] will allow us to go into markets like the DRC that we currently do not.” HSRS plays a role in reducing the costs and improving the ease of: • Seed certification – HSRS requires that varieties only be tested for two to three years in two SADC countries instead of two to three years in every country of intended export, reducing both the cost and time required to certify high-quality seed • Transacting across borders – three seed companies and an international agricultural research organization raised the issue of long delays at borders, which reduces the viability and germination of seed and often renders it unsellable. 48 Long delays happen because border officials either incorrectly determine that the accompanying paperwork is inadequate or they confuse seed for grain. One Zambian seed company recounted how it lost 250 metric tons (MT) of seed in one year at one border post. • Accessing information about potentially high-yield varieties in other countries – The public regional catalogue allows businesses to make decisions about what varieties to develop, export, or import based on information about the characteristics of varieties existing across the region. Two seed companies, along with ICRISAT and the Scaling Seeds and Technologies Partnership, pointed to in Uganda, 2011; Alene et al., The economic and poverty impacts of maize research in West and Central Africa, 2009 45 One Acre Fund Interview, May 19, 2019 46 IFAD, The impact of the adoption of CGIAR’s improved varieties on poverty and welfare outcomes: A systematic review, 2018, p.11 47 Notable examples include Suri, Selection and Comparative Advantage in Technology adoption, 2011, and Duflo et al., How High are Rates of Return to Fertilizer? Evidence from field experiments in Kenya, 2008 48 Syngenta, Harmonizing Regional Seed Regulations in Sub-Saharan Africa: A Comparative Assessment, 2015; World Bank, Africa Can Help Feed Africa: Removing barriers to regional trade in food staples, 2012 Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 79 this role the catalogue plays in business decision making and the promotion of seed production and trade.49 These improvements in the production environment also contribute to lower prices for high-quality seed. Lower seed prices help reduce overall costs for farmers already using high-quality seed—freeing up their resources for other opportunities. This has been proven in other regions, with the World Bank stating that “successful regulatory reform has contributed to increased supply and lower prices in the seed markets in Bangladesh and Turkey”. 50 It also enables more farmers to access high-quality seed for the first time—of crucial importance given that 83% of seed used in sub-Saharan Africa is uncertified, variable-quality seed sourced from the informal seed sector. 51 Improving this group’s access to high-quality seed could have significant impact, as outlined above. However, gaps in the ecosystem, on both the supply and demand side, continue to exist. A number of studies52 confirm what interviews with development partners, government officials, and seed companies suggest: there are significant issues across the ecosystem. For instance, 15 stakeholders considered lack of farmer appreciation of the value of certified seed use to still be a challenge. For its part, the World Economic Forum considers low seed production, inappropriate policies, distribution challenges, and low farmer demand as the key “bottlenecks” to increasing high-quality seed use in Africa. 53 A combination of these factors ultimately translates into low levels of high-quality seed use. A number of interventions are addressing other issues within the ecosystem that could undermine the effectiveness of STP. To the extent that these interventions are sufficient, they can support STP’s work in translating into increased use of quality seed. Interviewees identified at least 13 interventions that donors and governments are implementing across the high-quality seed use ecosystem. The scope of this evaluation did not include a full mapping of these interventions or a review of their sufficiency, but the presence of these interventions should instill a degree of confidence that other conditions are being addressed in a way that supports the effectiveness of STP’s work on seed policy harmonization. Considering this, STP is a valuable intervention focused on an impactful issue that, if successfully addressed in combination with the other issues, will translate into improved agricultural productivity. Ecosystem: To what extent is harmonization of seed policy an issue that USAID/REGO is well positioned to address? Given that there are many areas in which they could have intervened, the evaluation team also explored whether harmonization of seed policy is an issue that USAID/REGO is well positioned to address. 49 USAID, Mid-term Performance Evaluation of the Scaling Seeds and Technologies Partnership, 2017 50 World Bank, Enabling the Business of Agriculture 2017, 2017 51Amount from McGuire & Sperling, “Seed systems smallholder farmers use”, 2016. Note that the informal seed sector, also known as the farmer-managed seed system, refers to the “historical and traditional practices of farmers regarding…seed conservation, maintenance and selecting of seed diversity, and the saving, reusing, exchanging and selling of seed amongst family, neighbors and communities.”- African Centre for Biodiversity, Status report on the SADC, COMESA and EAC harmonized seed trade regulations, 2018 52 Such as AGRA, Africa Agriculture Status Report 2018, 2018; FAO, Appropriate Seed Varieties for Small-scale Farmers: Key Practices for DRR Implementers, 2014; Doss et al., Adoption of Maize and Wheat Technologies in Eastern Africa: A Synthesis of the Findings of 22 Case Studies, 2002 53 World Economic Forum, How to Improve Africa’s seed industry, 2015 Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 80 As seen in the COMESA and Economic Community of West African States (ECOWAS) experiences, the combination of regional and domestic change is critical to making progress in implementing seed policy harmonization.54 Our interviews with COMESA and a development partner in Malawi stressed the importance to seed harmonization of a regional body that supports the development of regional policy and has oversight over the entire region. An interview with COMESA also suggested that, in order to be effective, a regional harmonization project must also have the ability to influence domestic policy change and implementation. COMESA has been able to do this through its specialized agency, the Alliance for Commodity Trade in Eastern and Southern Africa (ACTESA), which operates in 19 COMESA countries. USAID/REGO’s ability to operate both regionally and domestically means it is simultaneously poised to interface with SADC and drive national implementation. USAID/REGO’s flexibility stems from its ability to work at both levels through its own organization, in addition to coordinating with bilateral missions when necessary. USAID/REGO’s willingness to support harmonized seed policy is also highly additional, in light of other donor funding coming to an end. Between 2004 and 2008, the Swiss Agency for Development and Cooperation (SDC) funded development of the MoU and technical agreements for SADC regional harmonization. 55 In addition, between 2010 and 2013, SDC also funded the SADC Harmonised Seed Security Project (HaSSP), implemented by the Pretoria-based Food, Agriculture, and Natural Resource Policy Analysis Network (FANRPAN). SDC decided not to renew this funding and subsequently turned its attention to other parts of the high-quality seed use ecosystem, such as stimulating farmer demand for certified seed in Mozambique. However, two stakeholders that have been involved with regional harmonization over the past ten years—a regional multi-lateral agency and national seed authority—reported that during the period since USAID stepped in, STP has been able to advance regional harmonization further than the SDC. Outcomes: How has STP performed against LoP targets and, considering the current trajectory, will STP achieve LoP targets? Given this report’s assertion that seed policy harmonization is a valid goal to pursue, the report now looks at how STP has performed against the LoP targets related to this goal. STP has achieved or is near to achieving several LoP output targets. This includes indicators such as the number of organizations receiving U.S. government support, number of policies supported, and the number of seed varieties on the regional catalogue in distribution. As of FY 2018, progress to date on several indicators, including quantity of seed traded under HSRS, remains at zero (see Table 4). However, STP activity in 2018—coupled with planned activity for 2019 and beyond—strongly suggests that most of these goals will be achieved. Two of the four indicators where progress remains zero relate to the quantity of seed traded under HSRS. By the end of the project STP will have completed more than one HSRS pilot. One such pilot is with Seed Co Zambia for the export of 200 MT of seed to the DRC. This alone will achieve almost 50 percent of the LoP target for quantity of quality seed traded under HSRS, and a significant portion of the LoP target 54 USAID, Performance Evaluation: Common Market for Eastern and Southern Africa Final Report, 2018; USAID, Mid-term Evaluation of the West Africa Seed Program, 2016 55 African Centre for Biodiversity, Status report on the SADC, COMESA and EAC Harmonized seed trade regulations, 2018. Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 81 for value of quality seed traded under HSRS. Interviews with the implementer and Seed Co Zambia indicate that this pilot will be completed in 2019, allowing STP to populate this indicator. Another indicator where progress remains zero is a binary indicator relating to whether a sustainable business model for the SADC Seed Centre exists. The SADC FANR Directorate will present the SADC Seed Centre sustainable business model to the SADC Council of Ministers in 2019. If the model is endorsed, and the Centre implements it, it will allow STP to achieve 100 percent of this indicator. Our interviews with the implementer and SADC indicate that this will be achieved within the life of the project. STP appears unlikely to achieve its LoP target relating to the percent of regional HSR administrative system costs covered by revenues generated. STP targets revenue covering 35% of regional HSRS administrative costs. Current achievement against this indicator is zero. The evaluation team’s calculations and interviews with SADC, the implementer and a regional multilateral organization suggest that it is possible to ultimately achieve this. However, as detailed further under sustainability (8.2.3), the need to build private sector demand and operationalize the system cast doubt on whether it can be achieved during STP’s LoP. Source: USAID, Feed-the-Future Seed Trade Project Annual Performance Report, 2018; Stakeholder interviews Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 82 56 STP documentation did not classify the indicators as output vs. outcome indicators in the way SATIH did; given this, the classification here reflects the view of the evaluation team. Table 7: STP indicators Indicator Output/ outcome 56 Absolute value achieved Progress against 2016/ 2017/ 2018 targets Progress against LoP Target EG.3.2-20: Number of for-profit private enterprises, producer organizations, water user associations, women’s groups, trade and business associations and community based organizations (CBOs) that applied improved organization-level technologies or management practices with USG assistance Output 54 600% 1080% EG.3.1-12: Number of agricultural and nutrition enabling environment policies analyzed, consulted on, drafted or revised, approved and implemented with USG assistance Outcome 11 550% 367% 4.5-Z03: Number of seed varieties in distribution from the regional HSR systems. Outcome 20 1000% 100% EG.3.2-4 Number of for-profit private enterprises, producer organizations, water user associations, women’s groups, trade and business associations, and community based organizations (CBOs) receiving USG food security related organizational development assistance Output 26 100% 65% EG.3.2-1: Number of individuals who have received USG supported short-term agricultural sector productivity or food security training Output 1,047 98% 63% 4.5-Z04: Number of seed companies engaged in regional seed varietal release Outcome 6 150% 60% EG.5.2-1: Number of firms receiving USG funded technical assistance for improving business performance Output 54 95% 45% 4.5-Z07: Value of grants distributed by the Seed Trade Project Output $415,071 45% 28% EG.3.2-22: Value of new private sector investment in the agriculture sector or food chain leveraged by Feed the Future implementation Outcome $100,000 40% 27% 4.5-Z05: Number of technologies or management practices transferred as a result of USG assistance Output 2 29% 20% EG.3.2-5: Number of public-private partnerships formed as a result of Feed the Future assistance Output 2 22% 14% 4.5-Z06: Quantity of high quality seed traded under HSR by crop Outcome 0 0% 0% EG.3.2-23: Value of targeted agricultural commodities exported with USG assistance Outcome $0 0% 0% 4.5-Z01: Sustainable business model for regional seed variety release; certification and quality assurance operational Output 0 Targeted 0 to date 0% 4.5-Z02: Percent of regional HSR administrative system costs covered by revenues generated Outcome 0% 0% 0% Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 83 The progress that STP has made in advancing HSRS harmonization in the region has been highly additional; This progress would have taken significantly longer to achieve without STP’s involvement. Thirteen stakeholders felt that STP had demonstrably contributed to their implementation of HSRS (see Figure 17). Two experienced seed sector actors felt that, whereas HSRS implementation had been slow for many years, the inception of STP accelerated regional progress. One of these actors went as far as to state, “Nothing would have happened without STP." The other noted how STP had especially made a difference in improving the functioning of SADC institutions. Effectiveness: How effectively has STP pursued seed policy harmonization? Given the strong performance to date against LoP targets, the following section will review the effectiveness of the specific tasks that contributed to this performance. Effectiveness: To what extent is SADC HSRS the optimal framework for STP harmonize? Throughout the evaluation, stakeholders noted the use of different regulatory frameworks, which were often used in parallel (for example, Zambia uses both SADC and OECD regulations). In response to these discussions, the evaluation team sought to confirm that SADC HSRS was optimal compared with other regulatory frameworks the project could have used. The team found that STP’s strategy of using SADC HSRS as the regulatory framework through which to harmonize seed regulations is well suited to the region. Theoretically, it would have been possible to harmonize OECD Seed Schemes or another regulatory framework. However, the choice to pursue regional harmonization through SADC HSRS is suited to the region in the following ways:  HSRS standards are more achievable for actors in the region to meet – OECD standards are more onerous and expensive to meet both for countries and companies; SADC HSRS’s lower level of standards can still represent significant improvements in seed quality given the region’s starting point  HSRS is more sensitive to local seed systems – for example, HSRS regulates landraces 57 that are commonly used in the region, although they satisfy slightly lower germination and purity requirements than does typical certified seed.58 Registration of landraces as quality declared seed (QDS) “provides an alternative for seed quality assurance, particularly designed for countries with few resources, which is less demanding than full seed quality control systems but yet guarantees a satisfactory level of seed quality”59 57 A landrace is a variable population, which is identifiable and usually has a local name. It lacks formal crop improvement, is characterized by a specific adaptation to the environmental conditions of the area of cultivation (tolerant to the biotic and abiotic stresses of that area) and is closely associated with the traditional uses, knowledge, habits, dialects, and celebrations of the people who developed and continue to grow it” – Negri, Towards a more comprehensive definition of ‘landrace’ than currently published, 2007 58 FAO, Quality Declared Seed System, 2006. 59 FAO, Quality declared seed system (QDS), 2019; SADC, Technical Agreements on the harmonization of Seed Regulations in the Region, 2008; OECD, OECD Seed Schemes - Rules and Regulations, 2019 Figure 17: STP interview codification – STP value-add Source: STP stakeholder interviews; n=21 Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 84  SADC standards put seed companies in a good position to meet standards for other regions within Africa – HSRS, like other regional systems such as COMESA and EAC, are modeled on OECD. While there is some variation between these systems, they are similar enough for a company to meet these standards once it has met SADC standards without having to begin completely anew60  SADC standards provide a country with a path toward OECD certification – Because they are modeled on OECD, once a SADC country has assimilated to HSRS and its implementation capacity is sufficiently strong, it need only begin a small number of additional activities to meet the technical requirements for OECD Seed Scheme membership61 OECD Seed Scheme membership is most advantageous to only a subset of Southern African countries that already have strong seed industries. Because the OECD Seed Scheme Regulations are an internationally recognized certification standard, and is for some countries an import requirement, country membership certifies that country’s seed companies and improves their global competitiveness. Joining and maintaining membership to the OECD Seed Schemes is costly. To join, a country must fund officials from the OECD to evaluate its country, and attend the OECD Seed Schemes Annual Meeting to present its application. Once a member, the country also has to incur membership fees, participate in the annual meetings, and begin testing for various issues that OECD requires and which existing national regulation may not.62 This is why seven stakeholders stated that, considering the additional costs, OECD membership largely makes sense for countries with mature seed industries. These countries can recoup these costs through fees to seed companies, or through existing state funding. This also explains the validity of STP supporting OECD harmonization activities in certain places (as it has in Zambia). Effectiveness: Is STP pursuing the right approach and set of activities to assimilate HSRS? STP is following a leading practice approach (through terms of capacity building and policy advocacy) to drive HSRS assimilation within its focus countries. Although there are few recognized leading practices within this space, the evaluation team found that STP has been adopting an approach in line with the approach taken by COMESA—the most advanced regional harmonization project on the continent.63 COMESA undertook regional harmonization through a combination of a) empowering a pre-existing regional structure to drive and govern the system and b) supporting specific regional countries in their efforts to create and implement new policies.64 In a similar manner to the work the COMESA project undertook, STP’s capacity building and policy advocacy work has prioritized the operationalization of the SADC Seed Centre and providing support to four focus countries in aligning and implementing different seed certification and phytosanitary policies. Development partners and seed companies reinforced the view capacity building and policy activities as necessary to implement HSRS in the region. An official from a regional multilateral 60 Syngenta, Regional Variety Release Test Cases: 2016 Findings, 2017 61 SADC, Technical Agreements on the harmonization of Seed Regulations in the Region, 2008; OECD, OECD Seed Schemes - Rules and Regulations, 2019 62 OECD, OECD Seed Schemes - Rules and Regulations, 2019 63 COMESA began the process of regional harmonization in 2008. COMESA benefitted from the early experiences of EAC and SADC to make fairly rapid progress. Through ACTESA, COMESA were able to pass the COMESA Seed Harmonization Regulations into binding regional law and get 5 out of 21 COMESA countries to assimilate their COMESA Harmonized Seed Regulations – African Centre for Biodiversity, Status report on the SADC, COMESA and EAC harmonised seed trade regulations, 2018 64 Senghor, Implementation of the Regional Harmonized Seed Regulation by Member States, 2017; USAID, Performance Evaluation: Common Market for Eastern and Southern Africa Final Report, 2018; USAID, Mid-term Evaluation of the West Africa Seed Program, 2016 Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 85 agency described STP’s choice of activities as “very strategic.” The officials also emphasized that their organization had faith in the direction that STP was taking with respect to driving regional harmonization. Within this approach, STP has undertaken an appropriate set of demand-driven tasks that has enabled it to respond to the unique policy and capacity needs of each focus country. Four of the five national seed authorities (NSAs) or national plant protection organizations (NPPO) the evaluation team interviewed described how STP’s work with them was primarily demand driven, reflecting their expressed policy change and implementation needs. The remaining NPPO instead shared how STP approached it with the intention of supporting the country’s alignment of its phytosanitary and quarantine regulations with HSRS. 65 It was through this initial conversation that the NPPO and the STP agreed to undertake specific policy change activities. Examples of STP’s responsiveness in this respect include: • The Zambian SCCI cold room – STP’s financial support for the refurbishment of the Zambian SCCI cold room arose because the SCCI was at risk of losing its ISTA certification • Mozambique seed law assessment – Mozambique’s DNSA requested STP to fund a seed sector assessment to resolve conflicting viewpoints it had received about the needed policy change Although STP is completing a necessary set of tasks, they are not fully sufficient to implement HSRS. Gaps remain in STP’s activity coverage: 1. STP is unable to meet the stated financing requirements of some regional countries. Twenty STP stakeholders—including 8 out of 10 Mozambican stakeholders—identified a continuing challenge with seed policy implementation, be it inadequate inspection capacity or insufficient inland and border laboratory infrastructure.66 STP has and will continue to build the former, but is constrained in its ability to support the latter as it can only provide grants of up to $100,000 (whereas these challenges often require much more significant funding), and it cannot fund construction. 2. STP does not include some of the region's largest import markets. The project’s four focus countries account for 96% of non-South African seed exports to the region but only 43% of non-South African seed imports from within the region.67 The project’s scope excludes Tanzania, Botswana, Namibia, and eSwatini, which together account for 54% of all seed imports from within the region, and would therefore benefit from HSRS making it easier to import seed into the region. These countries often have small seed industries and lower access to high-quality seed. For instance, even though 94% of maize seed planted in Angola is estimated to be farm-saved seed, Angola has among the lowest concentration of seed company activity in the region, with very little of this activity including local production and research.68 For these reasons, a regional seed policy expert and a national seed authority suggested that USAID expand STP’s focus to include one or more of the region’s importers. 65 Phytosanitary issues are those that relate to the plant health. Phytosanitary concerns for seed relate to whether seed is free from diseases and pests, especially when it crosses borders. 66 Tasai, the African Seed Index, found similar sentiments in Mozambique, Malawi, and Zambia. 67 UN Comtrade data accessed through the World Integrated Trade Solution 68 Access to Seed Foundation, Access to Seed Index 2019: Eastern and Southern Africa, 2019 Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 86 Effectiveness: How effective have those activities been in delivering the desired outcomes? Having considered the appropriateness of STP’s overall strategy and set of priority activities, this report now turns to the results that these activities have delivered. Broadly speaking, STP has advanced regional harmonization in the region. However, there is important variation in outcomes across the different types of tasks that STP has implemented. At the country level, driving policy change through advocacy is inherently difficult, but project documentation and various stakeholders report solid progress. In the capacity building space, country-level interventions have been strong—surveyed participants, direct beneficiaries and ecosystem actors give positive feedback on the outcomes of training events, awareness creation events, and technological acquisitions. That said, capacity￾building work on the regional level has progressed slowly—ecosystem actors and supplementary research shows that support to SADC institutions has not ensured that these institutions are operational. STP policy advocacy STP supported NSAs and NPPOs in Zambia, Mozambique and Malawi in drafting, updating and assessing their seed certification, release, and phytosanitary requirements. The form of this support varied across countries but included financing consultants, hosting review meetings, or funding STP’s seed development expert to advise on specific policy changes. Any assessment of the effectiveness of policy advocacy must recognize the inherent difficulty and slowness of a process influenced by so many political and bureaucratic factors. This is particularly the case when compared to the more transaction-oriented activities pursued by other parts of TFSP. The challenges associated with policy change include: • Existing backlogs in the departments that grant required approvals – e.g., the Zambian Attorney General at the Ministry of Justice (MoJ) has a queue of bills that it still needs to approve before the relevant department submits it to parliament. No other agricultural bill has passed through the MoJ since review of the Seed Act began in 2013. One stakeholder suggested that this was due to the Ministry having a shortage of legal staff capable of processing the reviews. • NSAs / NPPOs lack financial resources to implement the new policies – e.g., Mozambique is applying for financial support from STP to host events in which to train / raise the awareness of border officials on the new pest list and the findings of the seed law assessment. • Intrinsically slow legal changes – e.g., in Malawi, legal changes are often slow owing to resource constraints and political economy factors that electoral cycles exacerbate69 Given the above constraints, and the fact that most activity occurred after the project’s turnaround in 2018, the team recognizes that there is a short time window for countries to implement policy changes. 70 With this in mind, STP has advanced policy change toward HSRS implementation in most focus countries; however, these policies have yet to translate into implemented seed regulation. As shown in Table 5 below, STP has moved Malawi, Mozambique, and Zambia closer towards HSRS-aligned seed certification, release and phytosanitary policies. However, most of these policies are pending either approval and / or implementation. Nevertheless, ecosystem actors perceived the progress made to be strong, especially considering the barriers to policy change outlined above. 69 At the time of writing this report, the next Malawian election was upcoming, scheduled for May 2019. 70 All but one instance of policy change support that STP has provided occurred in 2018. Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 87 However, there has been no policy advocacy progress in Zimbabwe. USAID restrictions on working with the Zimbabwean government have resulted in markedly less activity in Zimbabwe than the other three focus countries. As a result, Zimbabwe is still not a signatory to SADC HSRS and none of the eight domestic policies that STP has been involved in shaping have been in Zimbabwe. Table 8: Policies or regulations supported by STP Country Policy or Regulation Supported Year Supported Status Malawi Malawi Seed Act Review 2018 Awaiting approval by Ministry of Justice, Parliament and Office of the President Malawi Seed Policy 2018 Approved and in force Review of Malawi Pest List 2018 Approved and in force Mozambique Assessment of Mozambique’s Seed Sector Legal and Regulatory Framework 2018 Approved but awaiting circulation with relevant stakeholders Mozambique Pest list Review 2018 Approved and in force Zambia Review and Approval of New Seed Regulations 2017 Approved and in force Drafting the Zambia Pest Risk Analysis and Phytosanitary Statutory Instrument 2018 Awaiting approval by Ministry of Justice Drafting the Pest Surveillance Statutory Instrument 2018 Awaiting approval by Ministry of Justice STP capacity building STP’s capacity building work falls into four categories: training events, awareness creation events, acquisitions of technology for seed regulation authorities, and support to SADC institutions. Training events During the evaluation period (fiscal years 2015 to 2018) STP undertook training events covering a variety of technical aspects of HSRS implementation. The training topics included laboratory and field-testing procedures for SADC HSRS and OECD, as well as quarantine and phytosanitary measures required under SADC HSRS. The event participants were largely seed inspectors and analysts at NSAs, NPPOs, and seed companies. Source: USAID, Feed-the-Future Seed Trade Project Annual Performance Report, 2018; Stakeholder interviews Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 88 STP training events have been relevant and have improved the technical capacities of their participants. Ninety-eight percent of surveyed participants found STP training events useful or extremely useful while 76% believed that the events had made them better able to perform the work that they already do (see Figure 18). For instance, referencing seed certification tests covered in a STP training, one participant stated: “The training(s) equipped me with expertise in DUS and VCU [DUS tests whether a variety sufficiently differs from existing varieties, while VCU tests the seed’s usefulness by assessing factors such as yield, quality, and disease resistance] to an extent that now I am one of the resource persons during the annual seed inspectors course [sic]” STP training participants deemed the training events to be valuable. STP training participants are also highly likely to recommend STP training events to friends or colleagues – STP training events obtained a Net Promoter Score (NPS)71 of 65. As illustrated in Figure 19 below, this is slightly higher than the TFSP NPS average of 6272. Also, 88% of event participants stated that they or their organizations would be willing to pay to attend STP training events. However, survey responses suggest that STP training events do not allocate sufficient time to practical aspects of seed testing and certification they cover. This warrants consideration, as STP training events seek to improve participants’ abilities to perform field and border tests according to HSRS; inadequate attention paid to the practical aspects of seed testing and certification limits the extent to which the event can meet its stated objectives. Participants’ most common recommendation is that the trainings should allocate sufficient time to cover practical demonstrations of training topics. The word “practical” appears in 7 of 43 responses, followed closely by “field” with six occurrences. One surveyed 71 NPS is a measure of customer experience based on the extent to which a customer would recommend the product/service to a friend or colleague. NPS scores range from -100 to +100 based on a categorization of participants into detractors, passives and promoters 72 NPSes for the other activities were: SATIH training 53, SATIH awareness: 45; TPI awareness 88; TPI training: 86. Source: STP training event survey; n=49 Figure 18: Percentage of respondents who applied knowledge from STP training events 76 55 41 37 37 24 14 12 10 8 4 I have expanded to work in new areas/on new activities I have progressed in my career I have been asked to share my knowledge or provided advice to others on the topics I can better perform work that I do already I have lobbied/conducted advocacy related to the topics My organization or business operates in new geographies My organization or business has developed a new strategy/policy I have conducted further studies/research on the topic to deepen my knowledge I have used this new knowledge to create new business leads I have successfully closed additional business deals I have not applied the training content Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 89 participant stated that training events could be improved “through having more time on hands-on” elements of the training topics Another said that they needed “to get the practical reality on [the] ground” through “more practical examples and visit[s] to a growing field, processing factory, border posts, and small-scale farmer/seed consumer or merchant.” Awareness creation events STP has given grants to seed trade associations in its focus countries to host seminars and workshops to raise awareness about HSRS. Whereas STP training events are meant to educate participants on specific technical aspects of HSRS implementation, STP’s awareness events aim to improve familiarity with HSRS in its entirety, provide updates on national HSRS implementation progress, and encourage participation in the system. The primary audiences for these events are seed companies, farmers, and government officials in the relevant HSRS implementing departments. STP awareness events have been useful to participants and effective in changing how participants work in or run their organizations. Ninety-six percent of surveyed participants found STP awareness events useful or extremely useful. Thirty percent of surveyed participants described their organizations as having developed a new strategy or policy following the event, while 10% of participants’ organizations or businesses are now operating in new geographies because of the awareness events (see Figure 20). However, STP awareness event participants are more passive about the events compared to participants of other TFSP events. STP awareness events obtained an NPS of 36, the lowest among all TFSP events (see Figure 19). This NPS score is a function of STP awareness events having the lowest proportion of promoters (people who rate their likelihood of recommending STP events as 9 or 10 on a 10-point scale) and the highest proportion of passives (people who give a 7 or 8 rating on the scale) among TFSP event participants. Whereas 73% of survey respondents across TFSP are promoters, only 50% of STP awareness event participants are promoters. Figure 19: Comparison of STP awareness event NPS Source: STP awareness event participant survey; n=42 STP awareness STP training TFSP average Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 90 While low relative to the rest of TFSP, one can also understand this result within the context of the purpose of STP’s awareness events. These events impart information with no expectation of participants having gained tangible skills or having embarking on immediate action following the event. As a result, the fact that a significant portion of participants would be passive about their likelihood of recommending the event is unsurprising. Technological acquisitions STP has financed two technological investments during the evaluation period, with plans to provide more support in this way over the project’s remaining lifetime. Both have been for Zambia’s national seed authority, the Seed Control and Certification Institute (SCCI): 1. Development of an online seed certification system that allows the SCCI to digitally report and process seed certification results 2. Refurbishment of SCCI’s cold room to make the Institute’s infrastructure once more fully compliant with International Seed Testing Association (ISTA) standards, as required by HSRS The SCCI and other Zambian and regional stakeholders report that both technological investments that STP has provided have improved implementation capacity. Online certification system: The SCCI stated that the online seed certification system has reduced the administrative burden of the seed certification process while improving its transparency and speed. Other stakeholders have also noted the value of the system. Unprompted, an STP training event participant wrote that “online registration and seed certification has also helped in reducing the time of doing business.” T wo stakeholders whom the team interviewed—one Mozambican and another Figure 20: Percentage of respondents who applied knowledge from STP awareness events Source: STP awareness event participant survey; n= 46 61 39 30 24 23 22 20 9 9 4 2 I can better perform work that I do already I have been asked to share my knowledge or provided advice to others on this topic I have used this new knowledge to create new business leads I have expanded to work in new areas/on new activities My organization or business has developed a new strategy/policy I have progressed in my career I have conducted further studies/research on the topic to deepen my knowledge. I have lobbied/conducted advocacy related to the topic My organization or business operates in new geographies I have successfully closed additional business deals I have not applied the content from the awareness event(s) Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 91 Malawian—spoke of how their respective countries could also benefit from such a system based on what they had seen. Cold room: The interview highlighted how the absence of a functional cold room was compromising SCCI’s ability to adhere to ISTA testing protocols. The SCCI was therefore confident that the new cold room would allow Zambia to retain its ISTA accreditation. Support to SADC institutions STP has provided policy and capacity support to the SADC Food, Agriculture, and Natural Resources (FANR) directorate to govern HSRS. STP directed a significant portion of this support towards the operationalization of the SADC Seed Centre, an institution under FANR tasked with running the system and advancing implementation in the region. STP’s support to SADC has consisted of: • Indirect financial support to SADC, including: o Funding key SADC meetings such as the SADC Seed Technical Meetings and SADC Seed Committee o Funding a consultant to be interim director of the SADC Seed Centre o Funding SCCI staff to assist SADC in registering new varieties on the regional catalogue • Technical assistance, including: o Collaborating with the FAO Southern Africa office to advise on SADC FANR’s development of Regional Seed Certification and Quality Assurance Guidelines o Reviewing the SADC Seed Centre Sustainability Business Model Both forms of STP support have invigorated these SADC institutions. STP has managed to support some regional policy change (see Table 6 below) and regional meetings that are necessary to govern the system. SADC, a national seed authority, and a regional multilateral organization highlighted how these would likely not have been achieved without STP. For instance, STP funding has contributed to the establishment of the SADC Seed Committee and has improved the regularity with which SADC Seed Technical Meetings occur. However, the SADC Seed Centre, the cornerstone of the regional seed governance structure, is still largely non-operational. The SADC Seed Centre still does not have a full-time coordinator. STP funded the appointment of an interim coordinator with a view that the position would be formally established and funded by SADC. However, SADC has yet to do so. Although SADC insists that this will happen, there is no clear plan for how this will be accomplished. Due to the absence of a full-time coordinator, STP still funds or carries out many of the activities that the SADC Seed Centre ought to lead. These include processing applications for new registrations in the regional catalogue and leading the piloting of the system. Table 9: SADC policies and regulations supported by STP Country Policy or Regulation Supported Year Supported Status SADC SADC Seed Centre Charter 2017 Approved by SADC Ministers of Agriculture but awaiting endorsement by two thirds of member Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 92 countries before becoming effective Drafting Regional Seed Certification and Quality Assurance Guidelines 2018 Status unclear Guidelines on Pesticide Management and Risk Reduction 2018 Status unclear Execution: How well did STP execute its tasks? Across all tasks, stakeholders praised STP’s follow-through. As illustrated in Figure 21, 73% of STP stakeholders reported high follow-through by STP management, citing specific ways in which STP had begun and executed on plans with them. One NSA even commented that whereas other donors may speak to them without subsequently doing anything, STP actually delivers “strong commitment”. Regarding training and awareness events, surveyed participants considered the events well-organized. As illustrated in Figure 22, the majority of surveyed participants agreed with statements about the well execution of STP events. For example, 91% thought STP events (across training and awareness) were well￾organized, 91% of stakeholders thought that facilitators were knowledgeable about event topics, and 85% of participants thought that the events enabled them to build useful networks. Source: USAID, Feed-the-Future Seed Trade Project Annual Performance Report, 2018; Stakeholder interviews Figure 21: Interview Analysis of STP Execution Source: STP stakeholder interviews Interviewee perspectives on communication / ease of working with STP (n = 26) Interviewee perspective on STP follow through (n = 25) High Medium Low Source: STP survey participants; n=89 Figure 22: Survey Responses To Select Statements About STP Events 1 2 1 1 2 1 1 6 1 8 4 6 7 63 49 48 71 25 39 43 20 0 The facilitator(s) had an engaging presentation style 0 The venue was conveniently located The facilitator(s) were knowledgeable about the topic(s) The event(s) were well organized 0 0 0 0 0 Disagree Strongly disagree Neither agree nor disagree Somewhat disagree Somewhat agree Strongly agree Agree Seed Trade Project (STP) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 93 However, there are several ways in which STP might improve the execution of these events. A number of common themes emerged from participants’ free form responses to the question, “How could future awareness/training events be improved?”: Awareness events: 1. Facilitators / presentation – Six participants identified ways in which STP could have presented the awareness events better in terms of both style and content. One called for presenters to have up to date information on HSRS, another thought that presentations across facilitators could be coordinated better, while another two thought that STP could have found professional interpreters for the events involving participants from non-English speaking countries. 2. Participant profile and reach – Ten survey respondents called for STP to do more work to ensure that all of the relevant stakeholders attended the awareness events. Such stakeholders could be within the country – e.g., “More seed companies, Senior Agricultural Officers, Zambia National Farmers Union (ZNFU) and agro dealers” – or from other countries – “expertise from other countries within the SADC region”. 3. Post-event follow-up – Six survey respondents thought that STP ought to provide more regular updates on national progress with respect to HSRS implementation. These respondents thought it would be valuable for them to have an ongoing understanding of how their countries are progressing, with one stating that it would help them “correct [any] diversions” from the optimal path towards HSRS implementation. Training events: 4. Event duration - Nine survey participants suggested that STP should increase the duration of its training events. Three of these participants thought more time was needed to allow for more practical training, adding further support to the finding regarding the effectiveness of STP training events. The other six participants did not suggest where this extra time should be allocated, and one of them merely suggested that existing time allocations force facilitators to rush the coverage of the topics. Seed Trade Project (STP) – Management USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 94 8.2.2 MANAGEMENT Scope of evaluative theme This section assesses how the management of STP by USAID/REGO and its contracted implementer affected its performance. This assessment considers stakeholder perceptions of the implementer, USAID/REGO oversight of the activity, and the management model (e.g., implementation structures and management processes) used to implement STP. Synthesized findings The evaluation team found STP to have successfully corrected management issues it faced during the first two years of its implementation. USAID/REGO implemented corrective measures that included changing activity leadership, relocating the project and re-aligning focus. These corrective measures, especially the relocation of the activity from Pretoria to Lusaka, have been received well by stakeholders and have resulted in the strong project performance the team found in relevance above. How is STP management perceived? STP’s management prior to the reset in 2017/18 faced a number of challenges. Project documentation reflects that these challenges included a lack of strategic implementation, weak engagement with key stakeholders, and limited progress against key indicators.73 USAID/REGO and the implementer addressed these challenges through various measures that paired changing activity leadership with strategic changes. In response to observing challenges with the activity, USAID/REGO and the implementer took several steps to improve performance. Importantly, these steps included but went beyond merely changing STP’s Chief of Party – they also included performing an assessment of STP led by the Bureau for Food Security in 201774, refocusing the project’s mandate to policy harmonization as opposed to broad seed trade, and relocating the project from Pretoria to Lusaka. Rather than using formal contractual management tools, many of these improvements were driven by the combination of direct conversations between USAID/REGO and the implementer, and decisiveness by both parties. Stakeholders have consistently praised STP’s new management, citing their strong communication, technical knowledge, and experience. As illustrated in Figure 21, most stakeholders were complimentary about STP’s communication and follow through. Several stakeholders viewed STP’s new management as having technical competence and the knowledge both of how to identify “what needs to be done” and “how to maneuver the system.” Another summarized STP’s management as a “fantastic team of officers who are visionary.” Where issues were raised about STP’s management, they were remarks about communication on specific tasks and how quickly STP grant applications were processed. This improved perception of STP’s management reflects the implementation of strategic and operational changes by its new leadership that have improved STP’s performance. 73 USAID, USAID Southern Africa Seed Trade Project Assessment, 2017; USAID, Feed-the-Future Seed Trade Project Annual Performance Report, 2017; USAID, Feed-the-Future Seed Trade Project Annual Performance Report, 2018 74 USAID, USAID Southern Africa Seed Trade Project Assessment, 2017 Seed Trade Project (STP) – Management USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 95 Stakeholders (including USAID bilateral missions, SADC, and seed companies) suggest in interviews that USAID has adopted the following changes: • Intensifying and focusing the issuing of grants – Our interviews with the implementer and seed companies confirmed a USAID evaluative finding from 201775 that STP had previously been considering grant applications to private seed companies for purposes less directly connected with HSRS (e.g., the acquisition of irrigation equipment). Since the new management took office, it has prioritized using the grants facility more often and with more direct alignment with STP’s harmonization agenda. By improving alignment with STP’s mandate, new management was able to issue four grants in 2018, compared to only one over 2016 and 2017. • Increasing interaction and communication with stakeholders – A strong recommendation of the 2017 USAID evaluation was that STP needed to improve its engagement with focus country governments and USAID bilateral missions, in particular. This issue appears to have been addressed, as reflected in positive feedback from bilateral missions, NSAs, and NPPOs regarding STP’s communication and engagement. • Improving relations with SADC – Until late 2017, STP had struggled to forge a collaborative working relationship with SADC.76 Our interview with SADC FANR and its sub-division, the SADC Plant Genetic Resources Centre (SPGRC), suggests that this relationship has markedly improved. A reflection of this is how the SPGRC now interacts with STP. Under its old management, SADC had hired a consultant to be interim Coordinator of the Seed Centre. Although the consultant worked from the SPGRC office, SPGRC felt no ownership over the role. Due to the new management’s better relationship with SADC, SPGRC is now actively working with STP to have the position of Coordinator funded and recognized by SADC. With this, SADC is planning to formally incorporate the coordinator role into its organizational structure and is willing to provide full administrative support to the Coordinator once they are in office. This is a positive step towards operationalization of the SADC Seed Centre. Does the project implementation structure (HQ location, country advisor model, USAID activity manager model) enable STP to succeed? STP’s location in Lusaka has contributed significantly to its success. This has allowed STP to forge closer relationships with key stakeholders given Zambia’s positioning as a “regional seed hub.” The implementer and two bilateral missions remarked that the STP’s location in Pretoria severely constrained its ability to foster stakeholder relationships and remain abreast of developments at the SADC Seed Centre (based at SPGRC in Lusaka). STP’s country advisor model also brings local proximity that has advanced the project’s success in its respective focus countries. STP currently has country advisors in Malawi and Mozambique. Six stakeholders (two in Malawi and four in Mozambique) out of ten country stakeholders specifically praised STP’s country advisors when asked to give their perceptions of the quality of STP’s management. These stakeholders referenced the advisors’ accessibility, charisma, and collaborative spirit. 75 USAID, USAID Southern Africa Seed Trade Project Assessment, 2017 76 USAID, USAID Southern Africa Seed Trade Project Assessment, 2017, p.9 Seed Trade Project (STP) – Management USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 96 To what extent do management processes (including USAID processes and contract structures) help or hinder the success of STP? The input of the bilateral missions, especially through the annual work plan development process, has proved an effective mechanism for STP to reflect local contexts in its work. All four of the bilateral missions spoke positively of how their contributions were considered by USAID/REGO and the implementer. Two of the four referenced how the process allowed the bilateral missions to consult the important stakeholders in their country; one stated that “the consultative process is making things work better, [allowing for] co-creation with the beneficiary.” As a member of the implementing staff and one of the bilateral missions recognized, this allows the program to “catch” contextual factors and implement locally relevant activities. USAID protocols inhibit the project’s flexibility and often underlie the few frustrations stakeholders had around STP management. • Expenditure rules are inflexible to local dynamics – Three stakeholders, one of which is a national seed authority, expressed frustration with USAID procurement rules that appear inflexible to local challenges. One of these stakeholders stated that many STP participants in his country complain that there are “many small, small things that must be followed to the dot.” Although these protocols may in small ways diminish the goodwill STP has built with stakeholders, they do not appear to have had an adverse impact on STP’s performance. • Travel approval processes that constrain stakeholder engagement – As a regional project, STP often demands that STP HQ staff travel to focus countries and other places for SADC regional meetings. However, given their inability to obtain travel approval at short notice, STP staff have, on occasion, declined to attend stakeholder meetings in the region when insufficient notice has been given. • Communication protocols that delay responsiveness – Mandatory USAID and implementer sign-offs can hinder the pace of country advisor communication with government officials. According to STP’s two country advisors, USAID and STP’s leadership have agreed that country managers cannot directly communicate with government officials in their countries. Instead, protocol requires them to first clear the intended communication with USAID/REGO and STP’s leadership. The country advisors stated that this delays communications and disregards the advisors’ experience working with those government officials. They suggested that approval processes could be better tailored based on government official seniority to minimize the frequency with which these sign-offs act as barriers while still allowing USAID/REGO and the implementer to manage senior relationships. Seed Trade Project (STP) – Sustainability USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 97 8.2.3 SUSTAINABILITY Scope of evaluative theme The evaluation team’s assessment of STP’s sustainability concerns the extent to which the regional and national changes STP has supported will endure beyond the LoP. Regionally, this section evaluates the extent to which SADC will be able to continue pursuing regional harmonization. Nationally, it assesses the extent to which policy changes will last and whether countries will sustainably implement the policy changes that STP has supported. Synthesized findings The evaluation team found that the SADC Seed Centre – the institutional vehicle for SADC’s implementation of regional harmonization – is not yet operational and that there is still a long path towards making it operational. This path involves building the private demand required to realize enough revenues from the regional release system. At a country-level, STP has made enduring policy advances but countries require further support to build the human and infrastructural capacity to sustainably implement these changes. Can SADC institutions continue advancing HSRS without the support of USAID? The SADC seed governance structure currently lacks the resources to govern and implement HSRS beyond STP’s lifetime. SADC seed institutions—in particular, the SADC Seed Centre—are the bedrock of a functional SADC HSRS beyond STP’s lifetime. The HSRS technical agreements task the SADC Seed Centre with, among other responsibilities, managing the regional release system, developing implementation guidelines, and promoting HSRS assimilation in the region.77 However, several interviewees—including SADC, a national seed authority, a country advisor, and a regional multilateral organization—express the belief that the Centre still lacks the financial and human resources to adequately perform these duties. The evaluation team found mixed evidence for whether STP’s current plan to remedy this is sufficient to result in an independently sustainable SADC Seed Centre after STP. Our interviews with SADC, the implementer, and an experienced seed sector actor reveal that STP and SADC’s plan to sustain the SADC Seed Centre consists of three parts: 1. Formally institute and recognize the position of SADC Seed Centre Coordinator – SPGRC, SADC FANR, and STP are developing a scope of work for the position of SADC Seed Centre Coordinator. According to SADC, this position will form part of the SPGRC organizational structure, and be accountable to the head of SPGRC. 2. Directly fund the Centre through STP for two years – According to the implementer and SADC, STP is in the process of issuing SADC a grant of $400,000 to fund the Centre for two years. 3. Rely on the regional release system and SADC funding to sustain the Centre after the two-year period – Based on the 2018 STP annual report 78 and an interview with the implementer, the plan is for the Centre to sustain itself from a combination of revenues from the regional release system and funding from SADC. 77 SADC, Technical Agreements on Harmonization of Seed Regulations in the SADC Region, 2008 78 USAID, Feed-the-Future Seed Trade Project Annual Performance 2018, 2018, p.11 Seed Trade Project (STP) – Sustainability USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 98 Considering each part in turn, the evaluation team has identified mixed evidence for the plan’s success. SADC is confident that it can formally institute the position, but other stakeholders have serious doubts. Two SADC staff members were positive about the likelihood of their formally creating the position. Both were unclear on how long the process would take but were confident it would happen before STP ends. However, three national seed authority interviewees were doubtful that SADC would be able to do this. They stated that the Seed Centre Charter, the Centre’s constituting document, still needs to be signed by two-thirds of member states for it to be binding. Indeed, besides finding that the SADC Committee of Ministers of Justice has endorsed the Charter, 79 as has the SADC Council of Ministers, 80 the evaluation team could not find evidence that the Charter had officially been signed by any member states. The legal standing of the SADC Seed Centre therefore remains unclear. The implementer and SADC are convinced that the grant will be approved by USAID/REGO and disbursed to SADC. The second part of the plan relies on USAID approving the $400,000 to SADC. This would be STP’s largest grant to date, far exceeding the standard $100,000 grant value limit for STP. Despite this, the implementer’s grants officer and SADC are positive that USAID/REGO will approve the dispersal of the grant given the conversations they have had and the documents that SADC has already submitted. Stakeholder interviews and the evaluation team’s calculations suggest that the variety release system could ultimately generate sufficient revenues to fund the SADC Seed Centre. Generating revenues from the regional variety release model would involve charging registration and label issuing fees to companies seeking to register and trade varieties on the regional catalogue. SADC, an implementation staff member, and a regional multilateral agency were confident that this would generate considerable revenues for the Centre. They considered the number of varieties, volumes of traded seed, and number of interested seed companies sufficiently high for the Centre to generate meaningful revenues. An exploratory calculation by the evaluation team suggests that the system could generate sufficient revenues if 13 percent of annual seed trade (about 4,300 MT of seed) in HSRS-aligned81 countries took place through the regional release system. This exploratory calculation is based on a single set of assumptions that could vary but do give an order of magnitude indication of the system’s potential. However, two experienced sources cast doubt on the viability of the model. First, an experienced seed sector actor with intimate knowledge of the evolution of SADC HSRS, and another implementation staff member, expressed doubts that this part of the plan would generate meaningful revenues. They viewed the number of commercially viable seed varieties as too low for the Centre to rely on. Second, the evaluation team could not find international experiences that support the viability of a regional catalogue of this nature. The EAC does not have a regional catalogue, in ECOWAS (like the EU), varieties registered on any national catalogue are automatically registered on the regional catalogue without additional applications or payments, and COMESA has yet to fully implement its regional release system.73F 82 79 SADC, SADC Ministers of Justice and Attorneys-General meet to discuss legal instrument, 2017 80 African Centre for Biodiversity, Status report on the SADC, COMESA and EAC Harmonized seed trade regulations, 2018 81 This refers to the 10 SADC countries that have signed the HSRS Memorandum of Understanding. The Seychelles, Mauritius, Madagascar, Angola, and Zimbabwe have not. 82 Syngenta, Regional Variety Release Test Cases: 2016 Findings, 2017; FAO, Seeds Toolkit – Module 4: Seed Sector Regulatory Framework, 2018; Interview with COMESA April 1, 2019. Seed Trade Project (STP) – Sustainability USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 99 Even if one assumes that the model can generate sufficient revenues, it will likely take some time to be viable. Our calculations suggest that the system would require about 4,300 MT of seed trade volumes through HSRS for it to be viable. Currently no seed trades through HSRS – the first batch of seed exports under HSRS will be the 200 MT of maize seed that Seed Co will export through the pilot with STP. Reaching 4,300 MT will therefore take some time, much of which will need to be spent on building seed companies’ confidence in the operability and value of the system. Also, the Centre will have to agree with NSAs on how they will deal with issues such as dispute resolution and label fee payments (considering that the implementer and the technical agreements require NSAs to print and distribute the labels, 83 while SADC expects to receive a fee on these labels). It is possible that this will all take several years to achieve given the magnitude of what is required. Although both SADC and the implementer are confident that SADC will meet any shortfalls in required funding, the evaluation team have not seen a clear path for how SADC will achieve this. The implementer indicated that SADC made this commitment as part of its grant application documents. However, SADC also acknowledged that it has been seeking to do this for the past few years without success. Are the changes in country-level policy and practice likely to sustain themselves in the absence of USAID support? STP has been instrumental in creating sustainable change. Two experienced seed sector actors felt that STP had accelerated regional progress on HSRS implementation since its inception. As identified above, STP’s focus countries have altered their policies to align more closely with HSRS and have improved their ability to implement these policies. The mere fact that many of these policy changes are difficult to roll back lends them an element of sustainability. That said, countries still lack the human and physical capacity to complete implementation of HSRS without further support. Twenty STP stakeholders identified a continuing challenge with seed policy implementation in their countries. These stakeholders described issues such as low numbers of trained inspection staff, insufficient laboratory infrastructure, and inadequate operational funding to NSAs (for vehicle fuel, laboratory consumables, etc.). These deeper challenges suggest that even if STP achieves its LoP targets, there will still be implementation difficulties in countries across the region, in STP focus countries and elsewhere. Moreover, the fact that these challenges all arise from the absence of sufficient state funding to HSRS implementation suggests little will on the part of national governments to provide this remaining support. To what extent are the wider conditions in the supply and demand ecosystem in place to aid sustainability? As outlined in the relevance assessment, the long-term success of seed harmonization and the sustainability of its outcomes hinges on improvements in other elements of the seed ecosystem—including use of quality inputs and good agronomic practices. Multiple donors are working to address these issues. The success of these efforts is crucial to ensuring that STP delivers sustainable results. Delivering the benefits of STP in a sustainable way and minimizing environmental risk depends on closer coordination with other initiatives. For example, an interview with the One Acre Fund showed that advancing the use of quality seed without corresponding provision of fertilizer 83 SADC, Technical Agreements on Harmonization of Seed Regulations in the SADC Region, 2008 Seed Trade Project (STP) – Sustainability USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 100 risks exacerbating nutrient mining and so lowering soil fertility. 84 In other words, there is a risk of environmental harm should STP make progress in the absence of a broader effort to improve the use of inputs and agronomic practice. To date, however, STP’s coordination with these other programs has been inconsistent across the four focus countries. Across the four countries, half of the ten stakeholders who spoke about stakeholder coordination in interviews identified it as an issue, while half did not. Three of the five stakeholders who found stakeholder coordination to be going well were in Mozambique, while three of the five stakeholders who saw issues were in Malawi. The Mozambican stakeholders who thought seed sector coordination was working well all referenced how STP, especially through its local country advisor, had collaborated with other seed interventions in the country and was active in the Association for the Promotion of the Seed Sector (APROSE). In contrast, in Malawi, one stakeholder described the landscape as "crowded and disorganized,” while another identified the need for STP to improve relations with an influential international agricultural development organization. 84 Nutrient mining refers to agricultural practices that result in a negative balance between nutrient input and output. This lowers soil fertility, ultimately reducing yields. Some high-quality seed varieties, such as hybrid maize seed, are nutrient-intensive to grow and may potentially lead to nutrient mining if the soil’s nutrient stores are not rebalanced through the application of fertilizer (Henao & Baanante, Agricultural Production and Soil Nutrient Mining in Africa, 2006). Seed Trade Project (STP) – Conclusions USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 101 8.2.4 CONCLUSIONS STP has largely been effective in the areas it has focused on, with the exception of work supporting SADC, which is yet to result in an operational SADC Seed Centre. The biggest challenges faced by STP are the areas that it has not yet addressed—e.g., need for wider financing support to meet infrastructure needs— and operationalizing SADC’s governance of the system. The opportunities for STP reflect these findings. 8.2.5 RECOMMENDATIONS AND OPPORTUNITIES During the remaining LoP STP strategy • Conduct a systems analysis of the regional seed sector – Systematically identify the most critical issues in the ecosystem that impact upon STP’s effectiveness, as well as areas where STP might collaborate with others. Such collaboration will also support STP’s sustainability (e.g., it may ensure that programs focusing on provision of quality inputs are minimizing the impact of nutrient mining). • In upcoming events, explore means of expanding the geographic reach of the project’s beneficiation to include more import-dominant countries (e.g., Angola) – For example, have more regional trainings that allow analysts and inspectors from more regional countries to attend, STP could also support improved capacity to implement HSRS in those countries. • Support National Seed Authorities (NSAs) and national plant protection organizations (NPPOs) in meeting structural infrastructure funding needs – Prioritize supporting NSAs and NPPOs in acquiring “hard” infrastructure such as laboratories, testing equipment, and vehicles . Where STP cannot finance this itself, consider partnering with DFIs and other organizations (e.g., the African Development Bank in Malawi) that are potentially willing and able to fund the infrastructure investment that STP cannot. STP Execution • Consider expedited protocols for rules related to expenditure, travel, and communications – Potentially work with the implementer to devise expedited processes for certain protocols given the regularity and extent of stakeholder interactions involved with implementing STP and frequent need for short-notice travel. Such expedited processes could still ensure the protection of USAID/REGO against financial and reputational risk while giving STP’s implementers more flexibility, reducing stakeholder complaints. For example, consider setting out clear guidelines for where USAID/REGO will attempt to accommodate short-notice travel requests (e.g., when the meeting is with an individual of a minimum level of seniority) Policy advocacy: • Provide needs-driven support to NSAs and NPPOs to enable implementation of STP￾supported policies – Ensure that more of the policies STP has supported are implemented by supporting NSAs and NPPOs in addressing specific constraints they have to implementing this (e.g., helping Mozambique NPPO to conduct sensitization of the country’s revised pest list). Training events: Seed Trade Project (STP) – Recommendations and opportunities USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 102 • Ensure that training includes sufficient coverage and time for demonstrations and participant exercises – Increase the likelihood that participants retain what they have been taught, and that they will apply the material when they return to their laboratories, testing fields, and border posts. Awareness events: • Put more care into the facilitation of STP awareness events - Work closely with seed trade associations as the grantees tasked with hosting STP awareness events to carefully select facilitators, coordinate their presentations, and consider additional presentation tools such as interpreters. • Provide regular updates to awareness event participants regarding HSRS progress in their countries and the region – Stay connected to past participants to advance building awareness about HSRS and give them information with which to lobby for HSRS implementation Beyond the LoP • Include other significant seed import markets within the scope for future HSRS implementation support – Potentially include countries like Tanzania, the DRC, and Angola to drive regional harmonization further and promote project sustainability • Strengthen commitment to operationalizing the SADC Seed Centre through deeper technical assistance – Either fund a consultant with a limited scope of work (centered on implementation of the regional variety release system – the core of the Centre’s business model) or give the SADC Seed Centre coordinator (once appointed) an additional grant that can only be used toward the regional variety release system. Trade Policy Initiative (TPI) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 103 8.3 TRADE POLICY INITIATIVE (TPI) Summary findings: TPI In assessing the relevance of TPI, the evaluation team found that the activity comprises a well-executed set of tasks that are relevant to improve the “agenda-setting” component of policymaking. TPI benefits from the complementary nature of its work, as well as an influential audience, which has helped the initiative in advancing the policy enhancement agenda in several specific policy areas. However, the lack of implementation capacity in the policy environment, although not a challenge that TPI sought to resolve, has hindered TPI’s effectiveness in translating this agenda into tangible policy change. At the program level, this represents a missed opportunity for SATIH to address implementation challenges and build on the agenda-setting work of TPI. TPI may be more effective if it widens its audience reach. Interventions set outside of government (e.g., workshops with the COMESA court and with journalists) have had high impact, although they represent a relatively small part of the activity. Meanwhile, the predominately male audience limits the equity of the intervention. From a management perspective, TPI’s stakeholders hold the activity in high regard, noting the deep credibility and expertise of tralac and its associate network. The management model of tralac also functions well—the associate model reflects good value for money and provides flexibility; the co￾operative structure of the agreement provides sufficient management control through the work plan development process, while also offering tralac with a sufficient degree of freedom to push a rigorous research and policy agenda that requires a level of independence from USAID oversight to maintain its credibility. From a sustainability perspective, the evaluation concludes that TPI’s successes to date imparting knowledge and setting the agenda would be sustained given the nature of those successes. Furthermore, the fact that tralac has other funders means similar activities would continue if USAID funds were to cease, although some activities (e.g., AGOA focused activities funded entirely by USAID) would no longer be a focus, with efforts continuing only where tralac saw broader utility in doing so. Regarding equity, the audience currently has slightly more male participants than female, although tralac are taking steps to address this in starting a women’s policy network, which TPI funding is contributing to. The following sections provide more detail and evidence on these overall findings. It is important to note in the case of TPI that tralac both serve as the implementer of TPI, but also of a wider set of programming funded by other donors (primarily SIDA). This report focuses specifically on USAID-funded activities under TPI, unless it explicitly references tralac work beyond TPI. 8.3.1 RELEVANCE Scope of evaluative theme The assessment of TPI’s relevance begins first by looking at how TPI’s work fits into the overall policy cycle and ecosystem, which shapes its potential efficacy. Next, the report evaluates how effective and additional the strategies TPI has pursued have been—both as three parallel strands of work, and in the way those three strands come together as a single activity. Trade Policy Initiative (TPI) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 104 Synthesized findings The evaluation found TPI interventions to be delivering excellent quality work that responds to needs at the agenda setting stage of the policy cycle. That said, the extent to which this work translates into good trade policy outcomes is reduced given significant challenges around a lack of capacity to implement policy changes fully. The true success of TPI’s work is not fully reflected in its MEL data, given the overemphasis on tracking outputs without connecting those to the outcomes the work goes on to deliver. That said, the evaluation team found that capacity building and policy dialogue participants reported good level of application of the knowledge they gained, and also identified examples of research (although not well tracked) being used in influential ways by key readers. The audience of TPI is limited however, focusing primarily on governments and academics, despite marked successes where the activity has expanded the audience in the past (e.g., to journalists or judges). Outcomes: How is TPI performing against its annual targets? Given TPI takes the form of a co-operative agreement, rather than an ‘acquisition’-based contract structure, a set of LoP indicators and targets do not exist in the way they do for other activities. Instead, plans are set through the work planning process. TPI appear on the most part to have delivered against the original plan for their last full year workplan (ending December 2017). This has included significantly overdelivering on volume of daily updates, discussion notes, and co-funded conference participants. Reporting on TPI interventions has been inconsistent, making it difficult to determine TPI’s performance against annual targets. The inconsistencies and insufficiencies can be noted in the 2017 work plan, quarterly reports, and M&E matrix as shown in Table 7. The 2017 work plan identifies indicators to track outputs and outcomes of planned activities, however the quarterly reports either do not report on these or report on them inconsistently throughout the year. The 2017 M&E matrix reports on only a small number of the activities—namely, the number of trade briefs and working papers, and, incomprehensively, on activities to monitor regional integration. The focus of TPI reporting is predominately on output-level performance, which does not reflect the extent to which these are driving outcomes and ultimate objectives. For example, the 2017 work plan states that the outcome for TPI trade policy monitoring and analysis is, “Trade policymakers and influencers have better understanding of current and emerging trade policy issues.” The quarterly report outcome indicators reflect the number of working papers, trade briefs, and discussion notes produced. These indicators do not necessarily inform the extent to which TPI has progressed against the outcome of improving trade policy, but rather measure activity. Metrics on training events, as another example, track the number of participants, rather than how participants apply the knowledge gained from the events. To determine the performance of TPI, the evaluation approach aimed to go beyond the output-level tracking of TPI interventions to determine their effectiveness. Given that the quarterly reports and data provided did not allow for determinations of how TPI interventions are improving trade policy (beyond making trade policy trainings and analyses available), the research, surveys and interviews developed and used in this evaluation aimed to capture the information needed to understand the extent to which outputs feed outcomes. For example, the evaluation identified how respected international and regional institutions were using the research, as well as tangible applications by event participants of the knowledge they acquired at capacity-building and awareness events. Trade Policy Initiative (TPI) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 105 Table 10: Reporting on TPI tasks from Jan 2017 – Dec 2017 Activity 2017 Workplan Target Reported in 2017 M&E matrix 2017 Quarterly report tracking Jan - Mar Apr - Jun Jul - Sep Oct - Dec 2017 Total Trade Policy Monitoring and Analysis (Research) Newsletters/ daily news updates 40 Not reported in M&E matrix Not reported Not reported 72 67 134 Discussion notes 25 Not reported in M&E matrix Not reported 19 16 20 22 Data updates 40 Not reported in M&E matrix Not reported Not reported 5 Not reported 4 Trade briefs 10 8 2 5 2 1 10 Working papers 8 4 1 2 0 6 9 Training (Capacity Building) Masters degree scholarships 10 Not reported in M&E matrix 0 0 0 0 0 Short courses - Trade remedies and safeguards, Standards (SPS and TBT), Rules of origin, Trade policy development for the 21st century; Regional community law 1 Not reported in M&E matrix 0 0 0 0 0 Other workshops/trainings (Intellectual property and SACU workshop) 0 Not reported in M&E matrix 0 0 1 1 2 Interns 2 Not reported in M&E matrix 0 0 3 0 3 Policy dialogue Annual conference participants (co￾funded event) 40 Not reported in M&E matrix 0 94 0 0 94 AGAO workshops (reported under capacity building in this evaluation) 4 Not reported in M&E matrix 0 0 0 1 1 Interns 2 Not reported in M&E matrix 0 0 2 0 2 Source: tralac, 2017 Quarterly report: Jan – Mar, April – June, Jul – Sep, Oct – Dec 2017 Trade Policy Initiative (TPI) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 106 Ecosystem: To what extent are TPI’s efforts well positioned to improve trade policy when reviewed against the policy cycle and wider policy ecosystem? In exploring the effectiveness of TPI’s work and outputs in feeding outcomes, the evaluation team begun by setting the work of TPI against the wider challenges in the policy ecosystem it responds to. Policy change is a cyclical process that starts with recognizing a need, and then designing and implementing a policy response. To enhance the quality of formulated and implemented, a breadth of policy stakeholders (policymakers, policy implementers, and policy target populations 85) have a role to play at the five stages of the policy process (illustrated in Figure 23). TPI interventions focus on improving the agenda-setting stage of the policy cycle by imparting trade policy knowledge. Policymakers and implementers in strategic management positions typically make up the audience for these interventions. TPI interventions seek to raise awareness among these decision makers of the implications of global and regional trade instruments and policies on national issues. Other development actors focus on the further stages of the policy cycle beyond agenda setting that are necessary to improve trade policy. During interviews, the EU, The World Bank, other USAID programs, and tralac noted they are supporting ministries with policy formulation, adoption, 85 Policymakers are government officials who decide on, draft, and approve policies; policy implementers are government and non-government officials who ensure the enforcement and use of policies; policy target populations are physical and juridical persons whose activities are impacted by the policy. Figure 23: Cyclical Process of Policy Making and Policy Implementation Source: Adapted from The Texas Politics Project, 2019 Agenda setting Identification of public issue by policy stakeholders Policy formulation Creation of legislative, regulatory and programmatic strategies to address issue Policy adoption Adoption of policy solution in the form of legislation or rules Policy implementation Translating policies into processes that are live in “on the ground” reality Policy evaluation Determination of effectiveness of policy and implementation procedures Policy cycle Primary focus of TPI Secondary TPI focus Secondary TPI focus Secondary TPI focus Trade Policy Initiative (TPI) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 107 and implementation. For example, The World Bank recently launched its Trade Facilitation Support Program to help countries seeking assistance in aligning with the WTO Trade Facilitation Agreement. In 2015, tralac supported the Namibian government with redrafting its International Trade Management Bill to ensure that it aligned with Namibia’s participation in SACU. These types of interventions play a crucial role in ensuring that agenda-setting work translates into on-the-ground policy change. Despite these interventions, however, stakeholders point to policy implementation as a consistent barrier. Twenty-five interviewees across TFSP—18 of whom were government officials and development partners—identified policy implementation as a challenge. Specific examples include: • The Ministry of Trade in Malawi explained that after receiving support to develop the country’s AGOA strategy, there was still an unmet need for additional support to implement the strategy • The Malawi Revenue Authority, even after successfully implementing an AEO pilot program, identified the need for more capacity to drive further systems changes and to sensitize border officials on the implications of the AEO program • The EU and Bayer-Monsanto, an international crop chemical company, both expressed the need to train border officials so that they could apply regulations properly • A USAID bilateral mission pointed out the need for implementation funding to help the Ministry of Trade with implementation of AGOA Given these constraints, although TPI’s work on agenda setting is necessary, the lack of implementation capacity hinders the effectiveness of TPI. TPI interventions will not improve trade policy if there is insufficient capacity to translate a policy agenda into on-the-ground changes in regulations and operating procedures. Although TPI did not attempt to address the issue of implementation capacity, it is important to note that TPI’s efforts on agenda setting are likely to be limited in their potential success given these challenges elsewhere in the policy ecosystem. Effectiveness: How effective have TPI and tralac’s capacity-building, training, and policy dialogue interventions been? This section will first look at some of the themes that surface across TPI interventions, before then evaluating the effectiveness of each category of work pursued by TPI (i.e., research, capacity building, and policy dialogue). Mutually reinforcing interventions There is a mutually reinforcing aspect to TPI’s interventions—for example, capacity-building efforts tend to reinforce research efforts. After attending capacity building events, interviewees report subscribing to tralac newsletters and publications and engaging with them. Three interviewees mentioned contacting tralac management directly to get more insights or data after seeing a publication. Policy dialogue interventions also complement each other. The associate managing the AGOA.info website reported that after publicizing AGOA.info at the 2019 Annual Conference, participants followed up with tralac to learn more details about how to further engage with the website. Trade Policy Initiative (TPI) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 108 Audience The audience for TPI interventions is heavily made up of government agencies and academics. The evaluation found that 90% of participants of trade remedy workshops held between 2014 and 2016 occupied roles in government ministries, trade centers, and other government bodies (e.g., competitions authorities). Meanwhile, at least some of the government actors the evaluation team engaged with reported that they receive a high volume of invitations to “capacity￾building workshops.” Furthermore, tralac’s registered readership comprises mostly academics, followed by (to a significantly lesser extent) government officials; these two groups account for 54% of the readership (Figure 24). The few TPI and tralac interventions that moved beyond these audiences have proven to be highly effective and have potential to be less impacted by capacity constraints identified around implementation. The 2018 trade journalist training widened the views of journalists on how to cover trade issues. With the knowledge gained at the training, some journalists attest to reporting on issues in more depth and, at times getting more reactions and responses from important trade actors in their countries. Following a tralac capacity-building intervention (not sponsored by TPI) for COMESA judges, a judge attested that he incorporated knowledge acquired at a 2016 capacity-building intervention into a judgment settling a Malawi Mobile case against the Malawian government. The sole use of English as the language used at events and in publications limits TPI’s reach. Tralac management has identified language as the main reason it cannot sufficiently engage with all audiences on the African continent, particularly Francophone Africa. In addition, a ministerial official in Mozambique explicitly identified language as the main reason Lusophone countries are unable to take full advantage of tralac’s useful content. A survey respondent who has attended numerous tralac conferences and workshops also noted the extent of this geographical limitation in the context of the CFTA: “Tralac needs to expand and bring in participants from the rest of the African continent, especially West Africa. This is for the benefit of the CFTA, which is gaining momentum.” MEL Current M&E practices track output more than outcomes, limiting both the evidence base for evaluation and the ability of TPI to promote its successes. For example, minimal data are collected on how TPI-funded research and the AGOA.info website are used by their audiences. Although encouraging examples have emerged as part of this evaluation, M&E processes are limited in what they address, and in turn reduce the ability of TPI to learn from past work, and to tell an effective story on its successes. Source: tralac, Content Management System (Joomla) and Google Analytics User Summary, 2019 Figure 24: Sectors represented by tralac’s registered readerships (%) 41 13 8 7 6 31 Academic Government Consultancy Legal NGO Other Trade Policy Initiative (TPI) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 109 This section of the report now turns to a review of each category of work conducted by TPI. Research TPI research refers to USAID-funded long and short publications produced by tralac that aim to provide high-quality, accessible trade policy and trade law analyses to enable the advancement of regional integration issues. These publications take the form of working papers and policy publications that reach its registered readership, with heavy representation from academics and government officials. Although use of research is not well tracked, examples were identified of influential readers using TPI research in high-impact ways. Unfortunately, there is little tracking of how readers make use of TPI research, which led the evaluation team to engage with TPI stakeholders to understand how they use the research. Although these insights reflect isolated examples, the evaluation team found that the research does inform tangible actions. For example: • In Malawi and Mozambique, ministerial advisors used the research to inform national policy documents or perspectives • A Malawi-based journalist used tralac’s research on Brexit to write about the implications of Brexit on trade between Malawi and the UK or the EU • A university lecturer from Mozambique assigns the research as reading material to students Citation data for TPI research reinforce the perception that it is highly regarded. TPI research is cited by in academic publications around the world. Both the OECD and the WTO acknowledge that TPI and tralac research efforts are relevant and have a policy impact 86. The ECA has also leaned on TPI research to elaborate an advisory document87. Beyond TPI, other tralac publications have been cited in research documents and reports produced by prestigious universities like Dartmouth College and influential research centers such as the European Centre for Development Policy Management and the Institute for Security Studies. Poor tracking of research application limits the ability of TPI to tell an effective story around its impact. Tracking of page view data or aggregate reader numbers is of limited relevance and reflects an output indicator, whereas the outcome metrics of how frequently and to what extent the research is applied goes untracked. Capacity building TPI capacity building interventions refer to in-person tralac events funded or co-funded by USAID with the objective of imparting knowledge. These include training and awareness events that covered topics such as trade remedies, AfCFTA negotiations, AGOA, and SACU. These events reached predominantly officials of national trade ministries or organizations many of whom have expressed the need for support with implementation. In addition, the events reached journalists and COMESA court officials. Capacity building also included masters and post-graduate degrees offered through the University of Cape Town (UCT), with some scholars being TPI-funded, although this funding is no longer offered by the activity. 86 OECD/WTO, Aid For Trade Case Study (by Sweden): Services Trade, Industrial Development and the African Continental Free Trade Area, 2017 87 Kun’gu A. (EAC), Some issues for consideration by the East African Community (EAC) in the negotiations towards the establishment of a continental free trade area (CFTA), 2017 Trade Policy Initiative (TPI) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 110 Attendees report that capacity-building interventions are useful, as they equip officials with knowledge and competencies they can apply in their work. The training event and awareness event surveys revealed that over 90% of participants found the events to be useful or extremely useful. Accordingly, the NPS of the awareness and training events were 88 and 86, respectively, the highest across TFSP. 88 Over 55% of event participants have applied the knowledge they acquired from the events at work to perform better or have shared their knowledge or advised colleagues about the topics covered in the session (see Figure 25). Specific high impact examples were surfaced during interviews – for example, a COMESA Court Judge noted how a TPI training session ultimately informed a judgement in the case of Malawi Mobile Limited vs. COMESA, following their discussions around the role of trade policy and regional treaties in national law. Figure 25: Percentage of respondents who applied knowledge from TPI training events Stakeholders also noted that the usefulness of capacity-building interventions reflects how well they complement those of international trade organizations. International organizations such as WTO, International Trade Centre (ITC), United Nations Conference on Trade and Development (UNCTAD), and Trade Policy Training Center in Africa (TRAPCA) offer trainings on topics such as trade regulations, trade statistics, and non-tariff barriers largely from a global perspective, while TPI capacity￾building efforts complement them by addressing the topics in the context of a specific region or country 89. For example, in Malawi, a past participant of tralac trainings and trade ministry official said that compared to the broader trainings, tralac trainings were “relevant and specific” in an ever-changing trade policy environment. 88 STP’s NPS score for training events was 65 and for awareness events was 36; SATIH’s NPS scores for training events was 53 and for awareness events was 45 89 Review of trainings listed on WTO, ITC, UNCTAD and TRAPCA websites. 58 56 38 31 31 21 17 12 6 4 2 I can better perform work that I do already I have progressed in my career thanks to the expertise I gained at the training event(s) I have expanded to work in new areas/on new activities I have been asked to share my knowledge or provided advice to others on this topic I have conducted further studies/research on the topic to deepen my knowledge I have lobbied/conducted advocacy related to the topic My organization or business has developed a new strategy/policy I have used this new knowledge to create new business leads I have not applied the training content I have successfully closed additional business deals My organization or business operates in new geographies Source: TPI training survey, 2019; n = 53 Trade Policy Initiative (TPI) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 111 Although the capacity-building interventions are useful, their location limits the audiences who can attend them. During interviews, past participants in Malawi, Mozambique, and Zambia expressed the desire for the events to be made available in their countries so that more people could attend them. Seventy-three percent of training event participants or their organizations would be willing to pay for them to attend TPI trainings while the remaining 27% are mostly constrained by financial resources. Of those willing to pay, 24% would only be able to do so if someone were to cover transport costs, or if the events were hosted in their own countries. Tralac alumni of post-graduate and master’s programs enrich capacity-building interventions. Although the post-graduate diploma and master’s programs were discontinued in 2016, tralac continues to nurture its relationship with its alumni because they enrich the organization’s work. Tralac management view its alumni as being essential to ensuring that tralac stays informed about the latest international and regional trade updates. Tralac leans on its alumni who work at organizations such as the World Trade Organization or the African Union Commission to help it obtain essential trade documents and information. Policy dialogue “Policy dialogue” includes tralac’s flagship event, the tralac Annual Conference, tralac’s monitoring and reporting on evolving trade matters (e.g., the daily newsfeeds), and the AGOA.info website. USAID partially or fully supports each of these activities. The tralac Annual Conference shares useful information and has proven worthwhile to its attendees. Surveys show that 98% of conference attendees found the conferences useful or very useful. Sixty-nine percent of the respondents can better perform their work when applying the knowledge gained from the conferences, 51% have been asked to share or provide advice on the topics covered at the conference, and 50% began working in new areas or on new activities following the conference (see Figure 27). The fact that the 2018 conference had the largest number of attendees thus far furthers these outcomes. In survey feedback, attendees provided further perspectives on their application of the knowledge imparted at the annual conference: • “As a result of the conferences, I have gained deeper knowledge in trade facilitation and hence, my organization is [sic] currently utilized my knowledge to undertake a project on cross border trade” • “The conferences were important in informing our regional strategy, and how to navigate the different countries [sic] level of trade integration in the region to promote U.S and Southern African trade” • “I have applied what I learnt from tralac conferences in speeches that I draft for my Directors and in the policy advice that I give to member states” Trade Policy Initiative (TPI) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 112 Figure 26: Percentage of respondents who applied knowledge from TPI awareness events The AGOA.info website’s global audience has grown since USAID support began. Since USAID began financially supporting the development of the website in early 2013, the number of unique users has grown from about 96,500 to 386,000 as of April 2019—a compound annual growth rate of 26% (Figure 27). Analytics of the website show that between August 2013 and April 2019, the majority of the new users were from the USA (see Figure 28). Source: TPI awareness survey, 2019; n = 81 70 51 46 46 34 28 27 15 6 5 3 My organization or business has developed a new strategy/policy I have used this new knowledge to create new business leads I can better perform work that I do already I have progressed in my career I have been asked to share my knowledge or provided advice to others on this topic I have expanded to work in new areas/on new activities I have conducted further studies/research on the topics to deepen my knowledge I have lobbied/conducted advocacy related to the topic I have not applied the content from the awareness event My organization or business operates in new geographies I have not applied the content from the awareness event(s) Figure 28: Proportion of AGOA.info new users between 2013 and 2019 as of April 2019 by country (% of top 5 countries) Source: AGOA.info, Website Data for AGOA.info, 2019 24 12 7 6 4 United States South Africa Kenya Nigeria United Kingdom Source: AGOA.info, Website Data for AGOA.info, 2019 Figure 27: Number of users of AGOA.info website between 2013 and 2019 Trade Policy Initiative (TPI) – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 113 American users accounted for 24% of overall users while 12% were from South Africa and users from the rest of the world made up the remaining 64% (with each country contributing less than 7% of the total). The website management team shared that some of the users of the website are the U.S. State Department, the Zambian government, and the South African embassy in the U.S. Each of these government offices has contacted the website management team directly requesting information they can use or share, as have traders looking to understand the AGOA eligibility regulations. Insufficient tracking by tralac means that there is little ability to assess the outcomes that the AGOA.info website leads to. Understanding of user volumes is typically an ‘output’ metric, that would feed an ‘outcome’ metric such as application of AGOA.info knowledge. There is insufficient systematic tracking of this as part of current M&E efforts (e.g., through in-website surveys to users), and TPI was unable to provide access to user lists therefore limiting the ability of the evaluation team to review this in more detail. Execution: How well-executed have TPI’s tasks been? Both tralac associates and participants of TPI events are highly satisfied with tralac’s management. Respondents to surveys about TPI events have positive opinions about the content and the organization of the events. At least 80% of capacity-building and policy dialogue event participants agree or strongly agree that the quality of facilitators and the organization of the events were good (see Figure 29). Of the respondents who identified factors that made the conference a success, 35% explicitly identified the quality and/or expertise of the facilitators. A tralac associate who has been significantly involved in TPI (and SATIH) interventions finds tralac’s associate network model convenient as it allows him to contribute his expertise when needed, while still granting him the flexibility to pursue other endeavors. The attractiveness of this model contributes to TPI’s ability to bring in highly credible experts. Figure 29: Participant perception regarding the execution of TPI training events (%) Source: TPI training survey, 2019; n = 53 Trade Policy Initiative (TPI) – Management USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 114 8.3.2 MANAGEMENT Scope of evaluative theme This report will focus on two main themes in its evaluation of TPI’s management: the quality of the management of TPI as a standalone activity and the extent to which TPI has been leveraged effectively by missions and across TFSP as a whole. Synthesized findings Overall, the evaluation team found that TPI’s management elicits strong positive feedback from stakeholders while the management model and structure of the cooperative agreement provide a good balance of flexibility and control to USAID. That said, given the low levels of awareness of TPI on the part of missions, as well as a paucity of efforts to coordinate similar trade policy interventions across TFSP, it is clear that more can be done to fully leverage the potential of TPI. To what extent is the management of TPI effective? tralac’s reliance on the associate network enables TPI to cost-effectively benefit from a network of trade experts from across Southern and Eastern Africa. Tralac’s management explained in an interview that after piloting other working models (e.g., having offices in other countries or having a full-time representative in East Africa) they found them to be more expensive to run than the current network model. Throughout the year, tralac associates represent tralac at events across the continent in their areas of expertise, and as evidenced by survey feedback, contribute to the success of the events. The structure of the arrangement with tralac as a cooperative agreement provides a good balance of flexibility and control to USAID. Although the cooperative agreement itself provides USAID less influence over TPI’s activities, the work planning process allows sufficient input for USAID to steer the direction of activities. Furthermore, the cooperative structure provides flexibility to TPI, enabling it to pursue activities without tight coordination with USG priorities, and minimizing requirements for USAID branding where work may gain credibility from being published under an independent organization. From the perspective of tralac management, however, the nature of the cooperative agreement (vs. an agreement with unrestrictive funding) limits the flexibility with which tralac can respond to new developments and dictates which countries tralac can work in. Meanwhile, USAID also noted that it has less control in a cooperative agreement when compared with an “acquisition”-style procurement tool. Overall, however, the evaluation found that although both parties have identified limitations of the cooperative agreement, it does not appear to impair tralac’s delivery and allows tralac enough space to respond to new developments. To what extent has TPI been effectively leveraged by other parts of TFSP and by missions? From a TFSP perspective, the lack of coordination between TPI and other parts of the program has led to missed opportunities to gain the most value from TPI. Although all three activities have a trade policy dimension, there is little coordination or alignment across the three areas. A tralac associate explained that although SATIH and TPI informally refer stakeholders to each other, there is no formal structure in place to facilitate the referral process. Trade Policy Initiative (TPI) – Management USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 115 There is potential to add value to both activities through better coordination. The trade facilitation component of SATIH often works further down the policy cycle on trade policy topics that are also of interest to TPI, yet no common work planning takes place at the program level to maximize these potential synergies. Furthermore, given the demand-driven nature of SATIH’s trade facilitation programming, leveraging TPI’s agenda-setting efforts to inform what type of work is done, and leveraging SATIH’s connection to the private sector to inform which trade policy issues are truly impacting businesses (which could then become TPI’s priority), could ultimately yield a more relevant and focused set of activities across the program. Bilateral missions report low levels of engagement with TPI, suggesting a further missed opportunity. Interviews revealed that although bilateral missions engage other ecosystem actors with well-informed policy perspectives (e.g., unions and chambers of commerce) to strengthen their policy knowledge, missions have not worked with TPI as part of these efforts. In Malawi, for example, the USAID bilateral mission and other development partners supported the Malawi Confederation of Chambers of Commerce and Industry (MCCCI) to better understand and prepare for discussions about the Control of Goods Act. Much of TPI’s existing work could inform bilateral missions and their perspectives without significant additional effort. Not making this connection represents a missed opportunity to better use and publicize TPI assets. Trade Policy Initiative (TPI) – Sustainability USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 116 8.3.3 SUSTAINABILITY Scope of evaluative theme This section of the report considers whether or not TPI’s results are likely to be sustained beyond the LoP, as well as how equitable those results have been to date. Synthesized findings This evaluation concludes that TPI’s results would be sustained given that tralac would continue to operate, although activities would be reduced in scale while some activities (e.g., AGOA support, which is fully funded by USAID) would likely cease altogether. From an equity perspective, there is room for improvement, with TPI’s audience having greater male representation, although plans are in place to diversify this through creation of a network of female policymakers. To what extent would the results of TPI in improving trade policy be sustained in the absence of USAID support? The nature of TPI interventions means beneficiaries continue to benefit from knowledge already imparted, however there is an ongoing need for further support in response to evolutions of the trade policy environment. It is also important to recognize that knowledge that is relevant today becomes less relevant over time. Trade policy environments continually evolve and new global and regional agreements need to be incorporated into the national agenda and implemented— requiring more enduring work and capacity. It should also be noted, however, that there are inherent sustainability challenges associated with the lack of implementation capacity, as outlined in relevance, in that it prevents outputs translating into sustainable outcomes in the first place. Ongoing work to push for trade policy improvements would continue without USAID support, given that tralac receives its core funding from SIDA, but at a smaller scale and with reduced AGOA focus. tralac’s funding model is not completely dependent on USAID funding, but it would not be able to deliver the volume of activities in the USAID work plan without this support. Furthermore, tralac management noted that USAID is the sole funder of AGOA-focused activities (e.g., research related to AGOA, and the AGOA.info website), which means that if USAID funding were to diminish, tralac would decrease its focus on these activities. Would people be willing to pay for the services of a TPI-like activity in the absence of USAID funding? If no, why is this the case? Reflecting the quality of TPI events, most participants and/or their organizations would be willing to pay to attend them, reflecting the value people place on them, but would be limited by financial constraints. Fifty-five percent of the respondents to the awareness event survey who answered a question about their ability or inability to pay indicated that budgetary constraints were a major factor. Some participants said the following: • “Due to budgetary constraints, paying for such important workshop [sic] is challenging.” • “Despite the strong willingness from me and my organization to attend such events, we lack the necessary resources” Trade Policy Initiative (TPI) – Sustainability USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 117 • “As a non-profit organization, funding is scarce to attend these events. However, if the content is relevant and of value to the organization, we will be prepared to contribute to the costs of participation. We will, however, be limited to participating in only a few events” This suggests that a sustainable business model for TPI-type activities would be difficult, but that some of the audience may be prepared to consider co-funding or paying in full for participation. Is the beneficiary base of TPI sufficiently equitable in nature? As a reflection of the gender imbalance in the trade policy space, the current base of TPI event participants has higher male representation than female. Interviews with the tralac management team revealed that when the implementer opens applications for training events and workshops, most of the applications are from men. Survey results show that 61% of the respondents to training and awareness events surveys were male, and 39% were female. Looking at the registered tralac readership, although the gender of the majority (65%) of readers is unknown and 3% prefer not to disclose, 20% are known to be male and 12% are known to be female. Tralac recognizes this imbalance and, with the support of TPI funding, is planning to develop a women’s network. The implementer is launching the pilot of a women and governance network with ten women from government ministries from the African continent. Although this is ongoing and therefore not within the scope of the evaluation, TPI funding will ultimately contribute to making the beneficiary base more equitable from a gender perspective. Trade Policy Initiative (TPI) – Conclusions USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 118 8.3.4 CONCLUSIONS Overall, the evaluation found TPI to be effective in delivering the activities it focuses on today. The lessons learned and relevant opportunities therefore look at how a combination of strong partnerships, wider stakeholder networks, and refined activity design could amplify the effectiveness and reach of TPI, in response to the ecosystem challenges around a lack of capacity in other elements of the policy cycle. Although opportunities for other activities are delineated around timespan, given that the current cooperative agreement ended in 2018 and has been re-launched this set of opportunities aims to inform both current programming and the next PAD. 8.3.5 RECOMMENDATIONS AND OPPORTUNITIES • Take steps to reinforce TPI’s efforts by addressing implementation capacity gaps – There are multiple options for doing so, depending on the level of ambition to scale TPI’s efforts: o Drive alignment across TFSP activities – e.g., ensure that SATIH builds off the agenda-setting work of TPI to improve implementation, and vice versa o Partner and coordinate with other implementing partners – ensure TPI’s efforts align with the efforts of other donors supporting trade facilitation implementation o Increase funding of implementation activities – At the same time, potentially bring in additional implementers to provide expertise in these areas in a manner that complements the agenda-setting work of TPI • Expand audience reach to ensure that TPI interventions benefit more stakeholders – To do this, TPI and/or USAID could: o Re-focus activities to include other high-potential audiences that can tangibly drive policy outcomes and are less susceptible to the implementation capacity challenges of government ministries; for example, build on the successes of work with journalists and COMESA, or consider catering to private sector audiences o Offer trainings and research materials in French and Portuguese – interviewees expressed although TPI’s work is valuable for the continent, the use of English makes the content inaccessible to franco- and lusophone stakeholders • Build greater awareness of TPI within USAID missions – in service of greater engagement with TPI content, both directly within USAID and the ability of bilateral missions to make others aware of it. This could be achieved through roadshows (as SATIH and STP have used), or including USAID bilateral in key events such as the annual tralac conference • Improve indicator design and tracking mechanisms to capture the effectiveness and value of TPI interventions – with a particular focus on proving the outcomes, and not just outputs. With this, provide greater clarity on the specific outcomes USAID/REGO desires within the project’s goal of improving the quality of trade policy in the region • Identify opportunities to introduce co-funding mechanisms where participants contribute to costs – enabling TPI funds to support more stakeholders or interventions. This recognizes that a considerable number of event participants expressed a willingness to pay to attend TPI events (although not all may be able to financially) • Continue to support and expand initiatives such as the tralac women’s network – Improve the equity of TPI’s audience through greater female participation Overall Trade and Food Security Project (TFSP) Findings – Recommendations and opportunities USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 119 8.4 OVERALL TRADE AND FOOD SECURITY PROJECT (TFSP) FINDINGS The report now turns to assess TFSP as a whole against the same set of evaluative themes, in addition to identifying best practices across the project. From a relevance perspective, TFSP’s balanced portfolio approach (combining transaction-oriented approaches with longer-term systems change efforts) has enabled it to contribute to a relevant set of challenges, although its actions have been insufficient to address challenges fully, in part given that it has worked on specific issues where upstream and downstream challenges remain unaddressed (reducing its effectiveness). The toolkit used by TFSP, in terms of both private-sector led approaches and regional programming has allowed USAID/REGO and TFSP to bring unique value. That said, there are missed opportunities to build on TFSP as a platform – for example, ensuring its work is uniquely regional (maximizing its value add, while avoiding overlap with the remit of bilateral missions), leveraging regional and Africa-wide relationships (as opposed to single-country actors), creating greater synergies with other USAID programs (e.g., the Africa Private Capital Group (APCG)), and fully leveraging relevant brands (both USAID and SATIH). From a management perspective, TFSP was conceived as a more connected set of activities under a joint PAD, although the extent to which synergies were realized was limited beyond this point. The evaluation team did identify missed opportunities – for example, to better leverage relationships across the three activities, and to align programming around common priorities in places. That said, these synergies are not significant, but equally would likely not require significant effort to realize. The evaluation team also looked at the contracting structures and management models designed at project level. It noted challenges in both SATIH and STP in driving effective contract management, with both using performance-based contracts with few tools to effectively hold the contractor to account (e.g., fees at risk). These contract management challenges were exacerbated by USAID/REGO capacity constraints. The COR model, as to whether one individual or several filled the multiple COR models, was a source of debate amongst stakeholders; both models have pros and cons and reflect a trade-off that should be considered by USAID/REGO during future program set up. The extent to which the project advanced sustainability at TFSP level was limited, in part given the nature of the project given work almost exclusively took place at activity level. That said, the evaluation team did explore whether there were missed opportunities. Given the size and scale of the project, there were opportunities for it to have a bigger influence and voice both within USAID/REGO and in the external ecosystem. However, USAID/REGO capacity constraints and poor MEL limited the ability of the project to form evidence-based opinions, harness learnings, and in turn influence wider trade and food security dialogues. The project-level evaluation also identifies best practices across the project, largely within and across the three activities – these include areas where a highly focused remit enabled better results, the benefits of closer geographic proximity and local relationships, co-creation of solutions with local actors, using pilots as a way to encourage actors to sustainably take on solutions, the benefits of the balanced portfolio approach, and of strong alignment with country priorities. The report now goes on to further detail these findings. Overall Trade and Food Security Project (TFSP) Findings – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 120 8.4.1 RELEVANCE Scope of evaluative theme In exploring the relevance of TFSP, the evaluation team took a different approach compared to the activity￾level evaluations – responding to the fact that TFSP has undertaken little activity at the project level, and largely reflects three loosely coupled activities under a single PAD (as will be elaborated on further within the Management evaluative theme). With this, this section looks at the extent to which the philosophies and approaches that informed TFSP as a whole (the role of private sector-led approaches, and regional programming) has been an effective set of tools, the alignment of these tools and the activities delivered to the intention set out in the PAD, and whether the project has missed opportunities to build on its strengths and positioning. Synthesized findings With this, the evaluation team found that although at this point of the program there is insufficient evidence to robustly compare effectiveness, both private sector-led approaches and regional programming has proven valuable, and are highly aligned with the direction of USAID priorities. There have been missed opportunities to maximize the value of the regional remit (e.g., to act as a single fact to cross-country actors), to forge relationships with continent-wide actors (given the sub-geographic region focus of SATIH in particular and USAID/REGO’s structure more broadly), and missed opportunities to build synergies with other USAID programming (including bilateral missions, and other programs such as APCG). Furthermore, although alignment with the PAD was generally strong, there are areas, particularly around gender and informal trade, that were prioritized in the PAD but did not appear as strong priorities within the programming delivered. To what extent is TFSP’s work set up for success when reviewed against the wider ecosystem? TFSP’s balanced portfolio approach contributes to it addressing relevant ecosystem challenges – which requires work that delivers both short-term transactional wins and long￾term enabling environment changes. The transaction-oriented work – whether that is connecting businesses into trade, facilitating investment through transaction support, or providing grants to seed companies to co-finance seed to be traded under HSRS – is crucial in proving that TFSP is delivering demonstrable results and building momentum. This work can also feed wider market systems change – with successes playing a market signaling effect (whether that is encouraging investors to consider a region having seen previous successes, or building the profile of Southern Africa businesses as attendance at trade fairs has done). That said, there are challenges in the ecosystem that require longer time horizon, systems￾level support that cannot be addressed through firm-level support alone. In SATIH, TFSP played an additional role in advancing longer-term systems change in establishing structured trade institutions (commodity exchanges, grain market information systems, and the warehouse receipt system). Likewise, STP plays a crucial role in driving longer-term policy harmonization. In both activities, these interventions required taking a long-term view and involved a significant amount of stakeholder engagement to bring about changes in the trade enabling environment. The ability of TFSP to balance these two approaches has contributed to the successes seen. Overall Trade and Food Security Project (TFSP) Findings – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 121 Despite tackling necessary challenges, TFSP’s efforts were insufficient to achieve its outcomes; the breadth of TFSP’s scope means many unaddressed upstream and downstream challenges hinder its effectiveness. This challenge arose consistently across the project, and reflects a lack of comprehensive systems analysis and stakeholder mapping. For example, frequent interference in the political economy undermined SATIH’s work both to strengthen structured trade institutions themselves and encourage lending to users of the WRS. The political economy challenge remained unaddressed, yet was frequently identified as a ‘deal breaker’ by stakeholders. Similarly, the lack of implementation capacity in the system undermined the effectiveness of TPI’s agenda-setting interventions, particularly given its primary audience was government stakeholders (where capacity constraints were greatest). With this, TFSP as a whole would have benefited from more robust systems analysis, and greater deliberateness around what problems it was addressing, the extent to which other prerequisites were in place or being addressed by others, and in cases a better perspective on whether attempting to address the underlying challenges was well-informed or feasible within the current context. Although the work TFSP undertook aligned with the PAD, the insufficiency of TFSP also extends to it not addressing the full range of relevant ecosystem issues identified in the PAD. The PAD itself set out clear ambitions to improve agricultural productivity and increase trade. From an agricultural productivity perspective, TFSP has made progress against the three priorities outlined in the results framework, although some gaps remain: • In line with the evaluative findings, efforts to improve seed availability have been well executed and appear promising, albeit will require further effort to reinforce • In improving technology, management practices and innovations, SATIH has achieved Y1 and Y2 targets against application of new technologies and management practices, although missed targets around forming of new PPPs. STP meanwhile has significantly outperformed its target on application of new technologies and practices • Improving regional policies has been a focus area across the program – including through work to harmonize seed policies, That said, given the ambition to advance ‘"an inclusive functioning regional institutional architecture for governance in the agricultural sector”, this work appears to fall short of the full ambition as articulated in the PAD itself In advancing trade, more significant challenges arise which were outlined throughout the evaluation; the fact that SATIH, which was the most significant activity in supporting this agenda, often delivered transactional results without fully translating into more systemic changes. With this: • Efforts to increase regional value-chain competitiveness did exist, although oftentimes this reflected work to connect trade-ready businesses into opportunities, rather than addressing underlying competitiveness issues more systemically • Improving the regional trade and investment enabling environment is reflected in the transaction￾oriented successes of the program, although the fact that this did not translate into more systemic changes in many cases suggests these efforts fall short of the full vision. From a trade facilitation perspective, TPI has made contributions, although the impact of these was hindered by a lack of implementation capacity in the system, as has SATIH, although again reflecting a relatively small contribution in budget terms against a broad and significant challenge across the region Beyond the results framework, the PAD also laid out gender analysis that explored both the role of women and informal trade. Despite this focus within the PAD, this did not translate sufficiently into the Overall Trade and Food Security Project (TFSP) Findings – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 122 activities. There was little focus, if any, on the role of informal trade. From a gender perspective, although activities exist that incorporated a gender lens, particularly within SATIH and TPI, there was little evidence of a more holistic gender strategy (an ambition that the PAD set out). To what extent is TFSP aligned with USAID global priorities and direction? TFSP’s emphasis on private sector engagement (PSE) aligns closely with USAID’s new global priorities. Across the program, TFSP has pursued strong private-sector-led approaches. This is most evident in SATIH’s transaction-oriented approach, but within STP, too, there has been significant work to improve the enabling environment for private sector seed companies and engage directly with them. STP has also provided direct support to private sector seed laboratories. TPI has less of a direct focus on working with the private sector, although its trade focus connects it directly with private sector issues, and there are examples such as journalism trainings that have led to positive engagement from the private companies around issues they find pressing. With the launch of Prosper Africa and the United States International Development Finance Corporation (USIDFC), there are further opportunities to build on the successes of TFSP. For example, where countries require greater financing for capacity building than STP is able to provide, it could look to fill the gap by using USIDFC’s tools (e.g., debt financing). Finance and Investment within SATIH could scale its activities further using direct USIDFC investments (that may enable other deals to take place that the private investor community is less interested in funding). Furthermore, the launch of Prosper Africa may create the opportunity to further US business interests through incorporating existing businesses within the TFSP network into US supply chains. How effective have the approaches pursued by TFSP at project level been? The evaluation team considered the extent to which the tools of TFSP have been well placed to drive the project’s objectives. This section looks first at the use of private-sector-led and transaction-oriented approaches, then considers the regional nature of the program, before identifying any missed opportunities to maximize the program’s effectiveness. Private-sector-led approaches Although available data are limited, TFSP’s private-sector-led approach appears to yield both development and trade benefits. The project generated insufficient data for the evaluation team to conduct a rigorous study the effectiveness of TFSP’s private-sector-led approach—in terms of the development benefits it generated. SATIH generates data only at the firm level, which makes it difficult to quantify any knock-on development impact on communities. Stakeholders nevertheless provided multiple examples of firm-level support leading to both trade and wider development benefits, such as mission￾driven companies who source their products from rural communities and work with smallholder farmers benefiting from trade linkages that led to exports to the U.S., that ultimately improve prices for smallholders. TFSP’s firm-level private sector support can yield wider market systems changes and create additional indirect trade and development impact. For example, engaging the private sector on pilot initiatives can serve as a proof of concept of the feasibility and profitability of programs with spillover effects on similar companies to encourage them to venture into new markets. This was reflected in Overall Trade and Food Security Project (TFSP) Findings – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 123 facilitating transaction support to entice other investors to see attractive investment opportunities in Southern Africa, or sharing export costs with seed companies to encourage regional exports. TFSP’s private-sector-led approach to development has garnered the respect of beneficiaries, ecosystem, actors, and other donors. Donors interviewed, particularly the EU, shared their ambition to emulate USAID’s private-sector approach to development in their next development strategy. They noted that working with the private sector helps achieve long-term development outcomes as it builds self-reliance and empowers the local private sector to finance solutions to development challenges. Seventeen interviewees broadly echoed these sentiments, recognizing the potential for greater use of private-sector engagement approaches in development. Regional programming TFSP’s regional nature allowed it to pursue a number of programmatic approaches that an individual bilateral mission alone could not. A number of approaches were evident in TFSP’s activities: • Regional linkages – USAID/REGO has the ability to act as a “single face” to regional stakeholders and represent multiple bilateral missions (e.g., in linking one stakeholder—such as a buyer, investor, or trade fair—with multiple companies across the region) • Regional strategies, executed through local interventions – USAID is able to pursue interventions that may have a regional layer, but require national-level solutions or systems to be in place. For example: o SATIH pushed the Zambia Grain Market Information Service (ZAGIS) forward in Zambia with the intention of sharing this information with neighboring countries interested in trading with Zambia. Neighboring countries shared best practices to facilitate the process of tailoring a market information system to the Zambian context o STP’s work by its nature required both regional institutional and policy change, complemented by national policy change. USAID/REGO had a unique ability to operate at both levels • Expertise on regionally-oriented issues – for example, topics such as structured trade, trade facilitation, and AGOA all have an inherently cross-country outlook, where access to regional expertise with a view across countries is beneficial • Demand-driven programming from bilateral Missions – promoting a regional buy-in mechanism has allowed USAID bilateral missions to fund country-level programming that aligns with USAID/REGO’s objectives. This was evident in SATIH’s work in Zambia where the mission asked USAID/REGO to implement some of its programmatic tasks under trade facilitation Missed opportunities Although TFSP is a well-positioned vehicle, based on its private-sector approach and regionally-oriented outlook, TFSP missed opportunities to make full benefit of these approaches. Although USAID/REGO had a clear regional value-add, there were cases where the delineation between USAID/REGO’s remit and that of the bilateral missions was less clear. For example, in Botswana SATIH developed market assessments of the Botswanan beef sectors and subsequently engaged with local stakeholders (e.g., Botswana Investment and Trade Centre) to discuss opportunities to export beef to the U.S. under AGOA. Although there is a trade element here, the extent to which this is distinctly regional-level work is unclear. Overall Trade and Food Security Project (TFSP) Findings – Relevance USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 124 TFSP missed the opportunity to leverage truly regional and continent-wide relationships that could have yielded greater trade. The majority of private sector actors engaged were engaged on a single-country level. Where SATIH engaged with regional actors with a presence across companies, this was often in the context of a specific country. In addition, with major multi-national corporations often viewing Africa as a single region, SATIH may have benefited from greater engagement with those – giving these companies the opportunity to engage with multiple African countries through a single face. The extent to which this was possible is to some extent limited by the current set-up of three sub-regional hubs, without strong relationship management coordination across all. That said, there was little evidence that SATIH had attempted to forge relationships of this nature. TFSP missed opportunities to create programmatic synergies with other USAID programs. At the regional level, SATIH had opportunities to create stronger synergies with the Africa Private Capital Group (APCG), which also works in the same region and is focused on transaction facilitation. A greater coordination between SATIH and APCG could have potentially given rise to a joint pipeline management and complementary activities to reconsider failed deals. Similarly, at the country level STP missed opportunities to forge closer working relationships with other USAID seed-related projects operating. TFSP failed to fully leverage the brand value of SATIH and USAID. SATIH is an established brand in the region and USAID is a leading actor in the trade and food security space in Africa. Both entities benefit from high brand recognition and have the potential to open doors and command a higher level of engagement from stakeholders; however, both of their names were applied and communicated inconsistently. In SATIH, investors engaged in transactions facilitation often had limited interactions with USAID, and despite being aware that the funding originates from USAID, had stronger brand awareness of CrossBoundary. These interview findings are also corroborated by the mixed perspectives gathered through interview codification. Within SATIH, 25 participants rated their awareness as high, while 30 rated it as medium, and 11 as low. Overall Trade and Food Security Project (TFSP) Findings – Management USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 125 8.4.2 MANAGEMENT Scope of evaluative theme Given the nature of TFSP at the project level, which reflected three distinct activities that were managed largely independently, project-level findings on management are less extensive – focusing primarily on whether there were more opportunities to manage the project in a more integrated manner, and exploring the role of contracting structures that were designed at project level. Synthesized findings The evaluation team found that there was little effort to drive synergies across the three activities beyond the inception of the project, where a common PAD was forged. That said, the distinctness of the three activities means the synergies are not extensive, but equally could be relatively easy to capture. In addition, contracting structures and contract management practices were a recurring challenge in both parts of the project that used acquisition-based procurement structures. This in part relates to capacity constraints at USAID/REGO. What role has TFSP-level management played in adding value above the activities? The overall structure of TFSP adds little value beyond that delivered by the three activities in isolation. Although the three activities share common themes, TFSP at project level reflects little more than a contracting body. With this, the three activities reflect little more than the sum of their parts. There are missed opportunities in this set up to drive synergies across the three activities. For example, SATIH, STP, and TPI were often working on similar trade policy related issues, yet there was no formalized channel to share best practice, or to align activities where there could be synergies (e.g., SATIH building on the agenda setting work of TPI with more pointed technical assistance). There was also potential to better manage relationships across programs – for example, bringing tools of SATIH to private sector seed companies that STP was engaging with. Ultimately however, the distinctness of the three activities means that these synergies are not overly significant, as the three activities benefit from different specialized expertise specific to that area. To what extent has the contracting structure and USAID/REGO management impacted the performance of TFSP and its activities? As outlined in SATIH and STP, use of performance-based contracts without appropriate contract management tools has hindered performance across TFSP. This to an extent conflicts with performance based contracting leading practices, including the need for: • Clearly defined and achievable outcomes • A baseline against which progress will be clearly measured • A framework to monitor and appraise performance—including a measurement mechanism that provides good insight to the contract issuer on the extent to which outputs and outcomes are being achieved and meet the standards of the contract[1] [1] Best Practices for Performance Based Contracting, The White House Overall Trade and Food Security Project (TFSP) Findings – Management USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 126 • A set of tools to hold the contractor to account for outcome delivery (these can include positive / negative financial incentives – e.g., fees at risk for the contractor) The difficulties around effective contract management was in part linked to USAID/REGO staffing constraints. Several current and former USAID/REGO staff pointed to a lack of capacity to actively engage with contract management. This naturally extends into the challenges identified around low levels of value add at the TFSP level, where USAID/REGO is not sufficiently staffed to forge connections across the distinct activities. USAID/REGO turnover has contributed to management challenges, although despite use of differing models there is little evidence to prove the optimal model for filling the COR roles. At the TFSP level, differing management models have been used throughout the project – including distinct individuals filling the COR roles for SATIH and STP in the early days of the current iteration, before moving to a single person filling both COR roles at the time of writing this report. Turnover was reported to be disruptive, given the importance of a good working relationship between the COPs and CORs, in addition to the time required to onboard and fully understand activities of this complexity. With regards to the optimal management model, stakeholders who the evaluation team engaged with on this topic expressed differing opinions (at times stakeholders were divided themselves) on whether a single individual managing the project added more value (e.g. through offering a single face to the bilateral missions, and being able to represent the entirety of TFSP within the system), or whether distinct managers was preferable (e.g., given their ability to provide more specialist expertise and management, and a de-risking element to distinct CORs). Overall Trade and Food Security Project (TFSP) Findings – Sustainability USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 127 8.4.3 SUSTAINABILITY Scope of evaluative theme In light of the approach to managing the project as three largely distinct activities, rather than as a more interconnected project, the majority of sustainability findings are reflected in the individual activities. The evaluation team did explore, however, whether there was more that TFSP could have done at project level to drive sustainability, and create change that outlasts the LoP. Synthesized findings The evaluation team found a missed opportunity for TFSP to leverage the learnings of the project as a whole in order to inform USAID programming as well as shape the narrative as part of the wider ecosystem. Exerting such influence would ensure project learnings extend beyond the LoP of the individual activities. TFSP’s shortcoming in this regard is partly explained by USAID/REGO’s lack of capacity, along with the project’s inconsistent performance in generating usable MEL data that provides robust evidence on effectiveness. To what extent was sustainability of the interventions driven at the TFSP-level? A project the size and scale of TFSP has potential to drive a more lasting impact beyond its individual activities. This reflects the ability of a project of this scope to have a strong influencing voice, to shape the agenda, and to generate learnings that inform the future programming of both USAID and other development actors. The extent to which TFSP has played this role at project level is low. Constrained capacity within USAID/REGO reduced the extent to which the team were able to draw out learnings from the project’s work. Furthermore, although during a mid-term performance evaluation the project would typically expect to have generated extensive data on the approaches pursued, enabling comparative studies of their effectiveness, a piecemeal approach to programming in places means this has often not been possible. Furthermore, limited recognition of TFSP, and at times even low recognition of the specific activities, hindered the extent to which what could have been a major flagship project was able to establish a brand and have a longer-term influence on the region. Overall Trade and Food Security Project (TFSP) Findings – Best Practices USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 128 8.4.4 BEST PRACTICES Each activity evaluation, in addition to the TFSP project-level evaluation, has elevated the lessons learned from the project to date. This section turns to best practices, elevating strong aspects of the project for future consideration. Table 11: Best Practices Best practice Examples and evidence Be deliberate and focused STP and TPI have both benefited from a clear focus, with a clear set of outcomes USAID/REGO and implementers can manage against. This contrasts with SATIH, which, despite strong indicator metrics being in place, has suffered from a broad remit and wide activities that often do not translate into a more systemic level change. Build geographic proximity The in-country presence provided through STP Country Advisors has enabled strong follow up and good local relationships, while many actors have identified the need for on-the-ground support to move activities forward. Although this capacity could be provided in many ways—e.g., greater support from bilateral missions—it plays a crucial role given the broad reach of a regional project and the risks of diluted effort Co-create solutions Ecosystem actors and bi-laterals had great perspectives on what was really needed and often provided valuable local context—often playing a crucial role in translating D.C. and USAID/REGO priorities into locally relevant solutions. This played out particularly in STP work planning exercises, where the input of the missions was recognized as highly valuable Pilot and prove the potential A pilot-based approach, when well designed, has potential to improve the sustainability of TFSP interventions. Whether that be proving the business case of attendance at trade fairs, or STP’s work to encourage seed companies to trade seed under HSRS—using a pilot approach has the potential to amplify the impact of the funding beyond the initial intervention Pursue a balanced portfolio TFSP’s balanced portfolio of activities allowed it to pursue complex projects that address structural issues in the enabling environment (policies, standards, access to market information) while simultaneously tackling issues such as export competitiveness of goods and increasing access to capital for SMEs. This balance has enabled the project to pursue the multi-pronged approach needed to drive trade and food security Align with countries’ priorities Where the project aligned closely with local priorities, it often produced a multiplier effect. For example, STP’s beneficiary-led approach ensured that it was tackling relevant problems, whereas when SATIH addressed challenges on the local agenda, local actors often provided additional capacity to see the change through. Overall Trade and Food Security Project (TFSP) Findings – Conclusions USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 129 8.4.5 CONCLUSIONS TFSP is largely the sum of three individual activities, with minimal management at the project level. The distinctness of the remits of the activities limits potential synergies, although the evaluation team did identify missed opportunities that inform the discussion of opportunities at the project level. The evaluative findings on the relevance of the tools used by the project, the regional approach, and challenges to sustainability at the project level also inform this set of project-level opportunities. Many of these opportunities feed into the next PAD, given that the existing project set-up remains largely fixed for the LoP. 8.4.6 RECOMMENDATIONS AND OPPORTUNITIES • Set a stronger project-level vision, informed by holistic systems analysis – This will enable distinct activities to be aligned against a common agenda, and also provide USAID/REGO with the insights needed to effectively challenge the implementing partner and form a strong opinion on where there is (and is not) appetite to intervene • Connect TFSP into the wider African continent to leverage opportunities across the sub-regions – Link with other USAID Trade and Investment Hubs on the continent and address regional stakeholders through a single face, ultimately in service of greater trade volumes • Create programmatic synergies with other USAID programs – Foster greater coordination between TFSP and other USG programs in order to give rise to synergies between projects and identify areas of programmatic complementarity; this can also support more efficient knowledge sharing between projects with similar focus • Build on emerging USG priorities – Understand opportunities to work with USG initiatives that place private sector development at the heart of development practice. These include the United States International Development Finance Corporation, and alignment with Prosper Africa • Maximize the potential of the portfolio approach – Continue to deliver both transaction￾oriented work and address structural challenges (while considering more private-sector led approaches to structural issues) • Deepen collaboration with other donors – Build on synergies and better align where donors are tackling a key part of the puzzle that enhances the effectiveness of USAID’s work or are adopting a similar approach to development (e.g., private sector development) • Align with local priorities – To increase the acceptability and successful implementation of the project. Country alignment is needed to ensure that relevant country actors will own the various tasks implemented beyond the LoP, and can garner additional capacity to support implementation • Strengthen capacity of USAID/REGO to provide strategic oversight, drive shared lessons learned, and support effective contract management – Greater capacity could enable a team to drive synergies, share insights across the activities, facilitate senior-level stakeholder management (acting as a single face for the project), and enable more time to be dedicated to setting direction and holding the contractor to account for performance • Refine project structure and management setups – Particularly in light of future scope of a TFSP-like project, make trade-offs between, on the one hand, the ability of a single individual filling COR roles to provide consistent direction and offer a single face to stakeholders across Overall Trade and Food Security Project (TFSP) Findings – Recommendations and opportunities USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 130 the project, and on the other, the benefits of more specialist expertise and a de-risking effect of more distributed management associated with separate individuals filling COR roles for each activity. With this, also consider the potential to use locally employed staff in COR roles that provide longer-term continuity • Leverage the insights of the project as a whole – Amplify these beyond the individual project to shape the work of USAID and other development partners, while also using the project as a vehicle to influence and shape the agenda based on the work of TFSP; in turn, improve the sustainability by creating lasting outcomes that extend beyond the lifespan of individual activities Annex 1: Overview of SOW and Evaluative Questions USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 131 ANNEX 1: OVERVIEW OF SOW AND EVALUATIVE QUESTIONS Statement of Work The full SOW is available on request from the USAID/REGO MEL office. In summary, the remit of the evaluation is summarized below. Objectives of the evaluation USAID/Southern Africa (USAID-SA), through the Regional Economic Growth Office (REGO), is currently implementing the Trade and Food Security Project. As the Project approaches the mid-point of implementation, USAID-SA, must evaluate the progress of the project relative to its implementation objectives and results achieved. To this end this performance evaluation of Trade and Food security project is to: • Provide specific information about the gaps and opportunities for the project that can be acted upon by project staff and USAID management staff in the implementation of the remainder of the project; • Understand the effectiveness of the project in meeting the intended results and; • To identify areas that needs to be modified/ improved to increase the likelihood of success. USAID REGO technical and management teams will utilize the findings for evaluation to understand the gaps in implementation. USAID-SA REGO and other stakeholders (the implementing partners as well as Government counterparts, SADC) will utilize the findings and adopt recommendations from the performance evaluation to understand the gaps in implementation and use the findings of this evaluation to inform future REGO programming. In addition, the evaluation should highlight promising practices, both technical and managerial, that should inform future REGO technical approach, organizational structure, and highlight any other challenges not addressed by the above topics. Deliverables and milestones • Inception Report: The evaluation team will prepare a detailed inception report. The report must detail the evaluation design and operational work plan, which must include the proposed data collection and analysis methods in order to address the Evaluation Questions noted in the main report and in the annex to the SOW • Debriefing / check-in: The team must present the major findings of the evaluation to USAID/SA through a PowerPoint presentation at the close of fieldwork and before the team departs the region. • Mid-term Briefing and Interim Meetings: The evaluation team will hold a mid-term briefing with the Regional Economic Growth Office on the status of the evaluation, including potential challenges and emerging opportunities. • Preliminary Draft Evaluation Report: The team must submit a first draft of the report and deliver a power point presentation that will be shared with USAID/SA’ s Regional Economic Growth Office (REGO) and COR for preliminary comments prior to final Mission debriefing. Annex 1: Overview of SOW and Evaluative Questions USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 132 • Draft Evaluation Report: A draft evaluation report of the findings, conclusions and recommendations. • Final Presentation: The evaluation team will hold a final presentation, in person and/or by virtual conferencing software to discuss the summary of findings, conclusions and recommendations with USAID/SA. • Final Report: Based on the provisions of the USAID evaluation policy, a formal and final evaluation report shall be presented to USAID/SA and the COR. The final report must incorporate the team’s responses to Mission’s comments and suggestions. Evaluative Questions As part of the inception phase, the evaluation team laid out a full list of evaluative questions, which are provided below. Note that for the purposes of this report, many of these questions were grouped together to aid readability. The inception report (available on request) provides a full evaluative design matrix that covers the evaluative questions, proposed analyses, data sources and relevant tools. SATIH: Evaluative questions Relevance • Does the ToC strike a balance between enabling a wide array of impactful activities while providing focus and a useful evaluative frame? • Is the logical flow of the intermediate objectives and intermediate results in the ToC consistent and are the assumptions underpinning that flow reasonable? • Are the critical assumptions that informed the development of the TOC still valid and what project adaptations were made because of changing context? • How is SATIH identifying activities to focus on and is this process effective? • Is SATIH undertaking the best possible activities to achieve its intermediate results? • How effectively implemented and to what extent was the result delivered additional for each task? • What impact did delivery of the tasks have? (note: impact refers to the results delivered by each task. This will include a triangulation of results captured in activity reporting, with beneficiary perspectives on the results and the extent to which these can be attributed to the tasks)? Management • Does the governance framework and team culture (roles, decision points, information flows) of SATIH promote effective decision making? • In hindsight, did the procurement process lead to sub-optimal project design in any way? • Does the current contract strike the appropriate balance between flexibility, incentives and accountability? (including appropriate M&E indicators, flexibility in budget allocation, co￾development of workplans) • What are reporting and information management guidelines and practices between contractors/ contractees and are these sufficient? • How did SATIH manage budget cuts from U.S. Congress across its programs, and what were the implications for the effectiveness and efficiency of programming? Annex 1: Overview of SOW and Evaluative Questions USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 133 • Who were the non-USAID potential partners and co-investors focused on the same objectives as SATIH in the space how effective was communication and coordination with them? • How effectively did USAID work with bilateral missions in country programs? Sustainability • How can programs or USAID REGO’s processes be designed to manage changes in political priorities? • What mechanisms have been put in place, by USAID/REGO or the implementing partners, to ensure sustainability of SATIH’s achievements? • Will key beneficiaries be able to self-sufficiently continue working towards project objectives in the absence of SATIH? • To what extent did SATIH develop activities that explicitly targeted women and how effective were these activities? Opportunities • Scale and focus of activities: Based on the interviews conducted with stakeholders, are there any opportunities that emerge for scaling an activity or adding a new one, including through new partnerships? • Strategic positioning & structure: Based on findings regarding potential synergies and overlaps across the three projects, are there opportunities to enhance the structure and strategic positioning of the activities? • Operational efficiencies: Are there any opportunities to shape organizational structure or processes to improve effectiveness and efficiency of operations? Best practices • Based on the findings and analysis of the preceding questions, what are the key strategic, programmatic, technical, and managerial features of the Project that should be considered when developing a new Southern Africa Trade and Competitiveness Hub Project in the region? Specifically, where are the best practices in: o Project selection and design processes o Tasks or activities that reflect wider best practices that could be scaled across TFSP o Management processes and operating model (including clarity of roles and responsibilities, processes, governance, metrics and incentives) STP: Evaluative questions Relevance • Is harmonization of seed regulations in the region the necessary and sufficient means of increasing trade of quality seed in SADC? • If necessary but insufficient, what complementary activities should USAID REGO support to increase trade of quality seed in SADC? Annex 1: Overview of SOW and Evaluative Questions USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 134 • How does HSRS contribute to increasing trade of quality seed in SADC relative to interventions undertaken by other actors working towards the same goal? • Are STP tasks the necessary and sufficient means to achieve seed regulation harmonization in SADC? • Has STP advanced HSRS domestication in the region? • Is STP likely to achieve key life of project outcome and objective targets by the end of 2020 given the current trajectory? • Has STP been successful in improving the capacities of private seed companies, government officials, NSAs, NPPOs and other seed sector stakeholders to implement HSRS? • Have STP awareness creation events resulted in stakeholder actions that have advanced HSRS domestication in focus countries? • Is the SADC Seed Centre the best placed organization to advance domestication of HSRS in the region? • Has the technical support given to SADC institutions advanced the harmonization of seed regulation in SADC? • Has the indirect financial support given to SADC institutions advanced the harmonization of seed regulation in SADC? • What type of support – technical or financial – should STP prioritize to maximize impact with given resources? • Has STP’s acquisition of technologies for domestic regulators been effective in promoting HSRS domestication in focus countries? • Has STP been able to influence key stakeholders to prioritize and advance HSRS domestication through policy and regulation? • Have the policy environment assessments that STP has undertaken resulted in improved country domestication through implementation of HSRS-aligned policy? • Have the policies that STP has supported in reviewing resulted in improved country domestication of HSRS through implementation of HSRS-aligned policies? • Have the HSRS aligned policies that STP has supported in drafting resulted in improved country domestication of HSRS through implementation of HSRS-aligned policies? Management • Has the implementer exercised appropriate project management over STP? • Has the implementer adhered to USAID-REGO protocols relating to approval and reporting? • To what extent do participants view STP-organized events to be well-run? (Registration process, communication, event materials, venue) • To what extent does the contract between DAI and USAID limit its ability to choose potentially impactful tasks? • To what extent does USAID have processes which have adversely impacted the implementer’s ability to execute on project tasks? • Does USAID's contract with DAI effectively enable USAID to hold the implementer accountable, and to what extent does USAID have capacity to do so? • Does USAID/REGO’s and bilateral country implementation structure (COR and ACOR in Pretoria, with bilateral mission Activity Managers in Maputo and Lusaka) allow for effectively holding the implementer accountable and supporting the implementer's effective execution? • To what extent does the implementer's regional and focus country implementation structure (field office in Lusaka with country advisors in Lilongwe and Maputo) allow for effective execution of STP tasks? Annex 1: Overview of SOW and Evaluative Questions USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 135 • Will STP have sufficient financial resources to achieve project objectives during its remaining lifetime? • Does the implementer have processes in place to reallocate expenditure to highest impact task groupings in the face of budget shortages or surpluses, and how effectively do these processes operate? • Does the administration process involved in STP's Strategic Partnership Fund allow for effective use of the SPF towards advancing harmonization of seed regulations in SADC? • Has STP's Strategic Partnership Fund been used in ways that advanced harmonization of seed regulations in SADC? Sustainability • Will SADC Seed Centre, NSAs, government institutions and private seed companies be able to continue promoting and implementing HSRS domestication beyond the project’s lifetime? • Will STP’s achievements with respect to advancing HSRS policy alignment last beyond STP’s lifetime? Opportunities • Scale and focus of activities: Based on the interviews conducted with stakeholders, are there any opportunities that emerge for scaling an activity or adding a new one, including through new partnerships? • Strategic positioning & structure: Based on findings regarding potential synergies and overlaps across the three projects, are there opportunities to enhance the structure and strategic positioning of the activities? • Operational efficiencies: Are there any opportunities to shape organizational structure or processes to improve effectiveness and efficiency of operations? • Is there potential for collaboration with other projects/donors that STP is not currently utilizing? Best practices • Based on the findings and analysis of the themes above, what are the key features of STP that should be considered when implementing a new Trade and Food Security Project or other REGO project/activity? Specifically, where are the best practices in: o Project selection and design processes o Tasks or activities that reflect wider best practices that could be scaled across TFSP o Management processes and operating model (including clarity of roles and responsibilities, processes, governance, metrics and incentives) TPI: Evaluative questions Relevance • Are tralac's activities (capacity building, research, policy dialogue) necessary and sufficient to enhance the quality of trade policy in Southern Africa? • Without tralac, would these activities still take place? Annex 1: Overview of SOW and Evaluative Questions USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 136 • How are tralac's activities used by missions? • Capacity building – in-person training, online training: o How relevant were the topics covered during tralac's training for the participants? o If not relevant, what topics would have been more been more relevant? o How has the tralac training changed how participants do their work or make their decisions? o If tralac had not offered this training, how would participants have gained the knowledge and/or skills they acquired at the training? • Capacity building – awareness workshops and policy dialogues o How relevant were the topics covered during workshops and at the policy dialogues events for the participants? o If not relevant, what topics would have been more relevant? o How has the workshop or event changed how participants do their trade policy work or make their decisions? o If tralac had not organized these workshops/events, how would participants have engaged with the topics covered at the workshops/events? • Research and development of online content o How relevant is tralac's website content (research and policy dialogue content) for tralac's subscribers? o If not relevant, how can the content be made more relevant? What other topics should be covered? o How has tralac's research informed the trade and trade policy decisions of tralac's readership? o If tralac’s website content was not available (or before knowing about it), how would subscribers engage with the type of research and content that the tralac website offers? • Development and maintenance of AGOA.info o Does the AGOA.info website have relevant content for users of the website? o If not relevant, how can the content be made more relevant? What other topics should be covered? o How has the AGOA.info website informed the decisions of its users? o How would subscribers engage with the type of research and content that the tralac website offers? o If the AGOA.info website did not exist, how would users be informed about AGOA trade policies? o To what extent do users believe the AGOA.info website is user friendly? • Capacity building – education support (scholarships) o How relevant was the master’s program for the scholars’ career? o If not relevant, how could the program have been made more relevant? o How has the master’s program impacted scholars’ careers? • Capacity building – graduate-level interns o How has the tralac internship impacted the interns career path? o What lessons from the tralac internship does the intern use in their current career? Management • What is the assessment of the quality of tralac's training (organization, facilitator’s skills)? • What could have been done better to improve the training? • What is the assessment of the quality of the workshop/event (organization, facilitator’s skills)? • What could have been done better to improve the session(s)? Annex 1: Overview of SOW and Evaluative Questions USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 137 • What is the assessment of the quality of tralac's research products? • How could the website content be improved? • To what extent are tralac responsive to website enquires? • What could have been done better to improve the organization of the scholarship and the academic program? • What is the assessment of the quality of tralac's internship program (professionalism, mentoring, etc.) • How does tralac's management structure enable achievement of the TPI objectives? • How does tralac's operating model enable achievement of its TPI objectives? • Does tralac have adequate financial resources to achieve TPI objectives (especially after budget changes)? • Does the nature of the agreement ensure that TPI objectives are sufficiently met? (Cooperative agreement versus other types of agreements/ contracts) • Does USAID exercise sufficient oversight on tralac’s activities to ensure that tralac is working towards the TPI objectives? Sustainability • What measures are in place to ensure the sustainability of TPI achievements beyond USAID's involvement? • How does the political economy of SADC impact the sustainability of TPI achievements? • Have TPI activities been sufficiently inclusive? (e.g., balanced representation of beneficiaries by gender) Opportunities • Scale and focus of activities: Based on the interviews conducted with stakeholders, are there any opportunities that emerge for scaling an activity or adding a new one, including through new partnerships? o Where are there activities that are working effectively today that could be further scaled, and where are there activities that could / should cease in the future? o For TPI specifically – which activities (new or existing) should be scaled to ensure tralac fully utilize their budget, while best delivering against TPI objectives? • Strategic positioning & structure: Based on findings regarding potential synergies and overlaps across the three projects, are there opportunities to enhance the structure and strategic positioning of the activities • Operational efficiencies: Are there any opportunities to shape organizational structure or processes to improve effectiveness and efficiency of operations? • Where is there potential for collaboration with other projects / donors for TPI-related initiatives that USAID is not currently capturing? Best practices • Based on the findings and analysis of the preceding questions, what are the key strategic, programmatic, technical, and managerial features of the Project that should be taken into account when implementing a new Trade Policy Initiative in the region? • Are there any promising practices technical, programmatic, and managerial that should inform future REGO technical approaches, organizational structure, and programmatic design. Specifically, where are the best practices in: Annex 1: Overview of SOW and Evaluative Questions USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 138 o Project selection and design processes o Tasks or activities that reflect wider best practices that could be scaled across TFSP o Management processes and operating model (including clarity of roles and responsibilities, processes, governance, metrics and incentives) Annex 2: Detailed Objectives and Tasks by Activity USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 139 ANNEX 2: DETAILED OBJECTIVES AND TASKS BY ACTIVITY 10.1 SOUTHERN AFRICA TRADE AND INVESTMENT HUB (SATIH) As outlined in the main report, SATIH’s programming is organized against four components, indicated in Figure 30 below. Figure 30: SATIH theory of change (TOC) The following section of the report will unpack key tasks within each component, including their relative level of focus. This level of focus has been determined based on interviews with USAID/REGO and the implementer, triangulated with annual and quarterly report data where possible. Activities where work volumes are more extensive are in bold typeface in the tables below. Component 1: Agribusiness Trade Project Objective Intermediate Results IR1: Intra-regional trade in staple foods, agricultural inputs and primary agribusiness products increased USAID Southern Africa RDCS GOAL: MORE INTEGRATED REGION WITH IMPROVED QUALITY OF LIFE FOR SOUTHERN AFRICANS Activity Goal Component 2: Export Competitiveness • Company-level technical assistance • Trade shows & B2B linkages • Marketing and promotion of Southern African products • Improve knowledge of market conditions • Association and export agency capacity building • Build capacity to increase AGOA utilization Component 3: Finance and Investment • Warehouse receipt system strengthening • Financial sector strengthening • Increasing investment into the region • Development of funding platforms & vehicles • Investment enabling environment Component 4: Trade Facilitation and Enabling Environment • Implement trade facilitation agreements • Address non-tariff barriers • Facilitate & increase trade • Support logistics sector improvements • Empower professional associations’ advocacy in support of trade policy reforms Gender Integration • Capacity building for advocacy by women’s business associations • Link and strengthen women’ business associations in the region • Increase lending to women-owned businesses • Support women-owned businesses through BDS support • Promote gender-sensitive job creation in agricultural enterprises IR 2: Global and intra-regional trade in higher value products increased IR 4: Trade and transportation facilitation measures implemented Intermediate Objective 2: An improved regional trade and investment enabling environment Intermediate Objective 1: Increased competitiveness of target value chains USAID Southern Africa RDCS Development Objective I: Increased Sustainable Economic Growth IR 3: Regional and multi-lateral trade & investment agreements & support institutions strengthened Southern Africa Trade and Investment Hub Project Objective: Expand Regional & Global Trade Investment Intermediate Objectives Components Component 1: Agribusiness Trade • Improve trade in inputs and services • Facilitate use of new agricultural technologies • Support trade in key non￾staple commodities • Strengthen trade associations & cooperatives Annex 2: Detailed Objectives and Tasks by Activity USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 140 The Agribusiness Trade component’s primary goal is increasing intra-regional trade in staple foods, agricultural inputs, and primary agribusiness products (IR 1). In working toward this goal, the component undertakes activities across four areas (known as “strategies”) outlined in Figure 31 below. Figure 31: Agribusiness Trade programming Source: ISP team synthesis of project documents and key personnel interviews Most of the effort under Component 1 occurred under two of the strategies: Strategy 1 worked to support structured trade, with a focus on enabling key market actors (e.g., warehouses, information systems, financial institutions) to grow formal trade in the region. Strategy 1 worked in close partnership with the Finance and Investment component (Component 3 of SATIH) on structured trade, finance, and investment. The report elaborates on this partnership in the section below on the Finance and Investment component. Strategy 4 worked to support trade in non-staple commodities. This work consisted of three distinct sets of tasks, each focusing on strengthening different value chains in different markets. Each of these tasks had the overarching goal of enabling regional or global exports. Key progress under each of these is highlighted below: • Support market development: SATIH developed market assessments of the Zambian and Botswanan beef sectors. In Botswana, SATIH then engaged with the local stakeholders (e.g., Botswana Investment and Trade Centre [BITC] and Botswana Meat Commission [BMC]) on opportunities to leverage AGOA to gain access to the U.S. beef market. 90 • Support for fresh produce trade: SATIH focused primarily on promoting investment in a “fresh produce market floor” in Zambia. Key to this was an event held at a Produce Market Association (PMA) conference in late 2018, where SATIH presented international investors with the business case and opportunities for investment. • Support marketing activities for mohair and wool: SATIH supported BKB Ltd., a leading South African mohair and wool buyer. SATIH’s support primarily focused on partnership 90 Scope and depth of this engagement to be further validated through the evaluation. Annex 2: Detailed Objectives and Tasks by Activity USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 141 development between BKB Ltd. and development finance institutions and mobile network operators to develop a digital payments system linking BKB Ltd. to Lesotho mohair and wool farmers. Component 2: Export Competitiveness The Export Competitiveness component’s primary goal is to increase global and intra-regional trade in higher value products (IR 2). In working toward this goal, the component has undertaken tasks across two areas outlined in Figure 32 below. Figure 32: Export Competitiveness programming Source: ISP team synthesis of project documents and key personnel interviews Under Strategy 1, SATIH identifies firms in specialty food and textile value chains that are at or near a state of export readiness and provides them with targeted support to make the firm’s products attractive to foreign buyers. Examples of the support provided to firms include providing training on Food and Drug Administration (FDA) regulations and helping companies meet the requirements set out in those regulations. Strategy 1 also covers work through several avenues to link export-ready businesses to international markets. One avenue is to support export-ready firms’ participation in international trade shows. For example, in 2018, SATIH took 16 Southern African specialty food91 companies to an event in New York. A second avenue is supporting locally held trade shows and buyer missions. Large regional and international buyers attend these shows. For example, SATIH partnered with Messe Frankfurt to bring a buyer to Source Africa, a premier trade show in the region. Under Strategy 2, SATIH focuses its programming through two avenues. The first is working with governments (ministries of trade) in the region to develop and implement national AGOA strategies—in 2018, for example, SATIH engaged with South Africa, Mozambique, Lesotho, eSwatini, and Zambia). The second is hosting workshops with businesses, private sector associations, and government to communicate the opportunities that AGOA offers to businesses seeking to export to the USA. 91 Within specialty foods, SATIH specifically focuses on dried fruit and nuts. Annex 2: Detailed Objectives and Tasks by Activity USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 142 Component 3: Finance and Investment The Finance and Investment component focuses on three intermediate results (IR): increasing intra￾regional trade in stable foods (IR 1), increasing global and intra-regional trade in higher value products (IR 2), and strengthening trade agreements and support institutions (IR 3). In working toward these goals, the component undertakes tasks across three areas outlined in Figure 33 below. Figure 33: Finance and Investment programming Source: ISP team synthesis of project documents and key personnel interviews SATIH’s has focused its efforts on two very distinct sets of programming. In the first set of tasks (linked mostly to Strategy 1), the Finance and Investment and Agribusiness components worked together closely to bring a systemic approach to trade and investment in the region (particularly Zambia and Malawi). The underlying premise was that addressing challenges to transparent, efficient domestic agricultural markets in target value chains would, in turn, enable more effective trading. Programming in Strategy 1 focuses on four broad, interlinked dimensions: • High-quality post-harvest storage – building on the efforts of the previous Hub, SATIH worked with local financial institutions and the European Investment Bank to invest EU35 million in warehouse infrastructure in Malawi. • Access to finance – SATIH provides access to finance in two ways: i) it provides direct technical assistance (TA) to financial institutions and links them to USAID’s Development Credit Authority (DCA) facility (e.g., Barclays Botswana) and ii) it works with warehouses (both in Malawi and Zambia), local financial service providers, and commodity exchanges to develop warehouse receipt financing for smallholder farmers. Annex 2: Detailed Objectives and Tasks by Activity USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 143 • Support commodity exchanges – SATIH works to support commodity exchanges in Zambia and Malawi. Examples of this work include facilitating partnerships with the Johannesburg Stock Exchange and running workshops educating stakeholders on working with the stock exchange. 92 • Advance regional market information systems: SATIH works primarily with Zambian stakeholders to establish an effective market information system; 93 its focus is building the system’s capacity and linking it to SADC’s regional equivalent. The second set of activities (under Strategy 3) saw SATIH contract CrossBoundary, an investment advisory firm, to work directly with private equity funds and high-potential investees to initiate, support and close investments. This aims to enable high potential Southern African businesses to access finance. Activities are spread across three broad spheres: • Pipeline sourcing: The Finance and Investment team and CrossBoundary leverage networks and networking events to develop a pipeline of potential deals. Investors (particularly private equity [PE] firms) and investees show interest by signing a letter of intent., CrossBoundary then assesses deals based on factors that include target sector, size, and potential outcomes. • Transaction and strategic support: Following deal sourcing, CrossBoundary offers bespoke services to the investor to facilitate the transaction. Tools in this category include opportunity validation, market intelligence, fundraising support, due diligence, financial analysis, structuring, neutral facilitation, and investment implementation planning. • Transaction close: CrossBoundary provides ongoing communication and strategic support through the transaction process to help push deals to close. This work has facilitated $113 million of transactions to date, against a target of $40 million. Component 4: Trade Facilitation and Enabling Environment The Trade Facilitation and Enabling Environment component focuses on two intermediate results: strengthening trade agreements and support institutions (IR 3) and facilitating trade and transportation measures (IR 4). In working toward these goals, the component undertakes activities across four areas outlined in Figure 34 below. 92 Commodity exchanges are central actors in any type of cross-border trade (through their rule setting and enforcement, as well as their standards and procedures for trading) and play a key role enabling warehouse receipt finance. 93 An effective regional market information system would enable trade by allowing countries to quickly identify countries with surpluses and deficits of a crop. For this to occur efficiently, countries need market information systems that can accurately collect, store, and manage crop information. This information is then linked to a regional system (to enable a shared understanding of who has how much of which crop). Annex 2: Detailed Objectives and Tasks by Activity USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 144 Figure 34: Trade Facilitation and Enabling Environment programming Source: ISP team synthesis of project documents and key personnel interviews The TOC prioritized trade facilitation as a core pillar of SATIH programming. However, a combination of factors (outlined at the end of this section) have resulted in little programming taking place under Component 4 relative to initial aspirations. The programming that did take place occurred under Strategies 1, 4, and, to a lesser extent, 3. An organizing theme across almost all these activities is reducing the time and cost associated with crossing borders in the region. Strategy 1 focuses on improving the application of sanitary and phytosanitary measures 94 at borders, with the aim of speeding up processing times. Within this strategy, SATIH undertook a wide array of demand￾driven programming (SATIH liaised with SADC and countries to identify activities). Most programming involved facilitation of training workshops with government officials, policy development, or institutional capacity building. Strategy 3 focuses on developing the Truck Monitoring System, a tool that tracks the movement of trucks across borders in Southern and East Africa. The tool is intended to serve as a mechanism for tracking progress, communicating challenges, and thereby prompting the development of better border policy. Strategy 4 focuses on supporting SADC in implementing the Trade Facilitation Program. Examples of activities undertaken include developing a situational analysis report on the state of coordinated border management for the SADC secretariat and providing technical support to the Ministry of Trade in Botswana. Also under Strategy 4, SATIH supported the Malawi Revenue Authority’s (MRA’s) efforts to develop an Authorized Economic Operator program, which aims to simplify customs procedures by 94 Measures to protect humans, animals, and plants from diseases, pests, or contaminants. Annex 2: Detailed Objectives and Tasks by Activity USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 145 accrediting supply chain actors who meet specified criteria, allowing them to move goods across borders with fewer restrictions. Gender Gender is positioned in the TOC as an element implemented across components. Although there is no evidence of a holistic gender strategy, there are elements of gender-specific programming. For example, Export Competitiveness work included the provision of ad hoc support to the African Women’s Entrepreneurship Program (AWEP) in Botswana, and in consultation with AWEP, the incorporation of value chains with a disproportionate representation of women in Botswana’s national AGOA strategy. Similarly, the Finance and Investment transaction support included a gender score in its “fit check”, which contributed to investments that had a specific gender lens. Annex 2: Detailed Objectives and Tasks by Activity USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 146 10.2 SEED TRADE PROJECT (STP) This report will now provide a more detailed breakdown of the two components of STP’s work, in line with the Theory of Change – capacity building and policy advocacy. It will also summarize the use of the Strategic Partnership Fund grants facility. Capacity Building STP capacity-building efforts seek to improve HSRS assimilation by strengthening the institutional capacity to implement HSRS within SADC institutions and decision-making bodies in focus countries. Capacity￾building efforts take three forms, illustrated in Figure 35 and outlined in further detail below: • Financial and technical support to the SADC institutions • Technical trainings and awareness creation events for seed sector stakeholders • Acquisitions of new physical and digital infrastructure for seed sector stakeholders in focus countries Most of these tasks do not map directly to any one activity objective—rather, each contributes to each of the activity objectives in different ways. The exception to this is acquisitions of new physical and digital infrastructure, which contributes solely to AO 2. Figure 35: Breakdown of STP capacity-building efforts to improve HSRS domestication Source: ISP team synthesis of project documents and key personnel interviews Financial and technical support to SADC Institutions Support to SADC Institutions • Technical assistance e.g. guidance on the development of a SADC Seed Centre charter, and SADC Seed Centre business model • Indirect financial support e.g. co￾sharing meeting costs of SADC Seed Committee and SADC Seed Technical Meetings • Training events for seed sector regulators and private companies in focus countries to implement HSRS • Awareness creation workshops and conferences to drive HSRS domestication • Acquisitions of laboratory and other equipment to facilitate implementation of HSRS requirements by focus country regulators • Acquisitions of digital information systems to support seed certification and regulation in focus countries AO1: Increased Availability of Improved Seeds in the Region Training and awareness events Acquisitions of new physical and digital infrastructure for seed sector stakeholders AO2: Increased Availability of Technologies Management Practices and Innovation AO3: Improved Regional Policies to Support Agriculture Growth AO2: Increased Availability of Technologies Management Practices and Innovation Annex 2: Detailed Objectives and Tasks by Activity USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 147 The SADC Food, Agriculture, and National Resources (FANR) is the directorate within the SADC secretariat charged with overseeing HSRS assimilation across the region. FANR performs this duty through the SADC Seed Centre, a semi-independent institution located within, and accountable to, FANR. The SADC Seed Centre is governed by the SADC Seed Committee, a multi-sectoral body that consists of SADC FANR along with representatives from national seed [authorities/agencies?] (NSAs), national plant protection organizations (NPPOs), and private seed companies from across the region. To increase the pace of HSRS assimilation in the region, STP has been providing indirect financial and technical support to these institutions, with a focus on the operationalization and sustainability of the SADC Seed Centre. In practice, indirect financial support has included covering portions of meeting costs for the SADC Seed Committee Meetings and SADC Seed Technical meetings hosted by SADC FANR. Technical support to these institutions has included: • Drafting and validating the SADC Seed Centre Charter to formally establish the institution (the role of which is outlined in Figure 36) • Helping the SADC Seed Centre to re-operationalize the SADC Regional Varietal Release System (specifically by following up and processing backlogged varietal release applications) Figure 36: Relationship between the relevant SADC Seed Centre institutions Source: Seed Trade Project, Written Response to ISP Information Request, 2019; Seed Trade Project, 2016 Annual Report, 2017; SADC Seed Centre Website, 2019 Technical trainings and workshops for seed sector stakeholders on HSRS technical issues To ensure that both the public and private sectors in focus countries have the awareness and means to implement technical aspects of HSRS, STP facilitates training and awareness creation events. These have Annex 2: Detailed Objectives and Tasks by Activity USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 148 particularly focused on “Certification and Quality Assurance” and “Quarantine and Phytosanitary Measures”. 95 These events take the form of seminars, workshops, and exchanges with model institutions in South Africa and elsewhere. While STP staff facilitate some events, most are facilitated by individuals from regional NSAs, NPPOs, SADC FANR, or independent consultancies. Figure 37 summarizes the reach of these events. Figure 37: Training and awareness creation events hosted by STP, 2016 – 2018 Number of events Number of participants Source: Seed Trade Project, 2018 Annual Performance Report, 2019; Seed Trade Project, Response to ISP Information Request, 2019 Acquisitions of new physical and digital infrastructure for seed sector stakeholders in focus countries with HSRS implementation STP supports NSAs and other seed sector institutions in focus countries with the acquisition of technologies and digital systems that improve their ability to implement HSRS. Through 2018, STP had only supported the Zambian NSA the Seed Control and Certification Institute (SCCI), although it has plans to provide more institutions with this support in 2019. In 2018, STP, through the SPF, gave the SCCI an in-kind grant of a refurbished cold room that is intended to help SCCI improve its seed testing standards. In 2017, STP financed SCCI’s implementation of an online seed certification system across all provinces in Zambia to expedite the seed certification and testing process in the country, and to support its obtainment of OECD accreditation. Policy Advocacy STP’s policy advocacy work entails providing financial or technical support on the review or creation of HSRS-aligned policies. This support has been provided to NSAs, NPPOs, SADC FANR, and government departments. STP policy advocacy efforts entail conducting or financing the following (outlined in Figure 38): • Assessment of seed policy and regulatory environments’ alignment with HSRS • Updating of existing seed sector policies and regulation to align with HSRS • Drafting of new seed sector policies or regulations to fill identified gaps 95 Seed Variety Release, Seed Certification and Quality Assurance, and Quarantine and Phytosanitary Measures for seed are the three core elements of HSRS. While Seed Variety Release is largely overseen by the SADC Seed Centre with support from NSAs, the other two are the key technical areas on which policy and implementation alignment is needed across the region. Annex 2: Detailed Objectives and Tasks by Activity USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 149 These activities focus on AO 3: Improved regional policies to support agriculture growth. Figure 38: STP policy advocacy efforts to improve the quality of HSRS-aligned policies Source: ISP team synthesis of project documents and key personnel interviews Table 9 outlines the various policies and regulations assessed, updated, or drafted. Table 12. STP policy advocacy outputs, 2016 – 2018 Country Policy or Regulation Supported Year Supported Malawi 1 Malawi Seed Act review 2018 2 Malawi Seed Policy 2018 3 Review of Malawi Pest List 2018 Mozambique 4 Assessment of Mozambique’s Seed Sector Legal and Regulatory Framework 2018 5 Mozambique Pest List review 2018 Zambia 6 Review and approval of new seed regulations 2017 7 Drafting the Zambia pest risk Aaalysis and phytosanitary statutory instrument 2018 8 Drafting the pest surveillance statutory instrument 2018 SADC 9 SADC Seed Centre Charter 2017 10 Drafting Regional Seed Certification and Quality Assurance Guidelines 2018 11 Guidelines on Pesticide Management and Risk Reduction 2018 Source: Seed Trade Project, 2018; Annual Performance Report, 2019 Assessments of seed regulatory environments • National seed sector law and regulation assessments for governments and NSAs to determine alignment with HSRS • Sponsoring or conducting revisions of national and regional policies e.g. National Pest Lists and national seed regulations/policies • Sponsoring or conducting the drafting of new national and regional policies e.g. SADC Seed Certification and Quality Assurance guidelines AO3: Improved Regional Policies to Support Agriculture Growth Updating existing policies to align with HSRS Drafting new seed sector policies where gaps have been identified Annex 2: Detailed Objectives and Tasks by Activity USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 150 Strategic Partnership Fund In addition to capacity building and policy advocacy, the contract between DAI and USAID/SA has a GUC facility of up to $1.5 million. The implementer uses this facility to issue grants to seed trade associations, national seed authorities, private sector companies, and others to complement capacity-building work. The grants are issued so that grantees either support STP with carrying out its tasks or else directly benefit from the grants themselves—for example, by funding investments in new technologies or capacities. STP deployed this facility, marketed as the Strategic Partnership Fund (SPF), five times between 2016 and 2018 (see Table 10). Table 13: Strategic Partnership Fund grantees, 2016 – 2018 Grantee Name Country Grant Value (USD) Grant Purpose Seed Trade Association of Malawi (STAM) Malawi 91,921 Conduct capacity building around the SADC HSRS and champion the assimilation of the SADC HSRS Association for the Promotion of the Seed Sector in Mozambique (APROSE) Mozambique 100,000 Conduct capacity building around the SADC HSRS and champion the assimilation of the SADC HSRS Zambia Seed Trade Association (ZASTA) Zambia 93,150 Conduct capacity building around the SADC HSRS and champion the assimilation of the SADC HSRS Seed Co Zambia Zambia 99,999 Pilot the SADC Seed Label managed by the SADC Seed Centre. Produce seed under the SADC HSRS in Zambia and export to the DRC SCCI Zambia 30,000 Refurbishment of the Seed Sample Store Room Source: Seed Trade Project, 2018 Annual Performance Report, 2019 Annex 2: Detailed Objectives and Tasks by Activity USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 151 10.3 TRADE POLICY INITATIVE (TPI) As outlined in the main report, the work of the Trade Policy Initiative is organized under three categories, which are outlined in further detail in this annex. Figure 39: Categories of work for the Trade Policy Initiative Applied policy research The aim of applied policy research is to reach key decision makers in the sub-regions of Southern and East Africa who may be able to influence important trade policy decisions in their countries. Research topics are informed by the current trade policy issues on the regional trade agenda for the two sub￾regions. Key activities include conducting applied trade policy research and producing research publications, i.e., working papers and trade briefs. Over the evaluation period, tralac’s TPI-funded publications and trade briefs have covered the following topics, among others: • AGOA developments affecting selected SADC member states • The Tripartite Trade Area • The impact of commodity prices on Chinese imports • The blue economy Capacity Building Capacity building has included the design and delivery of training programs, use of internships, short courses (online and classroom-based), and a master’s program96 to close knowledge gaps. Over the evaluation period, TPI capacity-building activities have included:  Hosting AGOA workshops and forums  Sponsoring SADC residents with scholarships for a master’s degree in trade policy  Training the Botswana Trade Commission to manage trade remedies  Training Common Market for Eastern and Southern Africa (COMESA) judges  Training Zambia government officials on how to prepare for the African Continental Free Trade Area (AfCTA) negotiations 96 Note the master’s degree program was discontinued during the life of the program—2013 was the final intake year. Research Capacity building Policy dialogue • Publication of working papers: in￾depth study of a topic, and is generally a technical paper, focusing on an applied trade policy topic • Publication of trade briefs: shorter trade policy paper; written in a non￾technical manner • Tailored trainings on selected trade topics • Online training modules on trade • Awareness workshops on trade topics • Education support (scholarships) • Graduate-level internships • Annual tralac conference co￾financed by USAID • Development of content for online presence (website, LinkedIn, Facebook, Twitter) • Development and maintenance of the AGOA.info website • Convening and participation of roundtables on current trade policy issues Annex 2: Detailed Objectives and Tasks by Activity USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 152  Training various actors about the Southern African Customs Union (SACU)  Training various actors about U.S. tariffs towards African countries  Launching a trade policy e-course Policy dialogues TPI also holds policy dialogues through workshops, seminars, roundtables, and an annual conference that has become an important platform for discussing the region’s trade agenda. TPI also uses the tralac website; social media channels such as Facebook, Twitter, and LinkedIn; and electronic newsletters to extend trade policy debates. In 2013, TPI funded the redesign of the AGOA.info website, which seeks to offer continuously updated information and tools for any business or country wanting to trade through the AGOA agreement. Through this website, tralac also responds to individual inquiries. Annex 3: Evaluation Team USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 153 ANNEX 3: EVALUATION TEAM Oversight from the ISP team was provided by Carlijn Nouwen, CJ Fonzi and Niall Saville as Project Co￾Directors. The full-time evaluation team consisted of Carl Hickson as the evaluation team leader, in addition to Nour Sharara, Scott Hosking, Sakhe Mkosi, and Palesa Thinta. Biographies for the team are provided below: Carlijn Nouwen – Project Co-Director Carlijn is a Partner in Dalberg's Johannesburg office and leads Dalberg’s global Inclusive Business expertise area. Carlijn brings 14 years of experience in strategy consulting plus an additional 2,5 years as a foreign service officer/ policy officer with the Dutch Ministry of Foreign Affairs. Carlijn’s Dalberg experience includes co-leading Dalberg work on analyzing trade flows between Africa and the US, finding commercial investment opportunities for orange flesh sweet potatoes in Kenya, South Africa and Malawi, and assessing agricultural value chains for commercial investment in Rwanda. She also has four years’ experience leading engagements for the Africa Private Capital Group in South Africa, focused on increasing development￾oriented private sector investment. Prior to joining Dalberg, Carlijn worked for McKinsey and Company for 7 years in its Amsterdam office, with a focus on healthcare, financial services and leadership development in transformational change. She holds a Master of Science degree in Industrial Engineering and Management Science (cum laude) from the Eindhoven University of Technology. CJ Fonzi – Project Co-Director CJ Fonzi is an Associate Partner, Director of Dalberg’s Kigali Office and co-leader of Dalberg’s Evaluation service line. His 16 years of development sector experience span a broad range of issues including impact investing, agriculture, and SME finance and support. CJ has recently supported the International Finance Corporation (IFC) to conduct the mid-term impact review of their $700 million Global SME Finance Facility, and conducted a mid-term evaluation of the Qatar Fund for Development’s $100 million Tunisia friendship fund. CJ was previously part of the founding team at the Global Impact Investing Network (GIIN), where he served as Manager of the Impact Reporting and Investment Standards (IRIS). Earlier in his career, CJ served as the Manager of Monitoring and Evaluation for the Clinton Foundation’s Health Access Initiative (CHAI) in India. CJ holds an MBA from Cornell University with a concentration in Sustainable Global Enterprise and a BS with high honors from Rensselaer Polytechnic Institute in Information Technology. Niall Saville – Project Co-Director Niall is an Associate Partner in Dalberg’s Dar es Salaam office. The head of Dalberg’s Inclusive Growth practice area, he has deep experience in investment and trade promotion and policy, with over 10 years’ experience advising national and regional governments. His recent engagements for Dalberg have included supporting USAID to develop the Feed the Future trade agenda and working with the government investment promotion body of a leading African country to develop and socialize actionable recommendations to unlock growth in textiles and apparel. Annex 3: Evaluation Team USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 154 Prior to Dalberg, Niall worked for Monitor Group in Dubai, Vietnam and South Africa, where his engagements included spending 18 months managing a team in one of the poorest provinces of Vietnam to create the provincial government’s “Socio-Economic Development Plan”. Niall holds an MBA with Distinction from INSEAD, and MA (Mathematics and Engineering) and MEng degrees from the University of Cambridge. Carl Hickson – Evaluation Lead Carl is part of Dalberg’s Johannesburg office and brings experience advising both development sector clients and multinational corporations on monitoring and evaluation, organization effectiveness, and strategy development. Most recently he led a private sector development and market landscaping project for USAID in Uganda, and prior to that has led significant organizational reviews (including for an African consumer goods organization, a transmission and distribution company, major INGOs, in addition to a range of other clients within the consumer goods, retail, energy, and telecoms sectors). Before joining Dalberg, Carl worked with Accenture Strategy based out of London, where he advised a range of clients across Europe, South Africa and the U.S. on organization effectiveness and operating model re-design. Carl holds a First-Class Master of Arts with Honours in International Relations from the University of St Andrews in Scotland, where he featured on The Dean’s List. Nour Sharara – Senior Evaluation Specialist Nour is part of Dalberg Advisors’ Dakar office. Her Dalberg experience spans strategy engagements, impact evaluations and implementation support across Africa. Her most recent impact evaluation prior to this engagement consists of assessing the overall impact of the Qatar Fund for Development’s $100 million Tunisia friendship fund set up to support the development and growth of SMEs in Tunisia. At Dalberg, Nour has worked with bi-lateral missions, philanthropies, multi-lateral institutions and governments. Nour holds a Master of Public Health from Harvard University, and a Master of Arts in Medical Ethics and Law from King's College London. Scott Hosking – Evaluation Specialist, SATIH Scott is part of Dalberg Advisors’ Johannesburg office. Scott’s experience with Dalberg Advisors includes supporting UNCDF to develop a detailed Theory of Change as part of a larger strategy engagement, a final evaluation of a Varkey Foundation teacher training program in Ghana, and a high-level evaluation of Lesotho’s healthcare system ahead of the client’s second compact with the country. Scott holds a masters’ degree in applied economics from the University of Cape Town. Sakhe Mkosi – Evaluation Specialist, STP Sakhe is part of Dalberg Advisors’ Johannesburg office. His previous Dalberg experience has seen him work with the Bill and Melinda Gates Foundation and the Higherlife Foundation to research the landscape of public health emergency response in Zimbabwe. Prior to Dalberg, Sakhe worked for the Pathways for Prosperity Commission at the University of Oxford, the Rhodes Trust’s Development Office and Deloitte South Africa’s external audit division. Sakhe holds MPhil degrees in Development Studies and Public Policy and Administration from the Universities of Oxford and Cape Town respectively. Annex 3: Evaluation Team USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 155 Palesa Thinta – Evaluation Specialist, TPI Palesa is part of Dalberg Advisors’ Johannesburg office. At Dalberg, she has conducted in-depth market research on small-scale productive use leveraging solar energy (PULSE) market opportunities in agriculture in Côte d’Ivoire. She brings professional experience in qualitative and quantitative research having worked for Statistics South Africa, Business Science Corporation, a data science advisory firm, the Abdul Latif Jameel Poverty Action Lab (J-PAL), and Organization for Economic Co-operation and Development. Palesa holds an MA in International Economic Policy from Sciences Po Paris and an undergraduate degree in statistics and economics at École Nationale Supérieure de Statistique et d'Économie Appliquée in Abidjan. Annex 4: Reports and Data Provided to the Evaluation Team USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 156 ANNEX 4: REPORTS AND DATA PROVIDED TO THE EVALUATION TEAM In addition to primary data gathered through interviews and surveys, and publicly available secondary literature, the following documents were made available to the evaluation team to support this work. Trade and Food Security Project (Project-level documents only) TFSP documents assessed Trade and Food Security Project Appraisal Document Southern African Trade and Investment Hub SATIH documents assessed SATIH Turnaround Plan, 2018 SATIH Short Term Technical Advisor List SATIH Section C - Performance Work Statement SATIH Performance Monitoring and Evaluation Plan SATIH FY2019 Work Plan SATIH FY2019 Activity Pool Summary SATIH FY2018 Work Plan SATIH FY2018 Q1 Quarterly Report SATIH FY2018 Q2 Quarterly Report SATIH FY2018 Q3 Quarterly Report SATIH FY2018 Q4 Quarterly Report SATIH FY2018 Annual Report SATIH FY2017 Work Plan SATIH FY2017 Q1 Quarterly Report SATIH FY2017 Q2 Quarterly Report SATIH FY2017 Q3 Quarterly Report SATIH FY2017 Q4 Quarterly Report SATIH FY2017 Annual Report SATIH FY 2017 And 2018 Technical Costs SATIH Fact Sheets SATIH Bureau for Food Security Presentation, 2019 Pragma Data Quality Assessment for Harmonised ACTE PMP Indicator Data, 2018 Genesis SATIH Baseline Study, 2017 CrossBoundary Post-Engagement Surveys Annex 4: Reports and Data Provided to the Evaluation Team USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 157 Other documents assessed Agricultural Commodity Exchange for Africa (ACE) Warehouse Receipt Data 2013 – 2018 AGOA National Strategy Botswana AGOA National Strategy Malawi AGOA National Strategy Zambia Agribusiness Trade ZAMACE training material Draft High-Level Strategy for Upgrading the Textile and Clothing Industry in Botswana, 2013 East African Trade Hub Final Evaluation 2010-2014 Final Evaluation Report IAPRI Feasibility for Investment: Enhancing the Zambian Warehouse Receipt System and Aligning the Food Reserve Agency Strategic Plans to the System, 2019 Southern African Trade Hub Developing the Textiles and Apparel Industry in Botswana, 2013 Southern African Trade Hub Final Evaluation 2016 USAID West Africa Regional Development Cooperation Strategy 2015-2019 ZAGIS Forum Resolutions, 2017 Seed Trade Project STP documents assessed STP FY2016 Annual Report STP FY2017 Annual Report STP FY2018 Annual Report STP FY2016 Annual Workplan STP FY2017 Annual Workplan STP FY2018 Annual Workplan STP FY2019 Annual Workplan Presentation STP Q1 FY2016 Quarterly Report STP Q2 FY2016 Quarterly Report STP Q3 FY2016 Quarterly Report STP Q1 FY2017 Quarterly Report STP Q2 FY2017 Quarterly Report STP Q3 FY2017 Quarterly Report STP Q1 FY2018 Quarterly Report STP Q2 FY2018 Quarterly Report STP Q3 FY2018 Quarterly Report STP PMEP STP Performance Indicator Reference Sheets Custom Indicators STP Regional Indicator Verification Spreadsheet STP REGO Regional 2017 PPR Seed Trade Project Assessment 2017 Southern Africa Regional Seed Sector Assessment 2016 STP Project Fact Sheet 2018 STP Malawi Information Sheet STP Mozambique Information Sheet Annex 4: Reports and Data Provided to the Evaluation Team USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 158 STP Zambia Information Sheet STP Zimbabwe Information Sheet Trade Policy Initiative TPI documents assessed TPI FY2017 Work Plan TPI FY2017 Q1 Quarterly Report TPI FY2017 Q2 Quarterly Report TPI FY2017 Q3 Quarterly Report TPI FY2017 Q4 Quarterly Report TPI FY2018 Q1 Quarterly Report TPI FY2018 Q2 Quarterly Report TPI FY2018 Q3 Quarterly Report TPI FY2018 Q4 Quarterly Report TPI 2017 Liquidation Reports TPI 2018 Liquidation Reports TPI 2017 Monitoring and Evaluation Matrix Other documents assessed Tralac Evaluation by Swedish International Development Cooperation Agency, 2017 Additional Annex Materials USAID Investment Support Program Trade and Food Security Project – Mid-Term Performance Evaluation 159 ADDITIONAL ANNEX MATERIALS The USAID Southern Africa MEL team holds the following documentation requested in the Statement of Work, separate to this main document: ● Original Evaluation SOW ● Full inception report – documenting evaluation methods, survey tool designs, and interview prompts ● Final versions of the survey tools distributed to participants ● Underlying interview codification and survey response datasets ● Signed disclosure of conflict of interest forms for all evaluation team members