PERFORMANCE EVALUATION: COMMON MARKET FOR EASTERN AND SOUTHERN AFRICA FINAL REPORT SEPTEMBER 2018 This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International (MSI), a Tetra Tech Company. PERFORMANCE EVALUATION: COMMON MARKET FOR EASTERN AND SOUTHERN AFRICA (COMESA) FINAL REPORT September 14, 2018 IDIQ No. AID-623-I-12-00001 Award No: AID-623-TO-17-00003 Prepared by Management Systems International (MSI), a Tetra Tech Company 200 12th St South, Suite 1200 Arlington, VA, USA 22202 DISCLAIMER The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ii | COMESA PERFORMANCE EVALUATION USAID.GOV CONTENTS ACRONYMS..................................................................................................................... III EXECUTIVE SUMMARY.................................................................................................. 1 FINDINGS.................................................................................................................................................................. 1 RECOMMENDATIONS.......................................................................................................................................... 4 EVALUATION PURPOSE AND EVALUATION QUESTIONS.................................. 6 EVALUATION QUESTIONS................................................................................................................................. 6 PROJECT BACKGROUND .............................................................................................. 7 EVALUATION DESIGN, METHODS AND LIMITATIONS........................................ 7 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS ...................................... 8 EVALUATION QUESTION 1............................................................................................................................... 8 EVALUATION QUESTION 2.............................................................................................................................22 EVALUATION QUESTION 3.............................................................................................................................26 EVALUATION QUESTION 4.............................................................................................................................27 EVALUATION QUESTION 5.............................................................................................................................32 RECOMMENDATIONS.................................................................................................. 33 ANNEX 1: BIBLIOGRAPHY.......................................................................................... 36 USAID.GOV COMESA PERFORMANCE EVALUATION | iii ACRONYMS ACTESA Alliance for Commodity Trade in Eastern and Southern Africa AFAP African Fertilizer and Agribusiness Partnership AfDB African Development Bank AGRA Alliance for a Green Revolution in Africa ASYCUDA Automated System for Customs Data CBC COMESA Business Council CIPE Center for International Private Enterprise COMESA Common Market for Eastern and Southern Africa COMSHIP COMESA Seed Harmonization Implementation Plan COMSIS COMESA Seed Information System CSPro Census and Survey Processing System DFID Department for International Development (U.K.) DP Development Partner DRC Democratic Republic of Congo ECDPM European Center for Development Policy Management EQ Evaluation Question EU European Union FAO Food and Agriculture Organization (United Nations) FY Fiscal Year GM Genetically Modified HACCP Hazard Analysis and Critical Control Points IFC International Finance Corporation IL Implementation Letter ILEAP International Lawyers and Economists Against Poverty IMF International Monetary Fund IPAA Integrated Partnership Assistance Agreement ISSD Integrated Seed Sector Development ITC International Trade Center KII Key Informant Interview KSP Kenya Support Project (USAID) LEAD Leadership Training and Capacity Building Program (USAID) iv | COMESA PERFORMANCE EVALUATION USAID.GOV MLN Maize Lethal Necrosis M&E Monitoring and Evaluation MSI Management Systems International NGO Non-governmental Organization RDOAG Regional Development Objective Assistance Grant REC Regional Economic Community REI Regional Economic Integration Office (USAID/KEA) RISP Regional Integration Support Program (EU) RMIC Resource Mobilization and International Cooperation SDMX Statistical Data and Metadata eXchange SME Small and Medium Enterprise SPS Sanitary and Phytosanitary Measures SQL Structured Query Language SSTP Scaling Seeds and Technologies Partnership in Africa STDF Standards and Trade Development Facility (WTO) STR Simplified Trade Regime TID Trade Information Desk TMEA TradeMark East Africa UK United Kingdom UN United Nations UNCTAD United Nations Conference on Trade and Development UNFPA United Nations Population Fund US United States USAID United States Agency for International Development USAID/KEA USAID/Kenya and East Africa USD United States Dollar USDA United States Department of Agriculture USG United States Government WTO World Trade Organization USAID.GOV COMESA PERFORMANCE EVALUATION | 1 EXECUTIVE SUMMARY The Common Market for Eastern and Southern Africa (COMESA) comprises 19 countries aiming to increase trade and enhance regional economic integration through a range of harmonization programs, funded by both member states and development partner contributions. The United States Agency for International Development (USAID) has supported COMESA programs for 20 years. Since 2009, the Integrated Partnership Assistance Agreement (IPAA) has funded a broad range of activities that have resulted in several significant achievements. This performance evaluation looks at the effectiveness and sustainability of seven of the 12 USAID-funded activities under IPAA Implementation Letter 11 to determine which activities have worked well and which are not achieving expected results. The seven evaluated activities include three under the specialized COMESA agency known as the Alliance for Commodity Trade in Eastern and Southern Africa (ACTESA): the COMESA Seed Harmonization Implementation Program (COMSHIP), the COMESA Fertilizer Initiative, and the COMESA Biotechnology and Biosafety Initiative. Another three programs fall under the COMESA Secretariat itself: the COMESA Sanitary and Phytosanitary (SPS) Initiative, Trading 4 Peace, and the COMESA Statistics Program. The final evaluated program is the COMESA Business Council (CBC), another specialized COMESA agency supported by USAID. This evaluation, conducted by Management Systems International (MSI) under the Kenya Support Project (USAID/KSP) involved fieldwork in five countries (Ethiopia, Kenya, Rwanda, Uganda, and Zambia) from October through December 2017. The fieldwork included meetings with officials from the COMESA Secretariat, USAID/Kenya and East Africa (KEA), and national-level officials from the public and private sectors. Using information from 53 key informant interviews (KIIs) and an extensive review of background documents, the evaluation team derived both quantitative and qualitative data. Based on this information, the team developed findings, conclusions, and recommendations in response to each of the five evaluation questions. FINDINGS Overall, the evaluation team found that USAID’s funding to COMESA was aligned with key USAID objectives, such as reducing poverty, improving food security, and increasing the resilience of vulnerable populations in the member states. These activities have enhanced COMESA member state capacity to engage in harmonization of policies across the region to deepen regional development and integration. Evaluation Question 1 asks which activities USAID should prioritize for funding. The evaluation team used four criteria to make this recommendation. In the order of importance, these were: effectiveness, direct effect on intra-regional trade, relevance to stakeholders, and whether USAID has a comparative advantage as a development partner. Based on achievements recorded over the period September 30, 2009 through September 30, 2017, the team concluded that all activities except the Fertilizer Initiative were effective. The evaluation team found that four of the seven activities demonstrated clear relevance to their stakeholders – to member states in the case of the SPS Initiative, Trading 4 Peace, and COMSHIP, and to the private sector in the case of the CBC. The same four activities were found to contribute concrete results toward the overall goal of increasing intra-regional trade. The Biotechnology and Biosafety Initiative and the Statistics 2 | COMESA PERFORMANCE EVALUATION USAID.GOV Program were not specifically designed to directly increase intra-regional trade but were complementary to the regional integration process. The Fertilizer Initiative engaged in relevant research, but did not advance past the research phase into taking meaningful action to advance intra-regional trade. USAID’s comparative advantage was not found to be a deciding factor in the team’s recommendations. While respondents believed that USAID had an advantage compared to other development partners in the areas of private sector development and regional integration, this advantage affected all activities, except for the Statistics Program. Of the seven activities studied, the evaluation team recommends three for priority funding: COMSHIP, the SPS Initiative, and the CBC. COMSHIP stands out as particularly effective, having negotiated a regional seed trade protocol and a regional Seed Variety Catalogue. Under this protocol, a seed variety that has been properly tested and registered in two COMESA countries is accepted by all the other countries. A seed variety that is registered and maintained in the COMESA Seed Catalogues means that the seed variety can be legally commercialized among all the COMESA countries. Seven COMESA countries have fully adopted the seed trade protocol into their national legislation. COMSHIP also appears to be responding to demand, with respondents in Ethiopia and Rwanda expressing interest in importing seeds under the harmonized regulations. The SPS Initiative has elicited strong support both from COMESA member states and international development partners, attracting at least six additional sources of funding for its four activities: Pest Risk Analysis, Regional Pest Listing, Quarantine, and Conformity Assessment. The evaluation team recommends that USAID continue funding the SPS activity with an emphasis on the most trade-inducing sub-activities: Pest Risk Analysis and Regional Pest Listing. The CBC was found to have improved its relevance to its private sector members and its usefulness to its local counterpart, the national chambers of commerce. In 2017, the share of its budget that comes from membership dues increased by 18 percent (with companies paying USD $5,000 a year in membership fees). The CBC Local Sourcing Partnership is a laudable example of supplier-buyer matchmaking, as it trains agribusiness companies and food suppliers of all sizes in Hazard Analysis and Critical Control Points (HACCP) certification, and in skills for negotiating and executing sales contracts with hotels and supermarket chains. The CBC can point to actual sales achieved and sustainable market penetration as evidence of its effectiveness and relevance. The evaluation team also found that, in funding the Statistics Program, USAID missed an opportunity to direct the Program to provide technical support to other USAID-funded activities. In future, USAID should ensure that any activity includes the support of the Statistics Program, where relevant. Evaluation Question 2 asks to what extent USAID’s support has leveraged, and not duplicated, other resources. Five of the seven activities under study — SPS, Trading 4 Peace, CBC, COMSHIP, and the Statistics Program — were found to have leveraged additional support. SPS and Trading 4 Peace were particularly effective at leveraging other resources as each attracted about USD 10 million in additional funds. The evaluation team did not find any examples of duplication or overlap between activities funded by USAID and those funded by COMESA itself or other development partners. USAID.GOV COMESA PERFORMANCE EVALUATION | 3 Evaluation Question 3 asks what COMESA has done to coordinate development partner support. COMESA has taken the following steps in line with its new Regional Integration Strategy and 2015 Management Decision: • Set up a Resource Mobilization and Technical Cooperation Unit under the Secretary General; • Mandated that all development partner-funded program activities coordinate with this unit; and • Convened a Development Partners Roundtable Partnership Forum in November 2016. Despite this progress, COMESA’s coordination of development partners continues to be an ad hoc activity, based on the efforts of activity managers, rather than conducted through institutional coordination structures. The evaluation team concluded that development partner coordination of USAID activities was not particularly effective. In response to Evaluation Question 4, the evaluation team concluded that COMESA’s current management structures and organizational systems were not conducive to achieving the objectives of the USAID-funded activities. The USAID Coordinating Unit could have more effectively executed its coordination role. The evaluation team identified many factors hampering the effectiveness of COMESA as a whole, which also threaten USAID-supported programs, including: • Heavy reliance on development partner funding, even for core functions, • Decreasing development partner support for the current fiscal year, • Failure to collect membership dues from 16 of 19 member states, • Poor monitoring and reporting capabilities, • Poor incentives to collaborate, • Ineffective coordination between the USAID Coordination Unit and the Resource Mobilization and International Cooperation Unit, and • High dependence on technical staff funded by development partner projects. The combination of a decrease in development partner support and a shortfall in membership dues represents an impending budget crisis that puts COMESA’s work at risk. Respondents noted that it will be difficult to undertake significant reforms before the new Secretary General takes office. However, a change in leadership will potentially create new energy to advance the reform agenda. As for the structures guiding the USAID Coordinating Unit, the evaluation team found that USAID did not adequately specify the expected results or provide appropriate indicators or targets with which to monitor progress. Regular reporting requirements were not clear. As a result, USAID and COMESA can point to achievements of the USAID-funded activities, but cannot demonstrate that the original expectations were met. Evaluation Question 5 asks to what extent the selected USAID-funded activities could be sustained with other resources. The evaluation team found that the prospects for sustainability through other funding sources are poor for the Fertilizer Initiative and Biotechnology and Biosafety Initiative. This is because they have neither self-financing options nor significant support from other development partners. The remaining activities have support from other development partners. In addition to development partner support, the CBC has demonstrated the ability to generate income from membership, and COMSHIP has the potential to raise its own funds through the COMESA Variety 4 | COMESA PERFORMANCE EVALUATION USAID.GOV Catalogue and COMESA seed labels and certificates. The evaluation team did not find that any of the activities studied could absorb a significant reduction in funding without adjusting their near-term work plans accordingly. RECOMMENDATIONS The following key recommendations are based on the findings of this evaluation: 1. USAID should consider a milestone-based agreement with COMESA, focused on the achievement of specific outcomes. 2. USAID should prioritize funding for the CBC, COMSHIP, and the SPS Initiative, as these activities are effective, relevant for stakeholders, and the most likely to induce intra-regional trade. 3. USAID should consider funding sub-activities, rather than funding the entire portfolio of activities. The evaluation team recommends the following sub-activities: • COMSHIP: Address lack of awareness of the COMSHIP Variety Catalogue by targeting seed companies and or agro-dealers for sensitization and training on how it works. • SPS: Support a systems approach to the management of SPS risks by targeting an identified SPS issue, the affected commodity/value chain and the relevant market. • CBC: Support the expansion of the Local Sourcing Partnership to include more countries. • Support the Statistics Program as a sub-activity of COMSHIP, SPS, and CBC, to provide targeted support to each activity. 4. USAID should consider supporting COMESA Secretariat institutional strengthening, specifically focusing on the development of an effective M&E systems and establishment of the Executive Support Division. 5. COMESA should focus on improving development partner coordination. Specifically: • COMESA should emphasize joint planning, reporting, and collaboration with development partners, in addition to its focus on resource mobilization. • COMESA should be supported to integrate the USAID Coordinating Unit into the Resource Mobilization and International Cooperation Unit and institutionalize development partners coordination. In this way, USAID’s Coordination Unit at COMESA would conform to the terms of the 2015 decision of the COMESA Council of Ministers, while reducing costs, increasing visibility, and strengthening the link for outreach to member states. • COMESA should consider offering training on resource coordination to its staff. This might include techniques for identifying sources of potential funding, facilitating collaboration, and leveraging resources. 6. COMESA should focus on efficiency by fully implementing the 2015 Council of Ministers decision on organizational reform, reducing its expenditures and focusing on its core mission. 7. COMESA should strengthen its monitoring and evaluation (M&E) system. USAID should ensure that future agreements feature well-designed indicators and targets, and clear reporting requirements. USAID.GOV COMESA PERFORMANCE EVALUATION | 5 USAID staff should review COMESA’s written reports thoroughly to ensure that progress is tracked appropriately. 6 | COMESA PERFORMANCE EVALUATION USAID.GOV EVALUATION PURPOSE AND EVALUATION QUESTIONS This performance evaluation is designed to evaluate the effectiveness and sustainability of specific USAID-funded Common Market for Eastern and Southern Africa (COMESA) activities implemented under the Integrated Partnership Assistance Agreement (IPAA) Letter 11. Part of the evaluation’s purpose is to help USAID/Kenya and East Africa (USAID/KEA) better understand which activities have been most effective and what factors are associated with their success, to inform future programming decisions. The specific USAID-funded activities covered by this evaluation are: • COMESA Seed Harmonization Implementation Plan (COMSHIP); • COMESA Fertilizer Initiative; • COMESA Biotechnology and Biosafety Initiative; • COMESA Sanitary and Phytosanitary (SPS) Initiative; • Trading 4 Peace; • COMESA Statistics Program; and • COMESA Business Council (CBC). The scope of work for this evaluation outlines five evaluation questions, listed below. EVALUATION QUESTIONS Evaluation Question 1: From the specific USAID-funded COMESA activities reviewed under this evaluation, which should USAID prioritize for funding to maximize results and exploit USAID’s comparative advantage? Evaluation Question 2: To what extent are USAID-funded activities successfully leveraging, rather than duplicating, efforts funded by other development partners (DPs) or stakeholders toward the same results? Describe the areas of current and potential leveraging and duplication related to the specified activities and expected results. Evaluation Question 3: Describe the steps COMESA has taken to coordinate development partner support for the specified activities and results. Evaluation Question 4: To what extent are COMESA’s management systems and organizational structures conducive to achieving the expected results of the specified USAID-funded activities? Evaluation Question 5: To what extent can the specific USAID-funded activities be sustained through other sources of funding? The task of the evaluation team was to derive key findings, conclusions, and recommendations for each evaluation question. USAID.GOV COMESA PERFORMANCE EVALUATION | 7 PROJECT BACKGROUND USAID support for COMESA goes back to 1998. Since 2009, the IPAA has governed the relationship, with specific funding provided through implementation letters (ILs). A new long-term agreement, the Regional Development Objective Assistance Grant (RDOAG), was signed in September 2016. The first IL linked to the RDOAG is yet to be signed. Under the present IL 11, USAID supports 12 COMESA activities implemented through the COMESA Secretariat or one of two COMESA independent agencies: ACTESA and the CBC. USAID/KEA’s Regional Economic Integration Office (REI) selected seven of the 12 USAID-supported COMESA activities for this evaluation. Three of the activities are under ACTESA: COMSHIP, the COMESA Fertilizer Initiative, and the COMESA Biotechnology and Biosafety Initiative. Three other activities fall under the COMESA Secretariat itself: SPS Initiative, Trading 4 Peace, and the COMESA Statistics Program. The CBC is an independent COMESA agency. Managing the COMESA activities requires that the COMESA Secretariat or its specialized agencies reach out to the 19 COMESA member states to promote common policies as part of the COMESA common market agreement for the free movement of persons, goods, services, and capital. COMESA activities aim to bring member states into compliance with common policies as far as possible so that any differences do not represent a distortion of the competitive environment or present barriers to trade. EVALUATION DESIGN, METHODS AND LIMITATIONS The evaluation team conducted fieldwork from October 13 through November 21, 2017. Because of COMESA scheduling conflicts, the team began in Lusaka, Zambia. From October 13–20, the team interviewed staff members of the COMESA Secretariat and its independent agencies. The team worked in Nairobi, Kenya, from October 20-26, interviewing USAID/KEA staff members and drafting the evaluation inception report. The evaluation team members then split up, visiting Ethiopia, Rwanda, and Uganda. The team reunited in Nairobi on November 12 to analyze data and prepare for a validation meeting with USAID/KEA. Additional interviews took place in Zambia between November 21 and 25. This evaluation was designed to draw primarily on qualitative data, although some quantitative data were also obtained for analysis. The evaluation team prepared data collection tools tailored for each category of stakeholders (see Annex III). Specific questions within the tools were linked to the main evaluation questions to ensure that data sources were adequate for each evaluation question. At the end of each interview, the team members transcribed their interview notes to create a transparent and permanent record of the information provided by the key informant interviews (KIIs). The qualitative methods that the team used permitted collection of in-depth information from the various stakeholders. Where possible, the evaluation team sought to triangulate information by posing questions to a range of stakeholders on the various themes. The team conducted 53 KIIs and supplemented this data with extensive background reading, examination of project documents, and follow-up questions by email. Team members examined more than 100 documents, approximately 40 of which they considered critical for responding to the five evaluation questions. 8 | COMESA PERFORMANCE EVALUATION USAID.GOV During the data analysis phase, the evaluation team reviewed and analyzed the qualitative and quantitative information collected, identifying recurring themes and linking them to the evaluation questions. Team members used inductive coding to convert qualitative information obtained during the KIIs into quantitative information, such as the number of respondents expressing the need for COMESA to improve its development partner coordination system. This evaluation contained some identifiable limitations. As outlined in detail below, USAID did not require COMESA to provide regular reports on a consistent set of indicators with targets. The failure of USAID to require an adequate monitoring system at the onset of funding made it difficult for the evaluation team to assess the effectiveness of the various activities in a rigorous way. In carrying out fieldwork, selection bias was a concern because the COMESA Secretariat provided the contacts for member states, relevant institutions, and beneficiary community representatives. Thus, the team heard from people who have a vested interest in each of the USAID-funded activities continuing, and who may have a more favorable opinion than the average stakeholder. The evaluation team mitigated potential bias by selecting and interviewing additional respondents from other institutions. Finally, each evaluation team member brought a slightly different perspective to the process, introducing some potential for personal bias. The evaluation team did not always pose the questions in the data collection tools in the stated order; they asked the questions as part of a conversation or discussion. To elicit meaningful responses, the evaluation team, at times, introduced a question as a “working hypothesis” to better frame it for the interviewee. The evaluation team attempted to control for bias by meeting often to take in other team members’ points of view, as well as by examining potential sources of bias for each theme discussed and, to the extent possible, correcting them. This occurred through regular debriefing sessions during fieldwork and the analysis process. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS EVALUATION QUESTION 1 From the specific USAID-funded COMESA activities reviewed under this evaluation, which should USAID prioritize for funding to maximize results and exploit USAID’s comparative advantage? The evaluation team was asked to examine seven of the 12 USAID-funded COMESA activities in Fiscal Year (FY) 2017, and recommend up to three for continued funding. The seven activities vary substantially (see Table 1). Some are broadly related to trade, while others are directly related to agriculture. Some have been funded by USAID for many years (since 2004, in the case of Trading 4 Peace); others began receiving funding more recently (in 2017, in the case of the Statistics Program). Most receive funding from many other development partners in addition to USAID. USAID.GOV COMESA PERFORMANCE EVALUATION | 9 TABLE 1. SEVEN USAID-FUNDED COMESA ACTIVITIES UNDER REVIEW Activity Activity Description USAID Funding Began FY 2017 Funding Amount (USD) Other Sources of Funding COMESA Biotechnology and Biosafety Initiative Supports the development of regulatory capacity in biosafety and biotechnology within member states. FY 2009/2010 $200,000 Primarily funded by USAID COMESA Business Council (CBC) Provides business support services and linkages, addresses constraints to business and competitiveness in the region and influences the policy agenda on behalf of the private sector and the regional integration agenda. FY 2010/11 $419,967 Various other DPs COMESA Fertilizer Initiative Promotes the accessibility and availability of high performance fertilizer across the region by promoting fertilizer standards, policies and regulations, as well as enhancing market access across the member states. FY 2014/15 $90,000* Various other DPs Sanitary and Phytosanitary (SPS) Initiative Focuses on harmonizing technical standards in SPS measures and development of policy to facilitate trade in agricultural commodities among member states and with other countries. FY2010/11 $319,125 Various other DPs COMESA Seed Harmonization Implementation Plan (COMSHIP) Focuses on harmonizing seed trade across member states, particularly through the improvement of seed quality and variety for a sustainable seed industry. FY 2010/11** $350,000 Various other DPs Statistics Program Focuses on improving the quality and availability of trade data among reporting countries by providing training and technical support to member states. FY 2017 $520,000 Various other DPs Trading 4 Peace Promotes peace and aims to reduce poverty in post-conflict areas across the Great Lakes region through strengthening formalized trade across borders. FY 2009/2010 $83,200 Other DPs fund other borders; USAID is sole funder at Ishasha *allocated, not used **no budget for FY2011/12, funding resumed in FY2012/13 10 | COMESA PERFORMANCE EVALUATION USAID.GOV The evaluation team used the following criteria to prioritize activities, listed in order of importance: • Effectiveness, as measured by the attainment of the expected results set in the series of Implementation Letters (ILs), as well as general progress; • Effect on increasing intra-regional trade; • Relevance for member states; and • Consideration of USAID’s comparative advantage as a development partner. Evaluating the effectiveness of the seven activities was challenging, given inconsistent reporting on indicators and a lack of targets. As Table 2 shows, IL 11 describes expected results and indicators for each of the original activities (Statistics is not mentioned, as it was introduced in IL 11, Amendment 2). Progress reports of the USAID Coordinating Unit provide an overview of activities and challenges, but do not report on a consistent set of indicators specified. The FY 2015 progress report features a different set of indicators than that mentioned in the IL, while the FY 2016 progress report does not systematically report on indicators at all. The FY 2017 progress report provides data on a set of indicators that do not correspond with the FY 2015 indicators or the IL indicators. Table 2 shows the expected results and indicators as written in IL 11; alongside are indicators, results, and targets that were included in the FY 2017 Progress Report. In many cases, the indicators in the FY 2017 Progress Report are better designed than the original indicators, but their apparent adoption later in the program and the lack of targets call into question their use for management purposes or for assessing the effectiveness of the activities. USAID.GOV COMESA PERFORMANCE EVALUATION | 11 TABLE 2. EXPECTED RESULTS, INDICATORS AND TARGETS OF THE SEVEN USAID-FUNDED COMESA ACTIVITIES Activity Source: Implementation Letter 11 Source: FY 2017 Progress Report Expected Results IL 11 Indicators FY 2017 Report Indicators Actual Result Targets COMESA Biotechnology and Biosafety Initiative - COMESA member state participation in implementation plan validation - Bt cotton commercial success in Africa will be studied through field visits - Sharing of experiences between member states Number of countries that participate in policy implementation plan validation and domestication Number of individuals who have received USG-supported short-term agricultural sector productivity or food security training as a result of USG assistance 131 100 Number of individuals supported in training/ study tours/ field experiences COMESA Business Council (CBC) - Five national monitoring committees adopting terms of reference for their operation and roadmaps for monitoring - Number of small and medium enterprises (SME) staff trained - The adoption of the CBC Rules of Corporate Governance and Rules of Engagement plan Number of National Monitoring Committee meetings conducted Number and type of business linkages created across the COMESA region to support SMEs. 127 200 Advancement of the monitoring roadmap through the policy continuum Numbers of policies, regulations, and administrative procedures in each of the following stages of development as a result of USG assistance 1 (Stage 4) 1 (Stage 4) Number of SME staff trained as a result of USG assistance Number of individuals who have received USG-supported short-term agricultural sector productivity or food security training as a result of USG assistance 260 200 Number of governance committee meetings conducted Number of new service industry members registered with CBC Secretariat 4 7 COMESA Fertilizer Harmonization Program - National fertilizer policy reports validated - Roadmap for standards and regulations drafted Advancement through the policy continuum as a result of USG assistance Not mentioned in FY 2016 or FY 2017 reports Sanitary and Phytosanitary Initiative (SPS) - Improved compliance with SPS import requirements by SMEs and small traders - Member state participation validation of a comprehensive SPS strategy Number of individuals trained on risk and policy analysis Number of individuals who have received USG-supported short-term agricultural sector productivity or food security training as a result of USG assistance 27 25 Advancement of relevant elements through the policy continuum as a result of USG assistance 12 | COMESA PERFORMANCE EVALUATION USAID.GOV Activity Source: Implementation Letter 11 Source: FY 2017 Progress Report Expected Results IL 11 Indicators FY 2017 Report Indicators Actual Result Targets - Reports on demand creation for aflatoxin mitigation tools Number of reports completed on demand creation for aflatoxin mitigation tools Number of countries supported in conducting pest listing and Pest Risk Analysis. 8 6 COMESA Seed Harmonization Implementation Plan (COMSHIP) - Distribution of materials and training on the COMESA Simplified Trade Regime (STR) for public and private stakeholders - Training of customs staff on the COMESA STR - Training on maize lethal necrosis (MLN) mitigation tools and measures - Alignment on member state seed laws Number of stakeholders trained on the COMESA STR and MLN mitigation Number of national policies for which an action has been taken toward the full implementation of a regionally agreed-upon policy as a result of USG assistance. Advancement of 2 2 member states through the policy continuum as a result of USG assistance Statistics Per IL 11, Amendment 2: - Four staff trained in Microsoft SQL - Four staff trained in SDMX - Four staff trained in CSPro - Two staff trained in survey sampling techniques - Three staff trained in various online statistical data analysis courses Number of officials who have trained in joint bilateral/mirror trade statistics reconciliation exercises 26 16 Number of countries that have successfully migrated to SQL server based on Eurotrace 2 2 Number of staff trained on different statistical competencies 4 4 Trading 4 Peace (T4P) - The establishment of peacebuilding structures at borders - Increased border trade by small-scale traders - Trade disputes resolved - Independence and sustainability of cross border trade associations (CBTAs) increased Number of peace￾building structures established or strengthened with USG assistance Number of individuals who have received USG-supported short-term agricultural sector productivity or food security training as a result of USG assistance 0 120 Number of trade information desk users Number of CBTA members trained USAID.GOV COMESA PERFORMANCE EVALUATION | 13 FINDINGS This section presents findings from each of the seven activities, related to their effectiveness, direct effect on trade, and relevance to member states. 1. Seed Harmonization Plan (COMSHIP) The COMESA Seed Harmonization Implementation Plan (COMSHIP) focuses on harmonizing seed trade across member states to increase the production, reliability, trade, and competitiveness of the seed industry in the COMESA region. The activity aims to facilitate farmer access to improved seeds through simplified customs procedures, speedy seed varietal release and distribution into other countries, and elimination of inhibitive quarantine requirements through the adoption of a common pest list. COMSHIP receives support from USAID and various other development partners, and COMESA’s specialized agency, the Alliance for Commodity Trade in Eastern and Southern Africa (ACTESA), implements it. In providing funding for COMSHIP in IL 11, USAID specified four expected results: distribution of materials and training on the COMESA Simplified Trade Regime (STR) for public and private stakeholders, training of customs staff on the COMESA STR, training on maize lethal necrosis (MLN) mitigation tools, and measures and alignment of member state seed laws. For the first expected result, distribution of materials and training on the COMESA STR for public and private stakeholders, no target was established. However, the USAID Coordinating Unit’s FY 2017 progress report suggests that activities in this area were completed by September 2017. The second expected result, training of customs staff on the STR, was accomplished in July 2016, per the FY 2016 progress report. The third expected result, training on MLN tools and measures, was accomplished in July 2016, with eight countries benefiting. With additional support from DFID, the fourth and most ambitious result was realized as five countries (Burundi, Kenya, Rwanda, Uganda, and Zimbabwe) have succeeded in adopting the range of COMESA harmonized seed policies and regulations. The FY 2017 progress report suggests imminent success in the Democratic Republic of Congo (DRC) and Ethiopia. Considering USAID’s expected results under IL 11, the evaluation team determined that COMSHIP has met its goals. An activity with promise that is not highlighted in the IL is the creation of the COMESA Seed Variety Catalogue, where, according to the COMESA activity manager, roughly 505 varieties were registered as of December 2017. The number of seed companies registered in the catalog is expected to rise from the present five to 20 by September 2021. In addition, the COMSHIP activity adds value to COMESA’s planned agricultural goals by encouraging collaboration among seed experts at all levels. Interviews with key informants suggested that the COMSHIP program has advanced further in the planned harmonization process than most of the other programs, and has the potential for boosting trade in the coming years. Officials interviewed from the Ethiopian and Rwandan governments are eager to import seeds more easily under COMSHIP as they face a deficit of these seeds. Private sector seed trade associations in Uganda and Zambia are already utilizing the COMSHIP regional mutual recognition framework to expand exports into neighboring COMESA countries. However, interviews with seed companies and a regional maize seed expert suggested that existing trade may be better explained by private companies taking initiative than the work of COMSHIP. The key successes associated with this activity include passage of the COMESA Seed Harmonization Protocol and the launch of the COMSHIP Variety Catalogue. The main challenge of the COMSHIP 14 | COMESA PERFORMANCE EVALUATION USAID.GOV program is to encourage liberalization of the seed market in each country and encourage seed companies to better market their improved seeds to the general public. In summary, COMSHIP is a program that has proven to be effective, has a direct effect on trade, and is relevant to member states. 2. Fertilizer Initiative USAID provided funding to the COMESA Fertilizer Initiative under IPAA ILs 10 and 11 to advance regional fertilizer standards, policies, and regulations, and to increase accessibility for the region’s farmers of high-performance fertilizer. The expected results listed in IL 11 included validation of national fertilizer reports and the drafting of a roadmap for standards and regulations. As reported in the FY 2015 progress report, the validation of national fertilizer reports was completed through support from the African Fertilizer and Agribusiness Partnership (AFAP). COMESA planned to develop the roadmap by convening regional fertilizer experts (four from each of the 19-member states) to a meeting. USAID’s designated funding for the Fertilizer Initiative was not sufficient to cover the total cost of the meeting (estimated to be USD $370,000). According to the USAID Coordinating Unit, the roadmap activity was canceled because the additional funds could not be raised. The USAID funding designated for the Fertilizer Initiative in FY 2017 was reportedly not used. The FY 2016 and FY 2017 USAID Coordinating Unit’s progress reports make no mention of the Fertilizer Initiative or explain the lack of results. The Fertilizer Initiative is currently inactive. USAID cannot claim much success with its recent support to the Fertilizer Initiative. Looking at the Fertilizer Initiative’s work more broadly, the main achievements have been the successful review of national and regional fertilizer policies and conducting national validation workshops. One challenge in advancing the fertilizer agenda at the regional level is that fertilizer is a complex and controversial subject: COMESA member states closely regulate production, importation, distribution, and use of fertilizers. Adding to the complexity is the fact that several ministries often oversee fertilizer production, use, and importation — including the agriculture, trade, and treasury ministries. The USAID Coordinating Unit’s progress reports confirm that national bans on fertilizer imports or exports are in place in many COMESA countries, limiting the potential for intra-regional trade. The evaluation team also noted that COMESA was working on an ambitious plan for self-financing involving regional bulk procurement of fertilizer, but that plan failed. Key informants across the member states, including ministries of agriculture representatives, fertilizer and seed experts, and representatives of private sector seed trade associations, stated their belief that the plan was unrealistic because soil profiles vary widely across the region. This presents a major challenge to the development of regionally harmonized fertilizer blends. Because of these challenges, the COMESA Fertilizer Development Program has stalled. By any measure, USAID funding to the Fertilizer Initiative was not effective. Given the complexity and the sensitivity of the subject, it is unlikely that member states would particularly welcome continued efforts, or that the efforts would lead to increased intra-regional trade. 3. Biotechnology and Biosafety Initiative Through its funding of the Biotechnology and Biosafety Initiative, USAID has been encouraging the development of regulatory capacity in biosafety and biotechnology within COMESA member states. The USAID.GOV COMESA PERFORMANCE EVALUATION | 15 activity, which receives the majority of its funding from USAID, also aims to strengthen awareness and communication on biotechnology and safety. Implementation Letter 11 specifies three expected results for the Biotechnology and Biosafety Initiative. The first result is member state validation of the COMESA Regional Biotechnology and Biosafety Implementation Plan, which was designed to operationalize the COMESA Policy on Biotechnology and Biosafety. The goal of the plan is to support the member states to become active participants in the global biotechnology enterprise through commercial planting of genetically modified (GM) crops, trade in products of GM technology, and acceptance of emergency food aid with GM content. The validation of the plan was completed after an exhaustive consultation process in the region, which culminated in three regional validation workshops. The second result required the Biotechnology and Biosafety Initiative to organize field visits for member state representatives to see the commercial success of Bt cotton in Africa. The Initiative led visits to Sudan, Swaziland, and Ethiopia, as well as a study trip to India. The number of participants was not systematically reported, but information in the progress reports indicates that the Biotechnology and Biosafety Initiative appears to have facilitated more than 100 individuals to take trips during the life of IL 11. The third expected result was the “sharing of experiences between member states.” It is not clear what experiences the IL refers to, apart from the field visits. IL 11 requires reporting the number of individuals who have received USG-supported short-term agricultural sector productivity or food security training. The USAID Coordinating Unit’s FY 2017 progress report indicates that 131 individuals received training, compared to a target of 100. Beyond the key expected results, the Biotechnology and Biosafety Initiative had other achievements. All 19 countries have approved the Regional Biotechnology Protocol, even those countries that are hesitant to allow GM organisms. By conducting training and sensitization in-country and helping supplier companies make contacts, the biotech program has contributed to Sudan’s cultivation of 120,000 hectares of Bt cotton and its success in exporting it, mainly to Turkey. Other accomplishments are the establishment of a regional-level biotechnology risk assessment system. The Biotechnology and Biosafety Initiative’s outreach activities have included trainings, stakeholder workshops across the region, and facilitation of media representatives to interact with biotech experts. Measuring the effectiveness of such outreach efforts is difficult, and COMESA did not attempt to do so. The evaluation team was unable to comprehensively assess the effectiveness of the outreach efforts, but journalists in Uganda and Rwanda confirmed that it enhanced awareness of biotech and biosafety issues in the region. The evaluation team found a general perception among respondents in a number of member states that U.S. interests drove the program and that it did not reflect demand within COMESA. The evaluation team heard numerous examples of how the issue of GM organisms can be divisive, even within governments. In Zambia, for example, the Ministry of Agriculture and Cooperatives is reluctant to encourage biotechnology because of food safety concerns and the national campaign to promote organic farming, while the Ministry of Tourism, Environment and Natural Resources views biotechnology as a potential way to combat the effects of climate change. Some key informants in the member states agreed that while the general public is fearful of biotech food crops, they may be more open toward non-food crops, such as cotton or biofuels. 16 | COMESA PERFORMANCE EVALUATION USAID.GOV The Biotechnology and Biosafety Initiative was successful in meeting some of its key results, but the relevance of the program for member states and the direct relation to intra-regional trade – other criteria applied by the evaluation team – are less clear. Efforts to promote biotechnology products are not always welcome, depending on the member state or the entity within it. Additionally, biotechnology products come from outside COMESA (e.g., from the U.S., China, India, Brazil, South Africa, Argentina) and are not traded between COMESA countries; therefore, they are not directly tied to intra-regional trade. 4. Sanitary and Phytosanitary (SPS) Initiative The COMESA SPS Initiative focuses on technical standards in SPS measures and development of SPS policy to facilitate trade in agricultural commodities among member states and with other countries. The activity enhances the SPS capacities of public and private sectors within the member states, and promotes regional leadership, coordination and collaboration on SPS issues, as well as private-sector￾driven common certification schemes/protocols and standards. It also supports monitoring, surveillance, diagnostic and emergency response systems for priority SPS risks. SPS is funded by USAID and various other development partners and has widespread support, in part due to the dynamism of the activity manager. Implementation Letter 11 specifies three expected results for the SPS Initiative. The first, improved compliance with SPS import requirements by small and medium enterprises (SMEs) and small traders, aligns with the achievements reported by the SPS Initiative. The training conducted on SPS import requirements for small-scale cross-border traders was clearly linked to this result. In fact, this activity, together with the sub-activity cross-border assessment that measured the costs associated with compliance in mid-2017, could be said to contribute to improved compliance with SPS measures and hence facilitated increased regional trade. This study could not have led to the first expected result in the timeframe of IL 11. The second expected result, member state participation in the validation of a comprehensive SPS strategy, was met in July 2016 when 14-member states participated in a strategy validation workshop. The third expected result reporting on demand creation for aflatoxin mitigation tools, was met in 2016 when the SPS Initiative completed a feasibility study in six-member states and 12 regulatory laboratories, and selected industry stakeholders received training on sampling for aflatoxin control. The evaluation team identified other SPS Initiative achievements that can be directly attributed to USAID funding. In Madagascar, USAID funding resulted in the SPS Initiative implementing a sub-activity, called Pest Risk Analysis, that provided laboratories with training and facility upgrades. This work led directly to the opening of South Africa’s market for lychee fruit in January 2017. USAID support has also promoted awareness-raising initiatives in the COMESA member states about the importance of combating aflatoxin, a food safety risk proven to lead to stunting in children, liver cancer, and other health problems for pregnant women and other vulnerable populations. For example, during FY 2016, specialists from 12 national laboratories received training in workshops on sampling for aflatoxin control. Uganda’s National Bureau of Standards achieved International Standardization Organization certification of aflatoxin testing capability in December 2017 following evaluation by the South African National Accreditation System. In an interview, an official from the Bureau of Standards asserted that USAID’s support directly increased the Government of Uganda’s technical capacity and enabled higher maize exports. This is because the maize exports require the aflatoxin analysis for food safety purposes. In a sign of local ownership, Uganda’s Bureau of Standards developed its own sampling standard USAID.GOV COMESA PERFORMANCE EVALUATION | 17 operating procedures and validated method of aflatoxin analysis, which has resulted in improved food safety and the ability to trade food products across borders. The SPS Initiative has had many other achievements and a range of funding sources. Some of its successes cannot be as closely linked to USAID support, but they nonetheless illustrate the strength of the initiative. As of October 2017, nine countries had approved the COMESA SPS Strategy and adopted the result areas and indicators into their own national SPS planning. According to the COMESA activity manager, all 19 COMESA countries are now working to report against the four result areas of the SPS Strategy. Ongoing common activities include: Pest Risk Assessment, Conformity Assessment, Quarantine and Regional Pest Lists. Respondents pointed out that the SPS Initiative provides policymakers and private sector actors with a useful way to know what is happening in each member state based on a framework with common results and indicators. The SPS Initiative has proven to be successful and valued by member states. One member state respondent said, “SPS is very, very important. The COMESA SPS Initiative has helped us organize our SPS system better, as we had a lot of different institutions playing different roles and the COMESA system got us working with each other and getting better organized. COMESA also helped raise awareness about SPS among producers.” In terms of its contribution to trade, SPS measures traditionally have been used to block trade, rather than promote it. However, the COMESA SPS Initiative now has COMESA member states “speaking the same language” when it comes to SPS, so that the future should feature fewer SPS disputes and those that do arise should be easier to resolve. 5. Trading 4 Peace The Trading 4 Peace Initiative falls within the peace and security cluster of COMESA activities. This is distinct from the agriculture or business-oriented COMESA programs supported by USAID. The aim of Trading 4 Peace is to promote peace and reduce poverty by strengthening formalized trade across borders. It is the longest-running activity the evaluation team studied, receiving funding from USAID since its inception in 2004. Under IL 11, USAID reduced its support to Trading 4 Peace to just one border crossing at Ishasha between DRC and Uganda, where it has supported a Trade Information Desk (TID) and the cross-border trade association. Implementation Letter 11 specifies four expected results for Trading 4 Peace: the establishment of peacebuilding structures at borders; increased cross-border trade by small-scale traders; resolution of trade disputes; and increasing independence and sustainability of cross-border trading associations. The establishment of the Ishasha Trade Information Center signaled the achievement of the first expected result. Determining whether Trading 4 Peace met its second result is more difficult. Data on the cross￾border trade of small-scale traders are not readily available. Trading 4 Peace reports an increase in the number of border crossings per day; although this is not an entirely satisfactory proxy, it provides some measure of activity. In response to the third expected result, the FY 2017 progress report states that the TID handled 302 visitors within the six-month reporting time frame of October 2016-March 2017. It also resolved complaints from 13 traders (eight of whom were women) on offenses ranging from bribery and intimidation to confiscation of goods. Without targets, it is difficult to determine whether this number meets expectations or represents success. For the fourth expected result, the USAID Coordinating Unit 18 | COMESA PERFORMANCE EVALUATION USAID.GOV did not directly report on the independence and sustainability of the cross-border trading association at Ishasha, although progress reports discuss some of the difficulties of the cross-border trading associations. Trading 4 Peace offers a relatively simple intervention that promotes and facilitates formalized trade. According to the FY 2017 progress report, 302 traders (155 women) visited the TID at Ishasha seeking trade-related information. Some of the information those visiting the TIDs sought included: • Commodity prices on the other side of the border; • The status of implementation of the COMESA STR; and • Procedures for joining the cross-border traders’ association. Under IL 10, when USAID was the sole funder of Trading 4 Peace, the activity’s work was an important factor in the acceptance of the COMESA STR by all COMESA countries. The COMESA STR is a major step toward bringing small-scale traders into the formal customs procedures. It also represents an opportunity to collect important trade data. The Trading 4 Peace TIDs were established to assist small￾scale traders in filling in the one-page document and filing it with the customs service at the border crossings. Uganda has now adopted the STR at all its borders. Trading 4 Peace has been collecting sex-disaggregated information on the number of people crossing the border, the number of those carrying goods, the number of inquiries at the TIDs, and the number of disputes resolved. Trading 4 Peace missed an opportunity, however, to gather additional trade data (including information on prices and volumes of goods) by using its staff at the border. Another missed opportunity related to the COMESA STR forms that are submitted to each member state’s central bank. IL 11 Amendment 2 provided funding for Trading 4 Peace to collect and aggregate this data on a regional basis, with the support of the COMESA Statistics Program, but this was not done. Further, the evaluation team observed that the Trade Information Desk at the Ishasha border crossing averages only about two inquiries a day, even though its role is to help small-scale, cross-border traders who need assistance filling in the STR form. It is difficult to draw conclusions from this single data point, but it seems to suggest that a share of the goods crossing the border remain outside the STR. USAID’s contribution to the Trading 4 Peace program is now dwarfed by the support given by the European Union and German Development Fund, which have expanded the initial USAID activity to cover all the COMESA borders of the DRC, including use of the COMESA STR, which facilitates the paperwork requirements for informal traders. 6. COMESA Statistics The COMESA Statistics Program focuses on improving the quality and availability of trade data among reporting countries by providing training and technical support to member states. The Statistics Program does not feature in IL 11 until Amendment 2, which authorizes USD $520,000 in FY 2017 for the training of staff in new software programs, data analysis techniques, and survey sampling techniques. The original IL 11 did not include funding for the Statistics Program, but when it was included, the expected results were quite specific, such as the number of staff who should be trained in a specific technology. The USAID Coordinating Unit’s FY 2017 progress report states that the Statistics Program exceeded the target for the number of officials trained in joint bilateral/mirror trade statistics USAID.GOV COMESA PERFORMANCE EVALUATION | 19 reconciliation (26 trained against a target of 16). The program reported meeting two targets: two countries successfully migrated to Structured Query Language (SQL) servers based on Eurotrace, and four staff trained on different statistical competencies. Specific achievements of the COMESA Statistics Program during the single year of USAID support include: • Migration of the Eurotrace software from Microsoft Access to SQL Server to handle daily data inputs — resulting in a great improvement in data availability. • Conducting mirror and reconciliation exercises to harmonize import and export statistics for Ethiopia/Djibouti, Kenya/Uganda, Zambia/DRC and Zambia/Malawi. • Provision of technical assistance to Rwanda, Kenya and Uganda on alignment of their merchandise trade statistics systems to the United Nations Manual of International Merchandise Trade Statistics; and • Engagement with the United Nations Conference on Trade and Development (UNCTAD) on Automated System for Customs Data (ASYCUDA) data quality issues and convening a pilot national workshop on ASYCUDA data quality for the Zambia Revenue Authority and Central Statistical Office. The program also conducted a regional ASYCUDA data quality workshop involving all COMESA countries that use ASCYUDA as a customs system. • Launch of a pilot program on implementation of statistical data and metadata exchange between COMESA and the Zambia Central Statistical Office. This included training workshops and actual implementation. • Capacity building on software tools relevant for the efficient validation, compilation and reporting of statistics. • Publication of the Status of Trade in Food Staples in Eastern and Southern Africa report. This report was used to provide indicators for the Feed Africa monitoring system. 7. COMESA Business Council (CBC) The CBC is a private sector institution of COMESA, charged with increasing private sector participation in the regional integration agenda, and representing private sector interests in COMESA programs and interstate meetings. The CBC works to raise awareness of the private sector and provides relevant business support services and linkages. CBC receives funding from various sources, including USAID. Implementation Letter 11 specifies three expected results for the CBC. The first is that five national monitoring committees will adopt terms of reference for their operation and roadmaps for monitoring. This activity was planned for November 2015, according to the FY 2015 progress report, but subsequent reports do not mention it. The second expected result is training an unspecified number of SME staff. The evaluation team heard of and read about many trainings that CBC held for SME staff, but it was difficult to determine the number of staff trained. The FY 2015 progress report mentions a target of “1,000 SMEs” and an achievement of 50 with no explanation for the deviance. The FY 2016 and 2017 progress reports don’t report the number of SME staff trained. The third expected result, adoption of the CBC Rules of Corporate Governance and Rules of Engagement Plan, was achieved in FY 2015 during the CBC Board meetings, according to project documents. While the expected results in IL 11 weren’t tracked closely, the CBC has achieved impressive results during the period of USAID funding. As of mid-FY 2017, 1,841 companies were registered with CBC online through its “BIZNET” site, and 13 companies had signed memoranda of understanding with the CBC. The CBC convened numerous dialogues with various private sector groups on topics including 20 | COMESA PERFORMANCE EVALUATION USAID.GOV barriers to doing business in various transport corridors and barriers to the movement of businesspeople in the region. The CBC also conducted studies such as mapping supply chains in six countries and constraints faced in the professional services industry. These sub-activities are all relevant to CBC’s objective and respond to issues of interest to its members, even though it is not clear how they will lead to higher-level, measurable results. The CBC’s Local Sourcing Partnership had more concrete results. With funding from both USAID and the Investment Climate Facility, the Local Sourcing Partnership provided training on HACCP certification for 480 SMEs in six countries (Ethiopia, Kenya, Malawi, Rwanda, Uganda and Zambia) in Phase 1. In Phase 2, 344 SMEs received training toward HACCP certification and 13 companies received the Local Sourcing Partnership certification of recognition. Beyond training and capacity building, the Local Sourcing Program can point to actual sales resulting from its matchmaking of buyers and sellers. For example: • Kigali Farms now sells mushrooms to Nakumatt stores in Rwanda and the Zucchini Greengrocer in Nairobi; • Rwandan water bottler Inyange is now selling to Nakumatt in Kigali; and • An agreement in Zambia paved the way for a woman-led SME to sell strawberries to Taj Pamodzi Hotel. Another successful CBC sub-activity with concrete results is the COMESA Business Visa, which is designed to allow businesspeople to move freely around all COMESA countries. Rwanda is now fully implementing the business visa, and it is scheduled for piloting in five additional countries (Zambia, Mauritius, Zimbabwe, Uganda and Malawi) in 2018. Beyond that, several COMESA countries are in the process of ratifying the protocols on free movement of persons. The CBC has successfully balanced the need to remain relevant to its members — in this case, private sector actors — with the need to achieve results. The CBC managed to increase the share of its funding derived from private sector sources from a low of 20 percent of total funding to close to 40 percent as of 2017. A strong network of national-level counterparts, particularly the national chambers of commerce, has also contributed to CBC’s successes. The CBC’s relevance to its members is evidenced by their willingness to pay the USD 5,000 annual membership dues. While it is not clear whether original expected results outlined in IL 11 have been met, the CBC has achieved more important results. USAID’s Comparative Advantage The section above describes the seven activities studied in terms of their effectiveness, relevance, and utility in promoting intra-regional trade. This section considers whether USAID as a development partner brings any particular advantage to an activity that should be considered when making funding decisions. More than half the key informants interviewed indicated that they saw USAID as having some comparative advantage in assisting COMESA vis-à-vis other development partners. Respondents most frequently mentioned this advantage aiding USAID’s work with the CBC, Trading 4 Peace, and the Biotechnology and Biosafety Initiative. USAID.GOV COMESA PERFORMANCE EVALUATION | 21 Interviewees from the private sector specifically noted that USAID is more adapted to working with the private sector than many other development partners are. The fact that USAID takes a regional approach was also mentioned; USAID was said to be relatively attuned to the need for standards, harmonized regulations and networks for buyers and sellers (i.e., the Local Sourcing Partnership). More specifically, support from USAID has allowed COMESA divisions like agriculture, industry and trade, as well as institutions like ACTESA and CBC, to interact with complementary USAID activities, such as the East Africa Trade and Investment Hub and the Africa Leadership Training and Capacity Building Program (Africa LEAD), and at times to benefit from technical assistance and technical expertise. These advantages, however, relate to all the activities studied, with the possible exception of the Statistics Program. The Coordinating Unit was designed to provide support for implementation and monitoring of activities, ensure optimal coordination with COMESA activities supported by USAID, and ensure that USAID received the data and results it required without overburdening COMESA. Achievements to date have depended largely on the effectiveness of the COMESA Secretariat, its independent agencies, and the activity managers. Development partners may have a comparative advantage simply because of their historical support or size of their contribution. USAID is the only major development partner to the Biotechnology and Biosafety Initiative, while the Agency is a new development partner to the Statistics Program, along with many other development partners. USAID used to be the sole development partner funding Trading 4 Peace, and now it is contributing much less than other development partners such as the German Development Bank, DFID, Trademark East Africa that have subsequently stepped in. However, it remains the only development partner to fund a key border crossing between Uganda and DRC and offers historical perspective as the original development partner. In summary, USAID may be particularly strong in trade promotion and private sector development compared to some development partners. However, the evaluation team did not find that USAID had a strong comparative advantage in funding some of the seven activities over others because of the technical assistance or linkages it can offer. As the only development partner that provided funding to the Biotechnology and Biosafety Initiative, USAID’s involvement was critical; its role in funding the Statistics Program was less so, given that activity’s more recent introduction. CONCLUSIONS All seven USAID-funded COMESA activities contribute to COMESA’s goals and are fully compatible with COMESA’s overall strategic objectives, as outlined in the COMESA Medium-Term Strategic Plan 2016–2020 (COMESA 2016). The evaluation team concludes that USAID should prioritize support to COMSHIP, the CBC, and the SPS Harmonization Program because they offer a combination of effectiveness, relevance, and potential to facilitate/promote increased intra-regional trade. USAID’s comparative advantage did not affect the final prioritization. Table 3 provides a rough depiction of how the evaluation team assessed the seven activities using the criteria. Additional reasons for this recommendation are summarized in the text box that follows. “I think USAID’s approach is the best for private sector development and trade promotion.” – Member state respondent 22 | COMESA PERFORMANCE EVALUATION USAID.GOV TABLE 3. ACTIVITIES AND KEY EVALUATION CRITERIA Activity Effectiveness Direct Effect on Intra￾regional Trade Trade Promotion/Facilitation Relevance to Stakeholders Biotechnology + N N N CBC + + + + ++ + + Fertilizer N N N N SPS + + ++ ++ + + COMSHIP + + ++ + Statistics + N N N Trading 4 Peace + + + + Key: ++ = Strong Positive Effect; + = Overall Positive Effect; N = Neutral/Effect Not Found Highlighted = Recommended for Prioritization of Funding WHY THESE THREE? COMESA Business Council (CBC) • CBC’s Local Sourcing Partnership can point to enabling actual sales through matching buyers and sellers. • Sellers achieve Hazard Analysis and Critical Control Points (HACCP) status, making trade possible. COMESA Seed Harmonization Implementation Plan (COMSHIP) • The COMESA Variety Catalog provides a regional platform for increased regional trade in seed. • Encourages collaboration among seed experts at all levels. • Key part of COMESA’s planned agricultural value chain work. Sanitary and Phytosanitary (SPS) Harmonization Program • Strong support in the member states. • If reoriented around trade promotion and facilitation, it has potential to improve regional trade flows. EVALUATION QUESTION 2 To what extent are USAID-funded activities successfully leveraging, rather than duplicating, efforts funded by other development partners or stakeholders toward the same results? Describe the areas of current and potential leveraging and duplication related to the specified activities and expected results. FINDINGS The evaluation team defined “leveraging” to mean that USAID funding of an activity allowed COMESA to attract additional sources of support. As Table 4 shows, other development partners support five of the seven COMESA activities (excluding the Fertilizer Initiative and the Biotechnology and Biosafety Initiative). USAID-funded activities are successfully leveraging efforts funded by other development partners or stakeholders with minimal duplication of effort. Based on information provided by the COMESA activity managers, USAID support for both the SPS and Trading 4 Peace programs (see Figure 1) has led USAID’s support to COMESA has leveraged more than USD $20 million in funding from other Development Partners. USAID.GOV COMESA PERFORMANCE EVALUATION | 23 directly to other development partners subsequently contributing about USD $10 million to each program. Examples of leveraging are summarized below. ● The United Kingdom’s Department for International Development (DFID) joined efforts to support COMSHIP, along with Monsanto, Zamseed, MRI Syngenta, the Alliance for a Green Revolution in Africa (AGRA), and the Scaling Seeds and Technologies Partnership in Africa (SSTP).1 ● Following initial European Union (EU) funding for SPS, USAID funding was leveraged to gain subsequent support from the African Development Bank (AfDB), Australian Aid, the Food and Agriculture Organization of the United Nations (FAO), the World Trade Organization (WTO), and the U.S. Department of Agriculture (USDA). ● CBC leveraged USAID money to bring in the International Lawyers and Economists Against Poverty (ILEAP), the AfDB Korean Fund, the AfDB Tripartite Capacity Building Program, the World Bank International Climate Facility, and the Center for International Private Enterprise (CIPE), to be joined in 2018 by the UN International Trade Center. ● USAID started Trading 4 Peace; subsequently, the World Bank Trade Facilitation Project, the German Development Fund, and the EU joined the effort. Of 50 people interviewed, only one cited an example of duplication. The respondent noted that, in Trading 4 Peace, the USAID program and the World Bank’s Great Lakes Trade Facilitation Project, funded by the German Development Bank, conducted the same activities at the Uganda border. Perhaps the rarity of duplication is because, as one key informant at COMESA noted, “When a new partner comes on board, we try to ensure it is value-adding, not overlap.” Five of the seven activities (excluding the Fertilizer and Biotechnology and Biosafety Initiatives) receive support from numerous other development partners. A determinant of success is whether COMESA activity managers collaboratively seek other sources of finance. The SPS and Trading 4 Peace activities have leveraged an additional USD $10 million from other development partners. Figure 1 shows the success of leveraging on the SPS program. Many respondents in the member states highlighted the effective lobbying efforts of the SPS activity manager to raise development partner funding for program activities. 1 USAID/LEO 2016. 24 | COMESA PERFORMANCE EVALUATION USAID.GOV FIGURE 1: NON-USAID FUNDS CONTRIBUTED TO SPS * An estimate of in-kind support; USDA has provided technical assistance to the SPS activities. USAID began funding the COMESA Statistics Program in 2017, after many other development partners had already been providing funding. For the Biotechnology and Biosafety Initiative, only the Bill & Melinda Gates Foundation has provided support in addition to USAID, and only to the regional biotechnology body located in Uganda. Table 4 highlights other development partners funding activities in addition to USAID, based on key informant interviews and reviews of reports. TABLE 4: DEVELOPMENT PARTNER PARTICIPATION IN THE SEVEN ACTIVITIES Activity COMESA Level Ethiopia Rwanda Uganda ACTESA Programs Fertilizer AfDB, FAO, EU, African Agribusiness Partnership None None Seed ISSD (Netherlands), DFID and AfDB, Monsanto, Zamseed, MRI Syngenta, AGRA, SSTP (Green Power Group) FAO, ISSD, AGRA AGRA, Belgian Technical Cooperation ISSD, USAID Feed the Future, AgVerify Biotech None Gates Foundation Gates Foundation COMESA Secretariat Programs SPS EU, FAO, Australia Aid, AfDB, WTO STDF, USDA Australia Aid, World Bank, Netherlands Embassy AGRA, TMEA and WTO USDA (Tuskegee University), World Bank, TMEA COMESA Business Council ILEAP, AfDB Korean Fund, CIPE, ITC, ICF, AfDB Tripartite Capacity Building NIL TMEA, World Bank, EU World Bank, EU and DFID USAID.GOV COMESA PERFORMANCE EVALUATION | 25 Activity COMESA Level Ethiopia Rwanda Uganda Trading 4 Peace International Alert, German Development Bank, AfDB, World Bank, and EU (starting in 2018 covering borders not covered by World Bank) N/A TMEA, World Bank, EU TMEA, World Bank, EU COMESA Statistics EU and AfDB DFID, U.S. Census Bureau, EU, World Bank, UNFPA, AfDB EU, IMF and UN Statistics East African Community Secretariat, TMEA, UNCTAD Note: AfDB = African Development Bank; AGRA = Alliance for a Green Revolution in Africa; CIPE = Center for International Private Enterprise; DFID = Department for International Development (U.K.); FAO = Food and Agriculture Organization (of the UN); IFC = International Finance Corporation; ILEAP = International Lawyers and Economists Against Poverty; IMF = International Monetary Fund; ISSD Africa = Integrated Seed Sector Development; ITC = International Trade Center; SPS = Sanitary and Phytosanitary; STDF = Standards and Trade Development Facility (of the World Trade Organization); TMEA = TradeMark East Africa; UNCTAD = United Nations Conference on Trade and Development; UNFPA = United Nations Population Fund; WTO = World Trade Organization. One COMESA activity manager made a distinction between “soft” technical support, such as U.S. assistance for policy development and informational materials, and “hard” support more typically provided by development partners like the EU, for instance, provision of computers and laboratory equipment. This distinction is one factor in avoiding duplication. More importantly, the EU uses five-year contribution agreements that have helped improve predictability and ownership, while USAID support is seen as less predictable and focused on “soft” issues that are difficult to directly attribute to results or to quantify. CONCLUSIONS COMESA has used USAID resources to address funding gaps, but not to duplicate activities. COMESA applies a scaling up approach. USAID funds are occasionally used to scale up an intervention to other locations/member states. For many programs, such as Trading 4 Peace, COMSHIP, and the SPS Initiative, there is no duplication of resources in the activities financed by different development partners, but COMESA can use other development partner funds to scale up the same intervention in another location. The evaluation team found that USAID support to COMESA has leveraged well over USD 20 million in funds from other development partners. For four of the seven activities studied, USAID funding leveraged substantial support from other development partners and stakeholders. The fact that COMESA activity managers are actively seeking other sources of development partner support or are self-financing is a positive sign. The 19 COMESA countries allow plenty of space for development partners to work without duplication. 26 | COMESA PERFORMANCE EVALUATION USAID.GOV EVALUATION QUESTION 3 Describe the steps COMESA has taken to coordinate development partner support for the specified activities and results. FINDINGS Multiple donor development partner coordination is a challenge for any complex organization. For COMESA, the challenge is amplified by its heavy dependence on development partner funding. In practice, coordination of development partners is often driven by individual activity managers rather than COMESA’s formal structures, and focus primarily on resource mobilization rather than joint planning, reporting, or collaboration with development partners. However, COMESA has taken some positive steps to improve resource mobilization and international cooperation and development partner coordination. Before 2015, the COMESA official responsible for resource mobilization and development partner coordination was in the Secretary General’s Cabinet. A 2015 Council of Ministers decision established the Chief for Technical Cooperation and Resource Mobilization, who is now a unit head reporting directly to the COMESA Secretary General. As a result of the 2015 decision, COMESA has taken the following steps: • Setting up of a Resource Mobilization and Technical Cooperation Unit under the Secretary General, • Mandating that all development partner-funded program activities coordinate with this unit, 2 and • Holding a development partner roundtable in November 2016. Notably, all DP coordinating units (e.g., the EU’s Regional Integration Support Program) are now operating and coordinated under the Resource Mobilization and International Cooperation (RMIC) Unit, except for the USAID Coordinating Unit, a fact that USAID/KEA respondents were previously unaware of. With the advent of the new mechanism, the USAID Coordinating Unit now operates outside the RMIC Unit, reporting directly to the Secretary General. The program managers and monitoring and evaluation (M&E) staff for the EU-funded programs are all within the RMIC Unit and are fully functioning. The extent to which COMESA relies on DP support is remarkable; 60 to 70 percent of the annual COMESA budget comes from outside assistance. Development partners fund more than 50 percent of staff positions (e.g., four of seven senior staff in the Agriculture and Industry Division). Typically, a regional intergovernmental organization would not need to rely so much on outside support. However, COMESA countries are not paying their membership dues on time. As of 2017, 16 of 19 member states were in arrears, and COMESA had run down its reserves to cover the shortfall in funding. 2 The mandate states: “The Aid for Trade, Brussels Liaison Office, RISP [Regional Integration Support Program] and USAID (and any other development partner-funded program) will coordinate their activities with the Technical Cooperation and Resource Mobilization Unit under the overall supervision of the Secretary General. The coordinator, Technical Cooperation and Resource Mobilization, through various program managers, will ensure that there is continuous dialogue and cooperation between COMESA and the cooperating partners and will support the design, implementation and monitoring of these programs.” USAID.GOV COMESA PERFORMANCE EVALUATION | 27 CONCLUSIONS Agreement was widespread within the COMESA Secretariat, USAID/KEA, and the member states that COMESA’s system for development partner coordination and resource mobilization needs improvement. Such heavy reliance on development partner support requires strong oversight, coordination, and engagement with the contributing development partners and potential new development partners. Some of the pitfalls of multiple-source funding is that the activity managers spend so much time interacting with development partners and fulfilling their reporting and monitoring requirements that they have little time left to carry out their normal technical work. This approach also entails challenges that include varying stakeholder expectations, unaligned grant funding cycles, and highly variable reporting requirements that place a heavy administrative burden on the beneficiary organization. Despite some progress, COMESA’s steps to ensure an adequate system for coordinating development partner support and overall resource mobilization have been insufficient, including collection of membership dues from member states. In addition to raising questions about member state ownership of COMESA’s agenda, this reliance on development partners also affects the sustainability of COMESA programs (many of which are almost completely reliant on development partner funding) and its ability to build institutional capacity. EVALUATION QUESTION 4 To what extent are COMESA’s management systems and organizational structures conducive to achieving the expected results of the specified USAID-funded activities? FINDINGS USAID’s funding to COMESA was linked to specific activities and expected results. In practice, USAID funding did go to the specified activities, but the expected results listed in the ILs were not often used to guide programming or reporting. This section considers COMESA’s overall management systems and organizational structures as well as those of the USAID Coordinating Unit. General Management Systems and Organizational Structures COMESA faces serious challenges that affect its capacity to operate efficiently and implement its activities, including those funded by USAID. While the COMESA Council of Ministers has agreed on reforms that might address these challenges, they have yet to be fully implemented. The COMESA Council of Ministers made a decision in December 2015 to restructure the COMESA Secretariat. The centerpiece of the new structure was the establishment of the Executive Support Division that would incorporate the Resource Mobilisation and International Cooperation Unit, the Research, Planning and Policy Harmonisation Unit, the Performance M&E Unit and the Statistics Unit. The evaluation team highlighted a few important issues to consider: • COMESA member states are not paying their membership dues on time. • Development partner support for the current fiscal year has decreased impacting greatly on the COMESA Budget. • Stakeholder dissatisfaction appears to be significant. • The COMESA Secretariat is experiencing high staff turnover. 28 | COMESA PERFORMANCE EVALUATION USAID.GOV • The current Secretary General’s term will expire in 2018, and term limits will require the selection and appointment of a new Secretary General. The combination of a decrease in development partner support and a shortfall in membership dues represents a budget crisis that puts COMESA’s work at risk. Figure 2 shows that COMESA has historically relied heavily on external funding. The situation is worse today, as only three of the 19 member states have paid their membership dues in full. Key respondents at the COMESA Secretariat expect that member states will eventually make up the arrears — no member state is more than three years behind in payment of dues. However, as it was also relayed by respondents, for many member states, refusal to pay membership dues reflects their dissatisfaction. Common complaints included COMESA’s lack of engagement with member states in prioritizing its agenda and its poor stewardship of resources. Some COMESA member state respondents (particularly government officials) said that COMESA tends not to adequately involve them in planning its activities. That is, rather than soliciting input on content in an interactive and ongoing fashion, COMESA tends to inform them once the activities have already been formulated. Respondents pointed out that although the activities added value, the results would be more effective if COMESA ensured buy-in and inclusion of members states in the process of program formulation. Nineteen key informants across the five visited countries cited insufficient outreach as an issue. One senior government official in Rwanda stated, “COMESA is not focused sufficiently on results. They like to organize meetings and sometimes lack a seriousness of purpose.” FIGURE 2: CONTRIBUTIONS TO THE COMESA BUDGET, 2007–2013 Sources: Delegation of the European Union to the Republic of Zambia and COMESA (2015); ECDPM (2016). Respondents within COMESA noted another challenge: an imminent budget crisis. In 2016 and 2017, to make up for the budget shortfall caused by arrears, COMESA used financial reserves it had accumulated over more than a decade to fund its general operating budget. According to one senior COMESA official interviewed in late November 2017, those reserves have declined substantially. USAID.GOV COMESA PERFORMANCE EVALUATION | 29 During the opening of the 37th Meeting of the Committee on Administration and Budgetary Matters, Mr. Ngwenya, the COMESA Secretary General, stated that, “the annual budget will decrease from USD 42 million in 2017 to USD 32 million for 2018. This translates into a 30 percent reduction.” He added that member states are expected to contribute USD 16.7 million, while cooperating partners are expected to provide USD 15.6 million. Relying on information provided by the Secretary General at the time of the COMESA Summit in November 2017, Figure 3 shows an expected budget decrease of about USD10 million for calendar year 2018. This is mainly attributed to an anticipated reduction in development partner contributions, even when only accounting for funding decisions already announced. 3 As a result of funding uncertainty, many COMESA staff working under contract are not having their contracts renewed, leaving uncertainty about their own futures and those of their activities. Within the unit responsible for development partner coordination, only one of five positions is permanent; the other four are funded by development partners and subject to funding availability. In calendar year 2017, USAID was supporting 21 full-time positions, including six within the USAID Coordinating Unit.4 COMESA respondents noted that USAID’s one-year funding cycle causes particular difficulties for planning. Other development partners, like the EU, provide more dependable, multi-year funding. One senior COMESA official noted that activity managers reliant on development partner funding are “negotiating their own survival.” While competition can be positive, highly technical staff funded by development partners unfortunately have incentives to prioritize development partner administration and program requirements over their normal COMESA technical work. In the long run, such an organizational structure harms the interests of all stakeholders and hinders sustainability of the institution. FIGURE 3: COMESA’S IMPENDING BUDGET CRISIS 3 The evaluation team assumes this figure includes the indicative amount in the RDOAG signed between USAID and COMESA in September 2016. 4 This represents a reduction from 54 full-time positions earlier in the decade. 30 | COMESA PERFORMANCE EVALUATION USAID.GOV Source: COMESA press release at time of COMESA Summit (November 2017). In addition to issues of resources, the evaluation team observed some challenges that could be addressed with different management approaches. For example, the evaluation team saw that activities achieved better results when they collaborated with complementary activities — for example, when the SPS team worked with the CBC on the HACCP training; the COMESA Statistics Program worked with the COMSHIP team on the design of the COMESA Seed Information System; or the Biotechnology and Biosafety Initiative used COMSHIP’s national-level network of seed dealers and experts. In general, however, the interviews with COMESA respondents suggest a lack of incentive for activity managers to work together. Non-COMESA respondents mentioned this lack of internal collaboration too. One member state respondent noted, “Too often we work in silos. If the people working on seeds were brought together with those working on fertilizers and those working on SPS and what you have told me about the CBC’s Local Sourcing Partnership, we could share information more easily and have a better understanding of how to help each other.” USAID Coordinating Unit Management Systems and Organizational Structures At the inception of the IPAA in 2009, COMESA set up the USAID Coordinating Unit at the Secretariat to facilitate implementation of USAID-funded activities, to ensure coordination between the Secretariat in Lusaka and USAID/East Africa in Nairobi and to facilitate required reporting. The USAID Coordinating Unit reports to the COMESA Assistant Secretary General for Programs. On the organizational chart obtained, the USAID Coordinating Unit is linked by a dotted line, indicating a coordinating relation, to the COMESA Resource Mobilization and International Cooperation Unit (see Figure 4). As of 2017, the USAID Coordinating Unit consisted of six USAID-funded positions, though the position of communications specialist remained vacant. The evaluation team was unable to obtain the terms of reference for the USAID Coordinating Unit. Thus, it examined the unit with respect to the responsibilities noted by stakeholders, including implementation support, coordination, and reporting. The evaluation team learned that USAID did not require regular reporting from COMESA as part of its agreement and often did not identify useful indicators for measuring expected results. The USAID Coordinating Unit did produce progress reports for FY 2015–2017 that contained useful information, but they do not refer to expected results and they do not provide consistent indicator reporting. The evaluation team observed that many of the original expected results and indicators were poorly conceived or worded. One expected result, for example, is “Sharing of experiences between member states.” This result cannot be evaluated because it is not clear what kind of sharing or what kind of experiences were intended, or for what purpose. Many of the indicators refer to the number of meetings or staff trained, which may be useful, but are insufficient to measure progress. After the Coordinating Unit hired an M&E specialist in 2015, the quality of the reports improved somewhat. The reports began to include some USAID standard indicators, although the set of indicators varied from year to year, preventing measurement of progress. The monitoring efforts of the USAID Coordinating Unit — modest as they are in comparison with USAID activities with typical USAID M&E requirements — are commendable in the context of an organization (COMESA) with no culture of monitoring results. COMESA appears to be making positive changes, however, by working with the EU USAID.GOV COMESA PERFORMANCE EVALUATION | 31 to establish an online M&E system. At the time of this evaluation, the online system was not yet fully operational. FIGURE 4: COMESA ORGANIZATIONAL CHART CONCLUSIONS The current budget shortfall notwithstanding, COMESA’s heavy reliance on development partner funding negatively affects its ability to chart its own course and manage accordingly. The fact that COMESA does not control such a large proportion of its resources — especially those required for core functions — hinders organizational planning and efforts to retain key staff. The result is a weakened core structure that is not optimally equipped to support or coordinate development partner activities. Respondents noted that it will be difficult to undertake significant reforms before the new Secretary General takes office. However, a change of leadership will potentially create new energy to advance the reform agenda. 32 | COMESA PERFORMANCE EVALUATION USAID.GOV The evaluation team found that USAID did not adequately specify the expected results or provide appropriate indicators or targets with which to monitor progress. For its part, the USAID Coordinating Unit could have improved reporting by using a consistent set of indicators (although this was not a requirement). Therefore, although USAID and COMESA can point to achievements of the USAID￾funded activities, they cannot demonstrate that the original expectations were met. Furthermore, the Coordination Unit did not play its coordination role, even with the other activities in the USAID portfolio. The placement of the USAID Coordinating Unit outside the newly established RMIC Unit hinders collaboration. EVALUATION QUESTION 5 To what extent can the specific USAID-funded activities be sustained through other sources of funding? FINDINGS As indicated in the previous discussion on leveraging and development partner coordination, aside from the Fertilizer Initiative and the Biotechnology and Biosafety Initiative, all activities studied have significant sources of funding beyond USAID (see Table 4). Ideally, COMESA would fund its core programs and staff positions with membership dues, and development partner support would be used to fund specific targeted activities. COMESA’s failure to collect membership dues is, in at least some part, reflective of member state dissatisfaction with the COMESA Secretariat. Member states do not see COMESA activities as relevant or reflective of their priorities. Lack of member state commitment harms the sustainability of USAID-funded activities. Table 5 outlines alternative funding prospects for each of the seven activities. The most promising in terms of self-financing are the CBC and COMSHIP. The CBC is on its way to sustainability and appears to be increasing its reach and effectiveness, having significantly increased the share of its budget represented by self-financing from 2016 to 2017 (see Table 5). USAID support has been integral to its growth and effectiveness. COMSHIP has the chance to become self-financing, but needs greater outreach to the private sector. The Fertilizer Initiative has failed in its effort to self-finance. TABLE 5: ALTERNATIVE FUNDING POSSIBILITIES FOR EACH ACTIVITY Activity Prospects for Alternative Sources of Funding Biotechnology and Biosafety Initiative No plan for self-financing. Possible migration to the Gates Foundation? Fertilizer Initiative Bulk purchasing plan to raise revenue for operations did not succeed. No other development partner funding at present. COMSHIP Some revenue raised from registration of 50 varieties in the COMESA Seed Variety Catalog. Significant additional revenue expected from greatly increased number of varieties registered and from COMESA’s share of sales of COMESA labels. Two years away from self-financing? Possibility of other DP support. COMESA Business Council Plan for 80 percent self-financing by 2022. Twenty percent of CEO’s salary and all six staff members’ salaries covered by self-financing. SPS Initiative Multiple other development partners have participated. Energetic activity manager seems skilled at raising additional funds. USAID.GOV COMESA PERFORMANCE EVALUATION | 33 Activity Prospects for Alternative Sources of Funding Statistics Program Multiple other development partners have financed discrete activities. Self-financing not viable. Trading 4 Peace Multiple other development partners; no prospect for self-financing, given that COMESA Simplified Trade Regime aims to reduce costs for small-scale, cross￾border traders. CONCLUSIONS The USAID-funded activities studied have varying prospects for sustaining themselves through other sources of development partner funding, self-financing, or a combination. The prospects are least positive for the Fertilizer Initiative and Biotechnology and Biosafety Initiative, as they have neither self￾financing options nor significant development partner support. The other activities have support from other development partners, and a few generate their own resources. The evaluation team did not find that any of the activities studied could absorb a significant reduction in funding without adjusting their near-term work plans accordingly. RECOMMENDATIONS 1. USAID Should Use a Milestone-Based Funding Mechanism USAID should use a milestone-based funding mechanism rather than the present activity-based funding mechanism. A milestone-based approach would require COMESA to achieve agreed-upon results. USAID should recognize that COMESA operates in a fluid environment. USAID and COMESA should review the terms of agreement periodically and adapt according to information learned or opportunities identified. COMESA reporting can aid in this process by documenting challenges and proposed solutions. 2. USAID Should Prioritize Funding to the CBC, COMSHIP and the SPS Initiative The evaluation team recommends that USAID continue funding the CBC, COMSHIP, and the SPS Initiative. If USAID is able to fund additional activities, the team recommends giving consideration to Trading 4 Peace and the Statistics Program (in support of USAID-funded activities), as they were also found to be effective. 3. USAID Should Fund the Most Productive Sub-Activities Rather than funding the entire portfolio of sub-activities in an activity, USAID should consider choosing the most effective, trade-inducing sub-activities. The evaluation team recommends the following: • COMSHIP: Address lack of awareness of the Variety Catalogue by targeting seed dealers for sensitization and training on how it works and the potential for increasing profits. • SPS Initiative: Support Pest Risk Analysis and Regional Pest Lists. • CBC: Expand the range of countries involved in the Local Sourcing Partnership. • Support the Statistics Program as a sub-activity of COMSHIP, SPS, and CBC to provide targeted support to each activity. 34 | COMESA PERFORMANCE EVALUATION USAID.GOV USAID might also consider funding sub-activities that oblige COMESA’s activity managers to collaborate, such as the COMESA Seed Information System (COMSIS) or Statistics Program work that supports other activities. 4. Prioritize Training and Provide Staff Incentives The COMESA Secretariat should consider offering its staff training on resource coordination. This might include techniques for identifying sources of potential funding, facilitating collaboration, and leveraging resources. COMESA should consider annual prizes for staff members who demonstrate innovative techniques. Furthermore, COMESA should prioritize training and technical capacity building over facilitating meetings. 5. COMESA Should Focus on Improving Development Partner Coordination COMESA’s Resource Mobilization and International Cooperation Unit, which is tasked with development partner coordination, should adopt a multi-tiered approach, as Table 6 outlines. 5 TABLE 6: SUGGESTED ACTIONS TO IMPROVE COMESA’S DEVELOPMENT PARTNER COORDINATION Short-Term Medium-Term Long-Term a) Integrate the USAID Coordinating Unit into the RMIC Unit b) Undertake needs assessment related to development partner coordination Seek technical assistance for the RMIC Unit Develop new strategy for engagement with DPs and the private sector beyond a focus on resource mobilization to promote joint planning, reporting, and collaboration with DPs 6. Reposition and Restructure the USAID Coordinating Unit The COMESA Secretariat should integrate the USAID Coordinating Unit into the COMESA RMIC Unit to improve coordination and collaboration, and to enhance program management. USAID should consider supporting a few critical positions focused on USAID programs, such as an accountant and an M&E specialist, who would support the head of the RMIC Unit in coordinating USAID-funded activities. In this way, the USAID-supported positions would conform with the terms of the 2015 decision of the COMESA Council of Ministers while also reducing costs, increasing visibility and streamlining outreach to member states. 7. Retrench and Plan COMESA Secretariat should focus on efficiency by fully implementing the 2015 Council of Ministers decision on organizational restructuring and reform, reducing its expenditures and concentrating on its core mission. USAID should consider supporting the COMESA Secretariat in this process of institutional 5 Note that the development of a new long-term strategy will require waiting until the appointment and installment of a new COMESA Secretary General in 2018, so as not to “tie the hands” of that person. USAID.GOV COMESA PERFORMANCE EVALUATION | 35 strengthening, specifically focusing on the development of an effective M&E systems and establishment of the Executive Support Division. 8. Improve Monitoring and Evaluation The COMESA Secretariat should strengthen its M&E system. This will require training activity managers and enlisting the support of senior leadership to encourage a culture of data utilization. USAID should ensure that future agreements feature well-designed indicators and targets and well-defined reporting requirements. USAID staff should review COMESA’s written reports thoroughly to ensure that progress is tracked appropriately. 36 | COMESA PERFORMANCE EVALUATION USAID.GOV ANNEX I: BIBLIOGRAPHY ACTESA (2016). “Second COMSHIP Mutual Accountability Framework Convening Report.” September. AFRICA LEAD (2017). “Africa Lead Quarterly Report East Africa Mission – Q2.” July. AfDB (African Development Bank)/CBC (COMESA Business Council) (2017). “Tripartite Local Sourcing for Partnerships Project - Training and Capacity Building Workshops for the SMEs on Food Safety and Standards.” June. COMESA (2009 - 2017). “IPAA Annual Reports and Progress Reports.” COMESA (2017). “RDAOG Proposed Programs Final.” November. COMESA (2016). “Medium-Term Strategic Plan 2016 - 2020.” October. COMESA (2014a). “COMESA Seed Harmonization Implementation Plan (COMSHIP).” July. COMESA (2014b). “COMESA Seed Trade Harmonization Regulations.” COMESA Secretariat Press Release, "Proposed COMESA 2018 Annual Budget Drops". Lusaka, Thursday, October 26, 2017. Accessed online: http://www.comesa.int/press-release/ CBC (2016). “CBC Biennial Institutional Report.” Daily Monitor (2017). “End of Development partner Grants to Eat into COMESA Annual Budget”. November 1. Uganda Daily Mail (2017). “Farmers Should Look Out for Fake Seeds.” October 14. Zambia. Daily Nation (2017a). “Dar, Nairobi Should Resolve Differences.” November 9. Kenya. Daily Nation (2017b). “Magufuli Digs in on Cows as Amina Pushes for Talks.” November 9. Kenya. Tralac E-Newsletter (2015). “Trading for Peace Initiative.” March 30. South Africa. ECDPM (European Center for Development Policy Management) (2016). “The Political Economy of Regional Integration in Africa (COMESA) Report.” January. FAO (Food and Agriculture Organization of the UN) (2012). “Legal Study on the Regulations on the Application of Sanitary and Phytosanitary Measures of the Common Market for Eastern and Southern Africa.” June. Private Sector Federation (2017). “Beyond Advocacy…” Rwanda. USAID/AFRICA LEAD (2012). “ACTESA/COMESA Organizational Capacity Assessment.” February 7. USAID (2012). “How-to Note: Preparing Evaluation Reports.” November. USAID.GOV COMESA PERFORMANCE EVALUATION | 37 USAID/COMESA (2016). “Regional Development Objectives Grant Agreement.” September. USAID/COMESA (2009-2017). “Integrated Partnership Assistance Agreement Implementation Letters Nos. 1 to 9.” USAID/KEA (USAID/Kenya and East Africa) (2016). “East Africa Regional Development Cooperation Strategy 2016–2021.” USAID/KEA (2016). “Report on the Organizational Risk Assessments of the Common Market for Eastern and Southern Africa.” Prepared by Deloitte. April. USAID/LEO (2016). “Evaluation of Sanitary and Phytosanitary (SPS) Trade Policy Constraints within the Maize, Soya and Groundnut Value Chains in Southern Africa.” Leveraging Economic Opportunities Report #35. 38 | COMESA PERFORMANCE EVALUATION USAID.GOV U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523