August 29, 2011 This publication was produced by Amex International and its Subcontractor, DevTech Systems, Inc., for the United States Agency for International Development. It was prepared by Phil Church, DevTech Systems, Inc and Cristian Rodriguez, AMEX International under Contract No. RAN-I-00-09-00008-00. EVALUATION USAID/Dominican Republic Tri-Project Performance Evaluation Dominican Republic – Central American Free Trade Agreement Implementation Project (DR-CAFTA IP) August 29 2011 This publication was produced by Amex International and its Subcontractor, DevTech Systems, Inc., for the United States Agency for International Development. It was prepared by Phil Church, DevTech Systems, Inc and Cristian Rodriguez, AMEX International under Contract No. RAN-I-00-09-00008-00. USAID/DOMINICAN REPUBLIC TRI-PROJECT PERFORMANCE EVALUATION DOMINICAN REPUBLIC - CENTRAL AMERICAN FREE TRADE AGREEMENT IMPLEMENTATION PROJECT (DR￾CAFTA IP) USAID/DR TRI-PROJECT EVALUATION: ECONOMIC GROWTH PROGRAM VOLUME I: DR-CAFTA IMPLEMENTATION PROJECT DISCLAIMER: The contents of the following report are the responsibility of the authors and represent our best efforts to complete the SOW given. The opinions expressed herein are those of the authors only, we accept responsibility for their accuracy and in no way do these opinions represent the policy of USAID, The Department of State, or the Government of the United States of America. CONTENTS EXECUTIVE SUMMARY ............................................................ Error! Bookmark not defined. Findings: Achievement of Results ........................................................................................................ 1 Findings: Project Design and Implementation Strengths and Weaknesses ........................................ 3 Conclusions and Recommendations.................................................................................................... 4 Lessons Learned & Best Practices ...................................................................................................... 6 1. INTRODUCTION................................................................................................................ 7 1.1 Background of the DR-CAFTA Implementation Project................................................................. 7 1.2 Purpose of the Evaluation .............................................................................................................. 8 2. EVALUATION METHODOLOGY....................................................................................... 9 2.1 The USAID/DR-CAFTA IP Development Hypothesis and Results Framework ............................. 9 2.2 Mixed-Methods Evaluation Approach to Assessing DR-CAFTA IP Impact ................................. 11 3. EVALUATION FINDINGS.................................................................................................15 3.1 Project Achievements (Results) ...................................................................................................15 3.2 Project Design and Implementation .............................................................................................21 4. CONCLUSIONS AND RECOMMENDATIONS .................................................................25 4.1 USAID Project Administration ......................................................................................................25 4.2 DR-CAFTA IP Implementation and Monitoring ............................................................................26 4.3 Mission Strategic Planning & New Initiatives ...............................................................................28 5. LESSONS LEARNED AND BEST PRACTICES ..............................................................29 ANNEXES TABLE OF FIGURES Figure 1: DR-CAFTA IP Results Framework ............................................................................11 Figure 2: Organizational Development Status Scores ............................................................12 Figure 3: Institutional Capacity Status for DR-CAFTA Implementation................................. 15 Figure 4: Studies and Events Sponsored by DR-CAFTA IP....................................................16 ACRONYMS AMCHAMDR American-Dominican Chamber of Commerce and Industry CEI-RD Centro de Exportación e Inversión de la Republican Dominicana CENIT Consejo Empresarial para la Negociación e Implementación de Tratados CNC National Competitiveness Council (Acronym for Spanish title) CODOPYME Dominican Confederation of Small and Medium Enterprises (Spanish acronym) CONEP National Council of Private Business (Acronym for Spanish title) CPP Competitiveness and Policy Program DGA Dirección General de Aduanas DIA Departamento de Inocuidad Agroalimentaria (del MAG) DICOEX Directorate of Foreign Trade (Acronym for the Spanish title) DSTA Dominican Sustainable Tourism Alliance Project (of USAID/DR) DIGENOR Dirección General de Normas (del MIC) DCRAB Departamento de Control de Riesgos en Alimentos y Bebidas del MSPAS DR-CAFTA Central America and Dominican Republic Free Trade Agreement with the US DR-CAFTA IP USAID DR-CAFTA Implementation Project EPA Economic Partnership Alliance of the EU, DR and Caribbean countries EU European Union FAS Foreign Agriculture Service (of the USDA) FTF Feed the Future Initiative of the US President GBTI/IQC Global Business Technology Initiative Indefinite Quantity Contract GCC Global Climate Change Initiative DGCP Dirección General de Contrataciones Públicas of the Ministry of Finance GODR Government of the Dominican Republic CPP Competitiveness and Policy Project (of USAID/DR) INDOTEL Instituto Dominicano de las Telecomunicaciones INTEC Instituto Tecnológico de Sto. Domingo ISPRI Institutional Support Project of the European Union IR Intermediate Result JAD Junta Agro-Empresarial Dominicana MAG Ministry of Agriculture and Livestock MIC Ministry of Industry and Commerce MNRA Ministry of Natural Resources and the Environment MSPAS Ministry of Health and Public Assistance (also ‘MS’ in many GODR documents) MTrab Ministry of Labor ONAPI Oficina Nacional de la Propiedad Industrial (del MIC) ONDA Oficina Nacional de Derechos de Autor (del MIC) OTCA Oficina de Tratados Comerciales Agrícolas (del MAG) PBMS Performance-Based Management System PPL/LER USAID Policy, Planning and Learning office of the Learning, Evaluation and Research Bureau PROCON Institute for Consumer Protection RED Rural Economic Diversification Project (of USAID/DR) ROO Rules of Origin SME Small and Micro-Enterprises SO Strategic Objective SPS Sanitary and Phytosanitary Standards TO Task Order TRQ Tariff Rate Quotas USAID United States Agency for International Development USDA United States Department of Agriculture USPTO United States Patents and Trademarks Office USTR United States Trade Representative (Office of the President, USG) WTO World Trade Organization 1 EXECUTIVE SUMMARY This performance evaluation was completed for the United States Agency for International Development (USAID) Mission in the Dominican Republic (DR) under the USAID Evaluation Services IQC. This report summarizes the findings of the evaluation of the Dominican Republic and Central America Free Trade Agreement Implementation Project (DR-CAFTA IP). Throughout the report the DR-CAFTA IP may also be referred to as “the Project.” Per the Evaluation Scope of Work (see Annex A) the purpose of the evaluation was to: • Document and assess the Project’s progress to date toward achieving its results targets of improved GODR capacity for DR-CAFTA implementation and enhanced public-private dialogue about the trade treaty’s benefits and challenges; • Examine appropriateness for additional USAID/DR short-term (Option Year 2) support for further advancement toward DR-CAFTA IP results targets; and • Offer recommendations – based on DR-CAFTA IP findings, conclusions and lessons learned – to the Mission’s long-term strategic planning process for future development assistance to the Dominican Republic. The evaluation applied a mixed-methods approach to assess how effectively the DR-CAFTA Implementation Project achieved its objectives of: a) building public sector capacity for enforcing DR-CAFTA treaty terms and conditions in the DR; and, b) fostering public-private dialogue about the risks and benefits of DR-CAFTA participation. The mixed-methods approach drew on four types of data sources: 1) documentation and indicator data contained in the Performance Monitoring Plan (PMP) of the Project’s Performance-Based Management System; 2) direct interviews with key informants from the public and private sectors; 3) event evaluations of beneficiaries who participated in Project-sponsored programs; 4) secondary data on Dominican trade and development. To examine this data, the Evaluation employed “Knowledge, Attitudes and Practices” (KAP) and Organizational Development (OD) analysis tools. The DR-CAFTA IP-sponsored workshop, seminar and training events, conducted studies and provided technical advisory assistance from subject matter specialists to improve institutional capacity and enhance public-private sector dialogue. Also of importance was the Project’s role in coordinating with – and in the case of USG agencies providing logistics for – other donor organizations – notably the European Union, IDB, IBRD, IICA – in building Government of Dominican Republic (GODR) capacity to implement its Treaty commitments. The DR-CAFTA IP project worked particularly closely with the Ministry of Industry and Commerce (MIC) Directorate of Foreign Trade (DICOEX- Acronym for the Spanish title) in supporting its role as coordinator and monitor GODR agency’s implementation of the DR-CAFTA. This performance evaluation examines the implementation of these DR-CAFTA IP interventions, as well as assesses their contribution toward achieving targets that USAID established for the Project. FINDINGS: ACHIEVEMENT OF RESULTS The evaluation finds that at the end of three years of implementation, the Project is meeting or exceeding the activity or output targets set in its PMP. These output targets included the number of events, number and type (sex and industry sector) of event participants, trade-related studies, and number of operational and communications tools created. These targets were set through results indicators 2-10 and 12-14 (Portman-Bingaman rural development indicators). 2 Between Project start-up in September 2007 and May 2011 prior to this evaluation, the Project funded 89 studies and conducted 125 events totaling 768 ‘training’ hours, according to Project records. Event participants ranged from fewer than 10 to over 100 from both the public and private sectors and from both sexes. Events ranged from two hours to two weeks, though most were less than a day. Achievement of outcome targets is more uneven. These outcome targets were set through results indicators 1 (institutional effectiveness) and 11 (trade readiness). The uneven achievement of these targets is due in large part to factors beyond Project control, namely assumptions related to constrained GODR fiscal capacity to fully fund and staff DR-CAFTA implementing agencies as well as the effects of the depressed US economy on demand for Dominican products. Four well-documented general findings related to the Project’s institution-building and public￾private dialogue objectives emerge from the evaluation. Specifically the Project has: • Helped the GODR get its trade legislative and regulatory house in order. Project achievements in capacity building for DR-CAFTA implementation are manifested in GODR compliance to date – in terms of accomplishing reforms and harmonization by Treaty deadlines – with all of the relevant chapters of DR-CAFTA compliance in which the Project has been involved. However, there remain some critical pockets of ignorance and unawareness among senior GODR ministry officials and legislators – in some cases in key ministries and legislative committees – about DR-CAFTA commitments and benefits; these have slowed progress toward full Treaty compliance, particularly in some critical areas such as agriculture safeguards and market access, where GODR interagency cooperation and coordination is critical. • Strengthened DICOEX as the ‘go-to’ GODR office for coordinating DR-CAFTA implementation. The Project’s effectiveness at increasing DICOEX capacity to carry out its DR-CAFTA responsibilities is best reflected in the independent ISO 9001 certification for the Directorate awarded to DICOEX in late 2009. The ISO certification underscores DICOEX advances in better trained staff and organizational structures in which the Project played a direct role. However, DICOEX still lacks three critical ‘tools’ in carrying out its Treaty DR-CAFTA coordinating responsibilities: 1) a website that is easy to navigate and is transparent for information sharing; 2) a Treaty-tracking system that is automated, transparent and accessible to the public; and 3) a systematic arrangement for engaging the private sector in trade consultations. All three are important DR-CAFTA IP outcomes yet to be fully obtained. • Helped foster structural adjustments in the Dominican economy that have gone beyond basic trade reform to include changes in competition policy that have lowered the costs of doing business and the costs of living for domestic producers and consumers alike. One example is cell phone number portability which has increased competitiveness among mobile phone service providers with lower service cost advantages for producers and consumers alike. Still, much work lies ahead as the country moves into periods of reduced protection for several sensitive (mostly agricultural) commodities where safeguards are soon to be removed – dairy products, meat, and grains. • Cultivated better private sector recognition of new rules for trade and investment. Key informant interviews and event participant surveys document well that private sector firms are much more cognizant of DR-CAFTA as a ‘game-changing’ reality for doing 3 business in the DR today, both for the domestic economy and for export. An on-going task for the Project is fostering mechanisms for periodic, substantive two-way consultative dialogue between GODR agencies and their private industry and trade counterparts. FINDINGS: PROJECT DESIGN AND IMPLEMENTATION STRENGTHS AND WEAKNESSES The evaluation obtained the following findings regarding project design and implementation strengths and weaknesses: Strengths: • USAID support through DR-CAFTA IP was critical to sustaining GODR Treaty implementation following DR-CAFTA negotiations supported by its predecessor USAID Competitiveness and Policy Program (CPP). For example, the Project has helped both sustain momentum for DR-CAFTA compliance coordination despite changes in MIC leadership at the minister, vice-minister and directorate levels. • The Project gave DICOEX and other GODR entities with DR-CAFTA implementation responsibilities – MAG/OTCA, MIC/ONAPI, INDOTEL, MSP and Hacienda/DGCP among others – access to competent consultant talent that accelerated their capacity to meet Treaty compliance requirements and deadlines. • Project support for consultants, studies, and events has released DICOEX budget resources to focus on pressing needs to build and train staff, equip its facilities and develop outreach and communications mechanisms, and track Treaty compliance through the GODR legislative and regulatory processes. • Project-sponsored awareness and training events have reached a broad spectrum of audiences – both sexes, across industry and commodity sectors and geographic regions. Weaknesses: • The USAID decision to design two ‘Option Years’ into the DR-CAFTA IP project has proven to be disruptive to implementation momentum. Uncertainties over USAID decisions to exercise option year extensions and USAID constraints on contractual commitments beyond approved funding periods have forced the DR-CAFTA IP implementer to postpone information events and to leave vacant technical staff positions unfilled at critical implementation periods. • USAID branding policy has constrained building DICOEX visibility and credibility. The required prominence of USAID logos has given the Directorate ‘second billing’ on publications, banners and slide show presentations at USAID (DR-CAFTA IP) funded information sharing events. • USAID could better lead and administer coordination between DR-CAFTA IP activities and other USAID Economic Growth (EG) office projects – particularly the Rural Economic Diversification (RED) Project. Lack of coordination and follow-up between projects has limited the spread of DR-CAFTA compliance benefits to USAID/RED target rural and low-income populations. • USAID introduced (or approved) the use of activity and output level indicators as proxy measures of outcomes. For example, the number of workshops, studies and 4 participants are poor measures of institution building and changes in practices when other means exist to better serve the Project’s measurement of results achievement. • The Project has made only limited use of its event evaluation process for planning, follow-up and feedback for the GODR and the private sector, which has limited its planning and capacity to follow up activities with initiatives to sustain capacity building momentum started during initial training and awareness raising efforts. • The DR-CAFTA IP team’s central role in events management activities appears to have fostered DICOEX dependency on the Project as its administrative arm. • Project-funded support to DICOEX website development has yet to succeed in designing an Internet presence that is easy to navigate and transparent in its information content. • The Project has yet to see returns on its support to DICOEX for establishing a sustainable private sector consultation and advisory system for soliciting feedback from firms and businesses on pending DR-CAFTA compliance draft legislation and regulations, or on sharing information about DR-CAFTA developments with trade association memberships. CONCLUSIONS AND RECOMMENDATIONS The evaluation has formulated fourteen major conclusions and corresponding recommendations from its DR-CAFTA IP design and implementation findings. These are organized into three groups as outlined below. USAID PROJECT ADMINISTRATION 1. USAID Option Year Policy. Conclusion: Delays in administering option years have detracted from effectiveness of project operations. Recommendation: Determine immediately if USAID will exercise its option to fund DR-CAFTA IP for a fifth year and if so promptly sign a fifth year contract with the DR-CAFTA IP implementing partner so that work can continue – with what course corrections should be made – without disruption. 2. USAID Branding Policy. Conclusion: USAID/DR can use Project resources more effectively if it exercises USAID branding policy waivers appropriately. Recommendation: Seek a limited waiver of USAID branding where it interferes with the Project’s capacity building. 3. USAID DR-CAFTA and USAID/RED Project Integration. Conclusion: Closer and more productive integration of efforts by DR-CAFTA IP and USAID/RED could have resulted from more pro-active USAID project design and management. Recommendation: USAID should convene the DR-CAFTA IP and USAID/RED Team leadership to develop, along with the COTRs, a mutually reinforcing work plan for the remaining periods of project implementation. One or two common indicators toward which both projects would make a contribution should be included in the plan. DR-CAFTA IP IMPLEMENTATION 4. USAID PMP Output and Outcome Results Indicator Measurement. Conclusion: Several Project PMP indicators have limited use in measuring results. Recommendations: De-emphasize activity-level indicators; introduce fewer, more relevant measures of Project performance – e.g., ‘customer satisfaction sample surveys’ 5 to assess impact of Project-sponsored events on participants’ knowledge, attitudes and practices. An example of a proposed performance indicator is included in Annex E. 5. Future DR-CAFTA Implementation Support Focus. Conclusion: GODR treaty implementation, while uneven among the Treaty’s 22 chapters, has advanced enough that the Project can focus on those chapters with greatest deficiencies and/or rationale per USAID’s strategic objectives. Recommendation: The Project should immediately review with USAID and where appropriate modify its criteria, set priorities and on that basis allocate its limited Option Year 2 DR-CAFTA implementation support resources to a smaller number of Treaty chapters and GODR regulatory and enforcement agencies. Those criteria might include those Treaty chapters that would have the greatest impact on USAID’s rural poor target populations – e.g. market access and SPS – or on USAID’s evolving strategic plans for future USG development assistance to the DR – e.g., in the environment and food security. Other important criteria include GODR capacity to ‘got it alone’ now after DR-CAFTA IP support or engage other donor support to pick up where the Project can leave off. 6. Private Sector Communications and Dialogue. Conclusion: There is underutilized capacity and goodwill among private trade/producer associations for getting out DR￾CAFTA messages to private enterprises. Recommendation: Help DICOEX work with trade/producer associations on getting critical DR-CAFTA messages to their memberships, largely by packaging messages for delivery through association electronic and print media. Also support DICOEX and key private sector organizations in putting a sustainable trade consultative mechanism in place for obtaining timely feedback. 7. DR-CAFTA Orientation for GODR Legislators and Senior Ministry Officials. Conclusion: Senior GODR officials are not equally aware of their unit’s DR-CAFTA compliance implementation responsibilities. Recommendation: Develop and implement a strategy for briefing high-level GODR legislators and administrators – perhaps political candidates and legislators as well – on DR-CAFTA issues. 8. DR-CAFTA IP Awareness and Informational Events. Conclusion: The wide range of DR-CAFTA IP-sponsored event topics has challenged the Project, DICOEX and GODR implementing agencies in their ability to follow up with and support events participants seeking more help and guidance. Recommendation: The mission should direct the DR￾CAFTA IP to scale back the number of events and participants and focus future event funding on Dominican ‘change agents’ rather than on just filling seats at seminars; in all cases the Project should support but not lead DICOEX in managing these events. 9. DICOEX Website Communications. Conclusion: The DICOEX website’s deficient navigability, structure and content accessibility limit its usefulness at present as a communications tool and constrain the Directorate’s role as the GODR leading coordinator and information-sharing entity on DR-CAFTA compliance, challenges and opportunities. Recommendation: The Project should reinvigorate its work with DICOEX not only in redesigning its website but also in building integrated and innovative systems that will strengthen DICOEX’s information and communication capacity. 10. DICOEX Treaty Compliance Monitoring. Conclusion: DICOEX role in treaty monitoring remains a work in progress with limited effectiveness beyond DICOEX as 6 currently conducted by its legal department. Recommendation: Help DICOEX install and operate an electronic score-card or equivalent tool for treaty compliance tracking and web-based reporting with remote public access. 11. Project Participant Event Evaluations. Conclusion: The DR-CAFTA IP could improve its implementation oversight by better collecting and utilizing the participant event evaluation process to get feedback on progress made and challenges to meet in raising awareness, changing attitudes and encouraging practices to benefit from DR-CAFTA engagement. Recommendation: The process with which evaluations are solicited should be improved so that participant evaluations can be more useful inputs for Project implementation. USAID STRATEGIC PLANNING AND NEW INITIATIVES 12. Global Climate Change (GCC) Initiative. Conclusion: Energy conservation and alternative energy contribute directly to greater Dominican trade competiveness. Recommendation: Support DR-CAFTA strengthening through the GCC where treaty compliance will foster investments in green production through use of alternative energy and conservation practices. 13. Feed the Future (FtF) Initiative. Conclusion: The DR-CAFTA has direct links to USAID’s FtF Initiative through access to food – by generating jobs and incomes; also health is impacted by poor food safety standards and practices. Recommendation: Support DR-CAFTA Sanitary and Phytosanitary Standards (SPS) application strengthening for greater rural export competitiveness, jobs and incomes, and health conditions 14. Education. Conclusion: There is a broad awareness and consensus in both public and private circles that Dominican trade competiveness is constrained by lack of a technically qualified labor force. Recommendation: Focus future USAID education resources towards supporting GODR technical training programs in skill areas needed by private business. LESSONS LEARNED AND BEST PRACTICES 1. International trade treaties can be useful tools for USAID’s development assistance kit. DR-CAFTA is more than a trade agreement as are almost all regional and bilateral trade agreements under the WTO today. DR-CAFTA also calls on the Dominican Republic to adopt a number of domestic fiscal and economic reforms that internally make the country more efficient and competitive. Best practice. Design and implement economic growth development assistance programs to include compliance with appropriate trade treaty measures. 2. Avoid option years as they can disrupt project momentum. Best Practice. Contract projects for their entire planned implementation period. 3. Exercise branding waivers when appropriate as they can lead to more cost￾effective use of USG funds for institution building. USAID has a mechanism for waiving or modifying its branding policy where justified. Best Practice. Train Mission, implementing partner and counterpart staff on how to know when waivers are appropriate and in how to justify and apply them. 7 1. INTRODUCTION The Free Trade Agreement between Central America, the Dominican Republic, and the United States (DR-CAFTA) 1 In scheduling compliance with the Treaty’s 22 chapters, the DR-CAFTA contracting partners granted the GODR grace periods of one or two years where there was little or no legislation, regulation or institutional framework in place. came into effect in the DR when the Treaty was ratified by the Dominican Congress and signed into law by the Dominican President on March 1, 2007. To achieve full Treaty compliance, the GODR committed to: a) removal of all tariff and non-tariff barriers to the import of goods and services from DR-CAFTA contracting partners; and b) harmonizing – reforming and updating – existing Dominican legislation with DR-CAFTA rules and regulations across a range of domestic production, investment, service and professional sectors. Depending on which Treaty chapter was involved, the GODR was granted a periods ranging from 1 to 20 years for full DR-CAFTA enforcement. 2 1.1 BACKGROUND OF THE DR-CAFTA IMPLEMENTATION PROJECT In other cases of specific commodities, particularly agricultural commodities produced in the DR, there are agreed schedules – in a few cases ranging up to 20 years from Treaty ratification – for annual reduction and eventual removal of protective quotas and tariffs. In all cases the GODR agreed to achieve full compliance by stated deadlines for harmonizing its legislation and regulations with agreed treaty terms and conditions. The objective to which DR-CAFTA IP is an ‘improved enabling environment’ that will contribute to achieving USAID/DR’s Economic Growth Strategic Objective (SO-1), “Increased sustainable economic opportunities for the poor” and particularly SO IR 1.1, “Improved conditions for a more competitive Dominican Economy.” The Project also became the principle trade capacity building (TCB) vehicle for meeting a broader USG agreement to help the DR, its Central American partners and other developing country World Trade Organization (WTO) members to honor their commitments to freer trade. The DR-CAFTA IP has employed workshop, seminar and training events and sponsored technical studies and advisory consulting assistance from subject matter specialists as the main tools for achieving its improved institutional capacity and enhanced public-private sector dialogue results targets. The quality and appropriateness of these interventions – as well as the attribution to these interventions of progress toward achievement of results targets – was a major focus of the Evaluation. Also of importance was the Project’s role in coordinating with – and in the case of USG agencies providing logistics for – other donor organizations – notably the European Union, IDB, IBRD, IICA – in building Government of Dominican Republic (GODR) capacity to implement its Treaty commitments. The DR-CAFTA IP project worked particularly closely with the Ministry of Industry and Commerce (MIC) Directorate of Foreign Trade (DICOEX- Acronym for the Spanish title) in supporting its role as coordinator and monitor of GODR agency’s implementation of the DR-CAFTA. USAID designed DR-CAFTA IP and contracted the Project’s implementation to Chemonics under a Global Business Technology Initiative (GBTI) Indefinite Quantity Contract (IQC) Task Order (TO). The purpose of the Project is to assist the GODR in meeting its compliance 1 Internationally, the treaty is known as “CAFTA-DR” as much to reflect historically that the DR was the last country to join the treaty negotiations. However, because in the Dominican Republic, the Treaty is referred to officially as DR-CAFTA, this evaluation uses that nomenclature for the treaty and DR-CAFTA IP for the Project. 2 Copies and descriptions of DR-CAFTA Treaty chapters may be found at: www.ustr.gov/webfm_send/2632 8 requirements under the DR-CAFTA with the US and the five Central American countries.3 During the first three years of Project implementation Chemonics issued four sub-contracts: The DR-CAFTA IP TO began on September 10, 2007 for a period of three years with up to two option years. USAID budgeted $7.4 million in life-of-project (5-year) funding for DR-CAFTA IP implementation, of which $5.0 million was for the first three years of operation. 4 • DASA Consultores Economicos y Financieros – for studies and administrative support • Grupo Consultora PARETO, for studies and workshops particularly for the private sector • INTEGRA Trade and Advisory Group – for studies and workshops, particularly for the public sector • IRIS (University of Maryland) – for coordinating trade research with Dominican academic institutions The purpose of these subcontracts – in addition to achieving more cost-effective Project implementation by using less expensive local firms in most cases – was to build capacity of Dominican firms for providing trade advisory services to the GODR and to industrial and commercial firms and trade associations. In Project Year 4 (Option Year 1) Chemonics extended only its contract with DASA whose staff performs most administrative functions of the DR-CAFTA IP with the exception for the Chief of Party and Financial Administrator functions. At the end of the third Project year, IRIS and PARETO completed their contractual commitments and INTEGRA had dissolved. The USAID/DR has not been the only source of USG support for the GODR in implementing its DR-CAFTA commitments. The USG has also provided treaty compliance support principally from the US Foreign Agriculture Service of the Department of Agriculture (USDA/FAS), the Food and Drug Administration, now under the Department of Homeland Security (DHS/FDA), the US Patents and Trademarks Office (USPTO), in some cases with Project logistical coordination and help. Further, the Project was to have limited involvement beyond coordination of efforts in DR-CAFTA implementation of treaty chapters related to the environment, labor, customs strengthening and trade capacity building, areas to receive support from other global or regional USAID or USG trade support programs. 1.2 PURPOSE OF THE EVALUATION Between June 7 and August 11, 2011, an evaluation team visited the DR to conduct a mid-term performance evaluation of the DR-CAFTA IP (also referred to as “the Project” throughout this report) funded by USAID/DR. This report contains the results from the performance evaluation which was conducted under the USAID Evaluation Services IQC. Per the DR-CAFTA IP Evaluation Scope of Work (See Annex A) the purpose of the performance evaluation was to: • Document and assess DR-CAFTA IP progress to date toward its results targets of improved institutional capacity for DR-CAFTA implementation and enhanced public￾private dialogue about the trade treaty’s benefits and challenges; 3 USAID/DR had been providing earlier support to the GODR during the period of DR-CAFTA trade negotiations under an earlier Competitiveness and Policy Project (CPP) also implemented under contract to Chemonics. 4 In Year 4 (Option Year 1) the Implementer has sub-contracted to a fifth firm, Bixel - a US small business - for designing and developing with DICOEX a trade agreement monitoring and tracking system. This was not yet in place at the time of the evaluation but BIXEL continues to work on this task as part of the DR-CAFTA IP team. 9 • Examine appropriateness – and if so, the type – of short-term support, during a possible Option Year 2 (Year 5) of the Project for further advancement toward DR-CAFTA IP results targets; and • Offer recommendations – based on DR-CAFTA IP findings, conclusions and lessons learned – to the Mission’s long-term strategic planning process for future development assistance to the Dominican Republic. This evaluation responds to the one (1) specific and ten (10) general questions posed by USAID/DR. (See Annex A: Evaluation Scope of Work.) Three and a half years after the GODR ratified DR-CAFTA, USAID seeks to determine what changes have taken place, particularly those that can be attributable to the Mission’s Economic Growth Strategic Objective of “increased sustainable economic opportunities for the poor.” This evaluation is also aimed at identifying design and implementation strengths and weaknesses in the original Project; helping the Mission make mid-course corrections; and designing new assistance initiatives in food security and global climate change for the next USAID/DR strategic planning cycle. This evaluation is also conducted to enable the Mission to be fully responsive to the USAID Administrator’s renewed emphasis on rigorous program and project evaluations both to document impact attributed to Agency interventions as well as to learn from past experiences about what works well and why. The evaluators have also focused their findings and recommendations on providing opportunities for USAID/DR to implement USG special initiatives on food security, global health, and climate change. Creating jobs, as well as increasing and diversifying the country’s rural economic base – and with it increased production, employment, trade and incomes – contributes directly to the capacity of Dominicans to meet the food needs of their households and improve the nutritional and health status of household members. And the fashion with which production takes place can be environmentally positive as well as enhance Dominican competitiveness – e.g., by cultivating organic crops, conserving water, and sustainably harvesting and using forest products. In short, the evaluation findings, conclusions and recommendations are crafted to assist the Mission in assessing progress to date and to serve the Mission as inputs for its next strategic planning cycle. In that context, the evaluation team has examined the extent to which DR￾CAFTA IP has contributed not only to the Mission’s current strategic objective of sustainable competitiveness and economic opportunity, particularly for the country’s poorest households, but also the degree to which it has laid the groundwork for food security and environmentally sustainable production. 2. EVALUATION METHODOLOGY 2.1 THE USAID/DR-CAFTA IP DEVELOPMENT HYPOTHESIS AND RESULTS FRAMEWORK The USAID development hypothesis for the DR-CAFTA Implementation Project is that “increased DR capacity to comply with the ‘rules of trade’ will improve the country’s enabling environment and economic incentives for the private sector to take and manage the risks of expanding investment, production and trade.” To support this hypothesis the DR-CAFTA IP is 10 designed and being conducted to achieve two results, or Program Elements (PEs), as defined in the Project’s PMP Results Framework. 5 • Program Element (Result) #1 – “Dominican institutional capacity for DR-CAFTA implementation strengthened” and The two PEs are: • Program Element (Result) #2 –“Effectiveness of public-private dialogue on DR-CAFTA implementation increased.” Figure 1 is the DR-CAFTA IP Results Framework that has guided project implementation and is used here as the framework for evaluating progress to date. The evaluation of the DR-CAFTA IP Project performance is structured around the two Program Elements (Results) and around the 14 performance indicators used to track, measure, and report progress toward project targets for these results. The 14 Performance Indicators are taken from the DR-CAFTA IP’s PMP and from each of the Project’s annual progress reports listed in the References to this report. Because this evaluation takes place during the fourth year of project implementation – and mostly covers progress during the first three full years of implementation – it focuses on activity and outcome indicators of performance and leaves an examination of the impact of the Project’s interventions on the development hypothesis to a final project evaluation when full implementation has taken place and attribution to project interventions can be more reliably measured. The team recognizes that it takes time to implement DR-CAFTA provisions and even more time for market demand and prices to adjust, investments to be made, and production to respond to new trading realities. This evaluation, therefore, is limited to changes in the Dominican ‘enabling environment’ that can reasonably be expected to contribute to the USAID overall Economic Growth Objective.6 5 USAID/DR Implementation Project, Performance Monitoring Plan, revised. March 2009. 6 During DR-CAFTA IP Year 4 (Option Year 1) the Project implementer introduced a modified results framework that reflected both changes in USAID nomenclature and slight shifts in focus to more fully integrate private sector engagement. These revisions also reflect the implementer’s and USAID’s awareness and responsiveness to of the implementation issues highlighted here and conclusions and recommendations made for addressing them. We are using the results framework applied during the first three years of DR-CAFTA IP because that is the context in which the evaluation is assessing changes during that timeframe. 11 Figure 1: DR-CAFTA IP Results Framework 2.2 MIXED-METHODS EVALUATION APPROACH TO ASSESSING DR-CAFTA IP IMPACT The new PPL/LER “USAID Evaluation Policy” lists 10 important criteria to apply to Agency evaluations in the future.7 The evaluators employed a mixed-methods approach to address the 10 questions in the DR￾CAFTA IP performance evaluation SOW and to assess how effectively the Project achieved its objectives of: a) building GODR capacity for enforcing DR-CAFTA terms and conditions; and b) fostering public-private dialogue about the risks and benefits of DR-CAFTA participation. The mixed-methods approach drew on four types of data sources: While issued after the RFTOP solicitation for this performance evaluation was released, the USAID Evaluation Policy guidance has been used to conduct this evaluation. This evaluation report, for example, fully documents through technical annexes all the data collection instruments and procedures – in this case a mixed-method evaluation approach – followed to support its findings, conclusions and recommendations. 1) Documentation and indicator data contained in the PMP of the Project’s Performance￾Based Management System – over 50 print and electronic citations were reviewed; 2) Direct interviews with key informants from the public and private sectors – 35 interviews were conducted; 3) Event evaluations by beneficiaries who participated in Project-sponsored programs – 1,200 evaluations out of 2,600 were examined; and 7 A copy of the USAID Evaluation Policy, dated January 2011 may be found at www.usaid.gov/evaluation. 12 4) Secondary data on Dominican trade and development – 12 key websites and data sources. 8 The two principle analytical methods the performance evaluation uses are: organizational development (OD) analysis and knowledge, and attitudes and practices (KAP) analysis. Throughout, the evaluation has disaggregated evaluation findings to examine program/project performance and impact among target populations of particular concern to USAID development assistance – notably, the participation of women and the involvement of small enterprises in DR-CAFTA treaty compliance and implementation. 2.2.1 ORGANIZATIONAL DEVELOPMENT (CAPACITY BUILDING) ANALYSIS The DR-CAFTA IP focuses on introducing reforms and building capacity to improve competitive conditions and increasing economic opportunities particularly for small enterprises in domestic as well as regional and international markets. In particular, the Project strengthens DICOEX, the GODR Agency responsible for tracking the compliance of other GODR entities with the trade treaty’s terms and conditions. Introducing trade reforms, including international treaties such as DR-CAFTA, and building institutions to implement those reforms is a process involving several steps along a continuum toward full compliance by the GODR. Since the adoption of DR-CAFTA rules of trade is still in process, the evaluation has applied a qualitative analysis to assess where responsible GODR agencies are located along the continuum of organizational development for monitoring, analysis, communications and public relations capabilities of DICOEX and the regulation and enforcement capacity of other GODR agencies engaged in implementing DR-CAFTA treaty compliance. The evaluators have scored and ranked the GODR’s DR-CAFTA implementing agencies on their status relative to where they are located along this policy reform and organizational development continuum. The status scores below are developed and utilized by the evaluators and are not part of the Project’s PMP. They provide only an approximation of progress made as well as serve as only one factor for identifying where further organizational development and institution building might warrant support. Further, the implementation status of these reforms is somewhat complicated in the case of DR-CAFTA because several of the treaty chapters are so far-reaching in coverage that more than one GODR entity may be involved. Still, the evaluation offers the attached OD rankings as an illustrative vehicle for validating the CAFTA IP’s own ‘score card’ for treaty compliance in that it goes beyond just what is written on paper for any particular treaty regulation to include a measure of the actual capacity in terms of people and funding to implement and enforce what has been promulgated. The scores for measuring the organizational development of Dominican entities responsible for DR-CAFTA implementation are qualitative and ordinal in nature as defined below in Figure 2. 8 The evaluation faced time and budget constraints that precluded inclusion of many key respondents who would have been able to provide more robust confirmation of some the findings and conclusions. This was particularly true for private sector firms and trade associations only a small number of which were reached. It is indicative, however, of the more distant relationships that DICOEX and DR-CAFTA IP teams have with the private sector that it was more difficult to get appointments through those teams with private sector informants; in some cases other channels through the embassy and USAID were necessary. Even then key interviews with CONEP, AIRD and AEIH among others could not be scheduled during the time period the evaluators were in country. 13 Figure 2: Organizational Development Status Scores Score Organizational Development Status 0 = No reform measure/policy exists or has yet been proposed 1 = Reform measure/policy drafted but not approved/legislated 2 = Reform measure/policy approved/legislated and implementing agency/office selected/established but not funded/staffed 3 = Implementing agency/office but not adequately staffed or funded 4 = Implementing agency/office funded and fully staffed 5 = Implementing agency/office fully funded, staffed and functioning effectively for at least one year Of these status scores, 0 through 4 can be verified by documentation such as approved legislation, budgets and staffing levels. A status score of 5 requires additional verification of feedback on ‘effectiveness’ of services provided. The evaluation did not attempt to verify ‘effectiveness’ from final beneficiaries given the short time-period in which DR-CAFTA has so far been implemented. Instead, this verification was conducted through key informant interviews. Of course, many GODR agencies existed before DR-CAFTA; still, to achieve DR￾CAFTA compliance these agencies often need added budgets and staff to implement their new Treaty compliance, regulation, and enforcement mandates. The evaluators are fortunate to have baseline and mid-term assessments9 To the extent possible, the evaluation obtained or consulted documentation and statistics from the Project’s PBMS and from independent sources to verify and establish organizational development status of Dominican entities responsible for DR-CAFTA legislative harmonization, regulation compliance and regulation enforcement. To supplement its documentation review of Dominican DR-CAFTA reforms needed to harmonize national laws and regulations with the DR￾CAFTA treaty terms and conditions and the resources – financial and technical – to implement new DR-CAFTA commitments, the Evaluation also used key informant and focus group interviews to triangulate the documentation of its organizational development measurements. of the treaty implementation capacity of the GODR entities targeted for DR-CAFTA implementation capacity building support under the Project. Those baseline and mid-term assessments were conducted in the first year of DR-CAFTA and Project implementation, 2007 and two years later in 2009. The baseline and mid-term assessments used staffing, budgeting, coordination and communications criteria to identify institutional strengths and weaknesses for DR-CAFTA IP. 2.2.2 KNOWLEDGE, ATTITUDES AND PRACTICES (KAP) ANALYSIS A second measurement tool that the evaluators have employed in examining DR-CAFTA IP impact is KAP Analysis. Through direct individual interviews with a small number of randomly selected direct participants (beneficiaries) in DR-CAFTA IP supported events, the evaluation has attempted to measure where beneficiaries fall in the increased knowledge, changed attitudes and improved practices continuum. This KAP analysis captures the extent to which beneficiaries are knowledgeable about new economic opportunities, are motivated to seek out these opportunities and are in fact engaging in such opportunities through new practices that 9 Hart, Donaldo, Elka Scheker and Patricia Mejia. Evaluacion Inicial de la Capacidad Institucional Dominicana para Implementar el DR-CAFTA. October 2007 and Landman, Robert, and Elka Scheker. Evaluacion de Medio Termino de la Capacidad Institucional para Implementar del DR-CAFTA. April 2009. 14 DR-CAFTA makes possible. The KAP analysis employed focus group and key-informant interviews to assess what constraints beneficiaries and potential beneficiaries face in engaging in or responding to new realities that DR-CAFTA treaty compliance involves. The evaluators were fortunate to have available to them comprehensive participant evaluations of Project-sponsored DR-CAFTA events. These participant event evaluations supplemented with selected focus group and key-informant interviews permit capturing the degree to which knowledge, attitudes and practices have evolved during Project implementation. In addition, the evaluators are able to draw conclusions from participant event evaluations and key respondent interviews about what has worked well and why, what should be dropped, and what should be adjusted or expanded in future initiatives implemented by USAID/DR. The evaluation team has applied its KAP analysis to assessing what is essentially an “intent to treat” approach to the provision of DR-CAFTA IP technical advisory and other services. “Intent to treat” analysis is a rigorous evaluation tool used most frequently in the health field to examine not the effect of, for example, a new vaccine, but rather how comprehensive, targeted, and cost￾effective the delivery of vaccination services was to the potential beneficiary population. A comparable application of “intent to treat” approach for the DR-CAFTA IP evaluation is to measure the cost-effectiveness of Project-sponsored events in achieving the ends of DR￾CAFTA treaty compliance of GODR entities and DR-CAFTA competitiveness of Dominican firms and enterprises in domestic markets with imported products and in international markets with Dominican exports. The evaluator’s KAP analysis draws on information from the evaluation forms completed by DR￾CAFTA IP-sponsored events. The Project has systematically collected the forms that were filled out by event participants at the closing of each workshop, seminar or other information-sharing activity. As of May 2011, the Project had sponsored about 125 such events reaching a total of 4,941 participants by Project estimates. One- or two-page participant evaluations exist in file copies for nearly all of these events. 10 While the quality and completeness of the participant evaluations is uneven, they provide a first, approximate understanding of the impact of Project workshop, seminar, and related events on participants’ awareness and likelihood of changes in attitudes and practices. To discern differences in levels of awareness and patterns in attitudes and practices among event participants, the evaluation compiled a sampling of 20 percent of the Project-funded events stratified according the following categories: 11 • DICOEX institutional strengthening workshops, seminars (n=3); • Other GODR agency institutional strengthening events (n=4); • Private business industry sector events - e.g. agriculture and non-agriculture (n=5); • Private sector and GODR agency Treaty chapter specific events - e.g. IPR, SPS, Customs, Rules of Origin, etc. (n=5); • Agribusiness and small and medium enterprise owners/operators events (n=8); “N” is the number of events that make up the sub-sample for each category of events. In summary, the evaluator’s mixed-methods approach to evaluation of the DR CAFTA-IP provides the Mission with the best possible substantiation of evaluation findings, conclusions 10 In some cases where events were very short – briefing of one or two hours and for senior level officials – the formality of the participant event evaluation was eliminated. 11 Note: The evaluators are aware and have verified that the same participant may have attended more than one of these events; we do not believe this introduces a significant bias in the findings, however, given the preponderance of responses obtained in the event evaluations. 15 and recommendations by applying the most appropriate evaluation tools from USAID’s “Evaluation Toolkit” for addressing the performance evaluation SOW questions. 3. EVALUATION FINDINGS This section lists: a) general findings related to DR-CAFTA IP achievement in increased institutional capacity for DR-CAFTA implementation and improved public-private sector dialogue on DR-CAFTA benefits and challenges and how best to address them; and b) specific findings related to Project design and implementation. 3.1 PROJECT ACHIEVEMENTS (RESULTS) 3.1.1 DR-CAFTA IMPLEMENTING INSTITUTIONS STRENGTHENED After DR-CAFTA signing and entry into force on March 1, 2007, the most urgent GODR needs that the Project helped address were: a) strengthening DICOEX, the Dominican government agency charged with coordinating and monitoring Treaty implementation; and b) building regulatory and enforcement capacity within the GODR agencies responsible for harmonization of domestic regulations with the terms and conditions of the 22 Treaty chapters.12 The USAID￾Figure 3: Institutional Capacity Status for DR-CAFTA Implementation Chapter Number DR-CAFTA Treaty Chapter Implementing GODR Agency1 Implementation Status2 1 Initial Provisions n/a n/a 2 General Provisions n/a n/a 3 National Treatment and Market Access for Goods MIC, MAG(OCTA), MSAP(DCRAB) 4 & 3 & 3 4 Rules of Origin MIC, DGA 3 5 Customs Administration and Trade Facilitation DGA 4 6 Sanitary and Phyto-sanitary Measures MAG (DIA), MSAP 3 7 Technical Barriers to Trade MIC(DEGENOR) 3 8 Trade Remedies MIC 4 9 Government Procurement Min Hacienda 4 10 Investment MIC(CEI) 4 11 Trade in Services MIC 4 12 Financial Services MIC & MinFinance 4 13 Telecommunications INDOTEL 5 14 Electronic Commerce MIC 4 15 Intellectual Property Rights MIC (ONAPI & ONDA) 5 & 3 16 Labor MinTrab n/a 17 Environment SEMARENA, MIC n/a 18 Transparency MIC 3 19 Trade Capacity Building MIC n/a 20 Dispute Settlement MIC 4 21 Exceptions MIC n/a 22 Final Provisions MIC n/a Sources: Project PBMS data and documentation, key informant assessments n/a = Not applicable to the DR-CAFTA IP because it is supported by other USG agency or USAID project or not 12 For a detailed summary of each of the DR-CAFTA chapters visit www.ustr.gov/webfm_send/2632. 16 relevant as an assigned chapter area of responsibility 1 – Status of lead or primary GODR agency responsible for implementing DR compliance with DR-CAFTA regulations 2 – Evaluation status based on the following Organizational Development Scores and defined below and in the Evaluation Methodology Section of this report: Status Organizational Development Status 0 = No reform measure/policy exists or has yet been proposed 1 = Reform measure/policy drafted but not approved/legislated 2 = Reform measure/policy approved/legislated and implementing agency/office identified or established but not funded or staffed 3 = Implementing agency/office funded but not adequately budgeted or staffed 4 = Implementing agency/office funded and fully staffed 5 = Implementing agency/office fully funded, staffed and functioning effectively for at least one year funded DR-CAFTA IP has been one of the GODR’s principle advisory, training and awareness￾raising resources for responding to those needs and for meeting the GODR’s Treaty obligations. Figure 3 summarizes the status of GODR implementation of reforms under each of the 22 DR￾CAFTA chapters for each of the government agencies tasked with Treaty compliance. Concurrently, the Project was also tasked with building the capacity of the Ministry of Industry (MIC) Directorate for External Trade (DICOEX as the ‘go-to’ within the GODR for both other GODR agencies and for private business associations and groups seeking guidance on DR￾CAFTA regulations and procedures and developments). DR-CAFTA IP provided support to DICOEX and to DR-CAFTA implementing entities through consultancies and informational ‘events’ – sector briefings, seminars and occasional topic-specific workshops, for example, how to set up an internet website for DICOEX staff. Between the Project’s start-up in September 2007 and May 2011, just prior to this evaluation, the Project has funded 89 studies and conducted 125 events totaling 768 ‘training’ hours, according to Project records. Event participants ranged from fewer than 10 to over 100 from both the public and private sectors and from both sexes. Events ranged from two hours to two weeks, though most were less than a day. Figure 4 summarizes the Project outputs for publications and events. The Project provided additional support to bring technical specialists for advisory consultancies to the DR to visit specific GODR entities responsible for harmonizing domestic and DR-CAFTA regulations and enforcement procedures. This involved coordinating with and providing logistical support to specialists from other USG agencies (e.g., USDA, FDA, USPTO among them) that were responsible for US compliance with DR-CAFTA. Finally, on very few occasions the Project sponsored visitations to other DR-CAFTA countries to exchange experiences on meeting Treaty compliance. Figure 4: Studies and Events Sponsored by the DR-CAFTA IP Event Participants Public Sector Private Sector Total Studies and Reports - 89 n/a n/a 89 Seminars & Workshops - 125 1,744 3,083 4,941 (1,663 or 34%) Note: Number and percent in parentheses are for women 17 The Evaluation finds that the Project helped the GODR make significant advances in both the capacity of its lead trade agreement agency, DICOEX, and in several GODR ministries and agencies responsible for managing the harmonization of national laws and regulations with those negotiated under the DR-CAFTA Treaty. DICOEX, the GODR agency responsible for coordinating and monitoring DR-CAFTA compliance, has been the central focus of Project institutional capacity building activities. The Project’s effectiveness at increasing DICOEX capacity to carry out its DR-CAFTA responsibilities is best reflected in the independent ISO 9001 certification for the Directorate awarded to DICOEX in late 2009. Similarly to other GODR Treaty enforcement agencies, Project technical assistance has improved the pace at which patents and business registrations are processed, goods are moved through customs, and certification of quality standards takes place. The evaluation finds that one of the greatest contributions of the Project has been its provision of key advisory and training assistance to DICOEX in implementing its agenda for raising awareness and capacity of Dominican entities charged with implementing DR-CAFTA. As an emerging and growing entity with responsibilities for coordinating and monitoring implementation of DR-CAFTA – and several other bilateral and regional trade agreements to which the DR is a cosignatory country – DICOEX has been fortunate to receive increased GODR funding to support staff that have grown from just a handful of professionals to nearly 25 specialists at the time of this evaluation. DICOEX has also expanded into new facilities and has equipped its offices with the latest information technology to support coordination and tracking of the range of trade treaty agreements that have expanded almost exponentially since WTO accession in the 1990s. In several cases, the Project made several of the specialists who served the government as negotiators during DR-CAFTA drafting available to DICOEX and to the GODR. Without the Project, DICOEX would not have been able to help equip key GODR ministries - MAG/OTCA, MIC/ONAPI, INDOTEL, MSP and Hacienda/DGCP among others - in preparing legislative and regulatory reforms necessary to bring the DR into DR-CAFTA compliance and then to monitor and enforce those new regulations. This work is ongoing, but the Project can be credited with launching the regulatory reform and enforcement process through the resources it provided through DICOEX and MIC to train and equip key staff in Dominican entities charged with DR￾CAFTA implementation. The evaluation recognizes that many factors besides Project interventions influence the level of organizational development and effectiveness of GODR agencies. Budgetary constraints are the most obvious. Also critical is the positioning of authority for DR-CAFTA compliance implementation in the institutional hierarchy of ministries. In too many cases - DICOEX in the MIC and Oficina de Tratados Comerciales Agrícolas ((OTCA) - acronym in Spanish), the Ministry of Agriculture and Livestock (MAG), for example – administrative levels are too far down the scale of authority to enable entities to make and keep commitments for mutual DR￾CAFTA implementation support. For example, DICOEX can only encourage participation in critical training events and meetings regarding implementation of a particular DR-CAFTA chapter; it cannot enforce involvement from other GODR entities. The Evaluation has measured achievements against PMP performance indicator targets where possible. However, during Project design, several performance indicators at the activities level 18 were selected as ‘proxies’ to measure outcome according to the PMP. This has made it impossible to track and measure Project accomplishments reliably at the outcome level. The evaluation report identifies these indicator deficiencies below in the discussion of ‘weaknesses’ and recommends measures for their correction. An example of a potential measure of DR￾CAFTA trade impact at the outcome level is summarized in Annex E for FDA detentions of Dominican shipments for failure to meet marking, SPS or other standards, to which DR-CAFTA IP contributed to introducing and institutionalizing under the Treaty. Overall, DR-CAFTA IP is reaching or exceeding most of its activity and output targets as defined by the Project’s PMP (see Annex B). During the first three years of implementation, the Project has met or exceeded the targets set for the number of events, number and type (sex and industry sector) of event participants, and trade-related studies and number of operational and communications tools - results indicators 2-10 and 12-14 (Portman-Bingaman rural development indicators). Measures of outcome performance against targets – results indicators 1, institutional effectiveness, and #11, trade readiness (PART) – are more uneven, due in large part to factors beyond Project control, namely assumptions related to constrained GODR fiscal capacity to fund DR-CAFTA implementing agencies and the effects of the depressed US economy on demand for Dominican products. In summary, there is well-documented evidence to support the following findings regarding the Project: • It has helped the GODR get its trade legislative and regulatory house in order. Project achievements in “capacity building for DR-CAFTA implementation” are manifested in GODR compliance – in terms of accomplishing reforms and harmonization by Treaty deadlines – with all of the relevant chapters of DR-CAFTA compliance in which the Project has been involved. Moreover, GODR entities responsible for coordinating, implementing, enforcing and monitoring DR-CAFTA Treaty regulations demonstrate improved capacity for doing so with larger and better equipped technical staffs than at the time of DR-CAFTA ratification by the Dominican government. The Project might be criticized for not directing more support toward helping the DR achieve greater openness and competitiveness in agricultural trade, which is a priority sector for USAID assistance in reducing poverty and advancing broad-based participation in the national economy. However, the phase-out of protection on sensitive agricultural commodities – notably meat, dairy and grains – is by far the most complicated, protracted and politically sensitive area of DR-CAFTA compliance. Project implementers, perhaps, have been wise – particularly during the initial years of harmonizing national legislation and regulations with DR-CAFTA – to support implementation of less controversial Treaty chapters such as Telecommunications, Rules of Origin, and Intellectual Property Rights. In this fashion the GODR is moving up the learning curve in DR-CAFTA compliance and building experience to take on more challenging Treaty chapter implementation later. • It has strengthened DICOEX as the ‘go-to’ GODR office for coordinating DR-CAFTA implementation and for tracking treaty compliance. DICOEX, the GODR agency responsible for coordinating and monitoring DR-CAFTA compliance, has been the central focus of Project institutional capacity building activities. The Project’s effectiveness at increasing DICOEX capacity to carry out its DR-CAFTA responsibilities is best reflected in the independent ISO 9001 certification for the Directorate awarded to 19 DICOEX in late 2009. Similarly, in other GODR Treaty enforcement agencies Project technical assistance has improved the pace at which patents and business registrations are processed, goods are moved through customs, and certification of quality standards takes place. Supporting its coordinating functions is the DICOEX website and communications arm that includes logistical, informational and programming support and some training for both public and private sector participants interested in understanding new DR-CAFTA rules and regulations. • The DR-CAFTA IP has introduced a level of global visibility, rigor and discipline into GODR policy and regulatory reform processes that otherwise would not have been possible. The DR-CAFTA prods the GODR to catch up and keep up with economic restructuring needed to be a player in global markets. DR-CAFTA is “an invitation to a party,” one evaluation respondent pointed out. As one evaluation key informant respondent said, “The DR needs to know how to dress, how to prepare itself to attend and to benefit. The DR-CAFTA IP has helped greatly.” • It has helped foster structural adjustments in the Dominican economy that have gone beyond basic trade reform to include changes in competition policy that have lowered the costs of doing business for producers and the cost of living of domestic for consumers. One example is the introduction in September 2009 of cell phone number portability that has increased competitiveness among mobile phone service providers. Aside from consumers at large who benefit by redirecting savings in their phone bills to other consumption such as locally produced foods from target rural areas assisted by other USAID projects, low mobile phone costs mean lower operating costs – and hence greater competitiveness – for medium and small agricultural producers who count on cell phone communications for managing their production and marketing operations as documented in an IDB-sponsored policy reform case study. • Institutional capacity for DR-CAFTA implementation remains uneven among the range of GODR entities that have been tasked with Treaty compliance responsibilities. Also relatively weak is the capacity of some GODR agencies to communicate effectively with private business – across industry and trade sectors – about the benefits, challenges, and risks that DR-CAFTA participation involves, a complaint voiced uniformly by private sector key respondents to evaluator interviews. Models and ‘best practices’ for effective public-private dialogue and collaboration for effective DR-CAFTA engagement are followed by a few GODR agencies, but this is not uniform across the government. One example of effective public-private collaboration is the mentoring caseworker arrangement that ONAPI adopts with small and medium micro-enterprises or groups of producers seeking to adopt individual or collective labeling and trademarks for their products. This type of mentoring and casework approach may have application in other sectors and with other agencies that assist firms in meeting DR-CAFTA requirements. • The evaluators found shortcomings that should be addressed in the mechanisms that DICOEX is using to get out information – useful “actionable” information into the hands of other GODR entities as well as into the hands of the public at large and the private business community in general. From personal experience – also echoed by several respondents in other GODR agencies and private trade and industrial associations – the evaluators found lacking the DICOEX website as an information resource on the status of DR-CAFTA compliance and on the regulations and procedures called for by the Treaty. While containing dozens of documents and frequent updates on DR-CAFTA- 20 related events, the website is difficult to navigate and opaque in its organization. Few documents examined from the DICOEX website were written in non-technical or non￾legal form to be more useful to public organizations or private trade associations which might seek to communicate Treaty implications to their memberships. 3.1.2 PROGRAM ELEMENT (RESULT) 2 – EFFECTIVE PUBLIC-PRIVATE DIALOGUE INCREASED In the Dominican Republic, effective DR-CAFTA IP implementation is not just about harmonizing treaty terms and domestic regulations and then building public enforcement capacity. It also includes changing the insular mindset and building the global competitiveness of the private business sector that has operated with government protection and support through tariffs, subsidies and preferential treatment. Further, the private sector has a legitimate interest in expressing its concerns and identifying where the government can be more transparent in application of new treaty requirements, particularly when it falls to government agencies to register and certify compliance with treaty terms – for example, in trademarks and patents and in SPS standards of quality. Recognizing that it falls to the private sector to take advantage of DR-CAFTA market opening opportunities, the DR-CAFTA IP was designed with the task of fostering and facilitating dialogue between public agencies charged with implementing DR-CAFTA treaty compliance and private businesses and business groups – associations, clusters, cooperatives, etc. – now responsible for meeting new regulatory standards for their products in foreign markets, or to compete against imports of competitive products now free to enter a Dominican economy free of tariff or non-tariff barriers. In addition to informing GODR agency staffs about DR-CAFTA implementation requirements, the Project has included a range of private entrepreneurs, trade associations, academic and media representatives in its training workshops, seminars and awareness-raising events. The objectives of this participation, in addition to information sharing, is to engage the private sector in a dialogue that will increase their recognition of how DR-CAFTA will affect their competitiveness in domestic as well as regional and global markets. This dialogue is also directed towards helping the private sector identify how best to invest in and conduct businesses so as to take advantage of new opportunities – and manage new risks – that the trade treaty will generate. The Project points to the number of private sector participants who have participated in its workshops, seminars and awareness-raising events as one of its main achievements. Figure 4 summarizes the number of private sector participants in those events from project start through the March 2011, the middle of Project Option Year 1. In summary, there is well-documented evidence that the Project has: • Cultivated better – if begrudging – private sector recognition of new rules for trade and investment. Event participants’ self-evaluations indicate that the Project raised awareness of GODR entities and Dominican businesses about the opportunities and challenges to the domestic and global competitiveness of Dominican products. Dialogue, however, seems in large measure to be one-way – DICOEX and other agencies telling the private sector the way it is and not getting feedback on what the private sector needs in order to function effectively under a new set of domestic rules generated by DR￾CAFTA compliance. 21 • Yet to engage DICOEX with the private sector in setting up robust consultative mechanisms for information sharing, feedback and dialogue. Uniformly among those private sector interview respondents was an eagerness to get out to their memberships messages regarding current and upcoming DR-CAFTA developments of concern to their business operations. Some private sector respondents shared with the evaluators copies of past trade association organs in which CAFTA topics had been featured as examples of their interest and capacity to reach broad populations of their member firms and businesses. They also indicated that their websites were another vehicle for communication of DR-CAFTA events. Two common complaints expressed by private sector respondents, one being the tardiness with which announcements about DR￾CAFTA developments – e.g., legislation or regulations being drafted for DR-CAFTA implementation. “We hear about these the day before the legislature is to consider them and we are asked to poll our memberships and prepare a position,” observed one trade association head. “We just can’t do that. It’s no way to engage the private sector in dialogue.” A second complaint vocalized with frequency among private sector respondents was the nature of information made available. “Business people don’t read academic reports and that’s all we see coming out of the government. We need practical guidance on how to prepare for and interpret new DR-CAFTA regulations.” 3.2 PROJECT DESIGN AND IMPLEMENTATION The evaluation identified and assessed the strengths and weaknesses it found in DR-CAFTA IP design and implementation. The evaluators have identified the following design and implementation strengths that contributed positively to the effectiveness of carrying out Program Element #1: • The Project has helped sustain a momentum for DR-CAFTA compliance despite changes in MIC/DICOEX leadership. DICOEX staff have witnessed the assignment of three Directors since DR-CAFTA signing; each had a distinctive agenda and vision for the Directorate including, in one case, moving DICOEX from a coordinating and monitoring entity to a trade promotion organization. Despite three Directors since DR￾CAFTA signing, DR-CAFTA IP has helped DICOEX maintain continuity by funding specialists that helped the entity to continue its original mandate of coordinating and monitoring DR-CAFTA compliance. • The Project has supported a critical mass of qualified and experienced trade specialists through their engagement as consultants, workshop leaders, and researchers. After DR￾CAFTA was negotiated and signed many of these specialists dispersed within government agencies, the business sector, academia and private consulting, taking with them their depth of understanding and skills needed to advance the DR-CAFTA compliance agenda and foster the structural changes in the DR economy necessary for local business to play a competitive role in global trade. The Project periodically engaged much of this local talent to conduct critical studies, workshops and seminars to support and equip DICOEX and DR-CAFTA implementing agency staffs responsible for Treaty compliance. Moreover, other GODR entities responsible for coordinating, implementing, enforcing and monitoring DR-CAFTA Treaty regulations demonstrate improved capacity for doing so with larger and better equipped technical staffs than at the time of DR-CAFTA ratification by the Dominican government. The evaluators found the following weaknesses in USAID project design and administration: 22 • The USAID decision to design two ‘Option Years’ into the DR-CAFTA IP project has proven to be disruptive to implementation momentum. At the end of project Year 3, USAID did not officially (sign) authorize an Option Year 1 extension until within 30 days of the original contract expiration, leaving staff uncertain about their future and the contractor unable to arrange for any further consultancies or events. At the moment, toward the last quarter of the current contract option year, the IP contractor is unable to recruit and fill a critical technical staff vacancy because it cannot offer employment beyond September 30, 2011. This has placed heavy burdens on existing staff in meeting Year 4 targets. In addition, because the project cannot bring short-term technical consultants to the DR, they are unable to deliver workshops and events that require advanced preparation, which are the core of the project’s technical support. • USAID branding policy has adversely affected the DR-CAFTA IP project objectives of institutional reinforcement for DR-CAFTA. USAID branding may be appropriate or acceptable for bags of wheat distributed in famine relief programs, but not for inclusion on reports and in events of a sensitive economic nature. For example, DR-CAFTA IP effectiveness at building DICOEX capacity and credibility as the Dominican ‘go-to’ agency to address DR-CAFTA issues has been constrained by USAID branding policy that literally gives the Directorate ‘second billing,’ or no billing, on publications and at information-sharing events. DR-CAFTA information events, slide presentations and event banners are examples where the DICOEX logo is absent or minimal compared to the required USAID branding logo and project title. • USAID has not well administered coordination between DR-CAFTA IP activities and other USAID Economic Growth (EG) office projects – particularly the Rural Economic Diversification (RED) Project. While DR-CAFTA IP has conducted a range of workshops for USAID/RED target cluster populations of agricultural and forestry producers, limited coordination and follow-up between projects has limited the spread of DR-CAFTA compliance benefits to USAID/RED target rural and low-income populations. The evaluation notes that the GODR’s DR-CAFTA harmonization and compliance schedule supported by the Project overlaps very little with the Mission’s EG strategic objective of broad-based and sustainable economic growth – particularly among low-income rural populations – addressed by other EG office programs. Where there is a common nexus of DR-CAFTA IP with other USAID/DR EG projects – e.g., removing SPS and other trade barriers to create a better enabling environment for small agriculture production and export – USAID program and Project implementation documents reveal only limited DR-CAFTA IP engagement with other USAID/DR projects (notably RED). In cases where there have been shared project activities – e.g., Project-conducted DR￾CAFTA informational events for RED commodity cluster groups in regions around the country, there is little evidence in program reports that the Project coordinated and conducted any systematic follow-up with event participants (individual agriculture producers or producer groups) to guide them in how to benefit from reduced trade barriers and increased access to new Central American and US partner markets. In short, the DR-CAFTA and USAID/RED Projects did get bodies in chairs at events but did little together to help get brains and brawn in gear in farmers’ fields to benefit from DR￾CAFTA trade openings with increased rural investments, jobs and incomes. 23 The evaluators suggest that one mechanism to ‘join at the hip’ the two DR-CAFTA IP and USAID/RED projects would be some common outcome indicators beyond Portman￾Bingaman to which each should make a mutually supportive contribution. The Evaluation suggests as a candidate the FDA data on detentions of Dominican shipments at US ports of entry. (See Annex E.) Not only is this data available at no cost online and in almost real time, but the information is of sufficient detail to provide guidance for the two projects as to what were the reasons for detentions, in which regions and commodities and even which firms were delinquent. • USAID introduced (or approved) the use of activity and output-level indicators as proxy measures of outcomes. For example, the number of workshops, studies and participants are poor measures of institution building and changes in practices when other means exist to better serve the Project’s measurement of results achievement. Ten of the fourteen indicators in the DR-CAFTA IP PMP measure activity or output performance – e.g., numbers of events and numbers of event participants – and are used as proxies for outcomes of better, more DR-CAFTA-responsive practices and improved institutional capacity. These indicators might have been more useful if backed up by any independent ‘customer satisfaction’ or practices survey data that could help determine if events are having the desired impact on institutional capacity building and increased two-way public-private dialogue. The evaluators found the following weaknesses in Project implementation: • Project performance has been deficient in establishing mechanisms for channeling DR￾CAFTA information to potential beneficiaries in a usable fashion. An example here are the two expansive studies prepared on “tariff rate quotas” which explain in rather technical fashion the schedules for removal of protection over the next 20 years on a number of sensitive agricultural commodities, notably, rice, sugar, chicken and dairy products.13 • Deficiencies in the process of participant evaluations of DR-CAFTA IP-sponsored events. The DR-CAFTA IP has not adequately utilized its event evaluation process to get feedback from participants on the degree to which their awareness has been raised, attitudes changed and practices improved for implementing DR-CAFTA. The Project is to be applauded for systematically collecting and archiving participant evaluation forms, but there is little documentation to show how this information has been used in planning resource use (in terms of which sectors, Treaty chapters or other emphasis) to give to Project interventions. The process with which evaluations are solicited – often by last￾minute distribution of evaluation forms as participants are preparing to leave an event, rather than earlier in the course of the event when ideas for improvement or topics can These guides, each prepared specifically for the private sector, are written in what this evaluator perceives as a voluminous and academic format and with so few copies that the MAG/OTCA considers its only copy to be ‘worth its weight in gold.’ Copies exist in the Project files and the files of DICOEX, but are hardly accessible to business users. Similar shortcomings exist in the lack of transparent access to content on the DICOEX website. 13 Zuniga, Juan Luis. Guía para la Comunidad Comercial Republica Dominicana: Continentes Arancelarios de Importación bajo el DR-CAFTA, Marzo 2009 and Zúñiga, Juan Luis, Guía para la Comunidad Comercial Republica Dominicana: Salvaguardia Agrícola Especial (SAE) bajo DR-CAFTA, May 2009 24 be covered or more thoroughly covered – has also limited the usefulness of participant evaluations. While event participants’ self-evaluations indicate that the Project raised awareness of GODR entities and Dominican businesses about the opportunities and challenges to the domestic and global competitiveness of Dominican products, it is not clear whether this has led to significant changes in attitudes and practices, particularly in the private sector. Private sector participants’ evaluations of awareness-raising events and technical seminars and workshops are too spotty and uneven in their feedback to be reliable documentation of the Project’s contributions to building a better understanding of DR￾CAFTA or of how to benefit from the Treaty’s terms and conditions. Less than 30 percent of participant evaluations contain sufficiently detailed responses to be of use. No evidence exists from objective, third-party ‘customer satisfaction’ or awareness surveys about the impact on attitudes and practices of participants in Project-sponsored events, studies and reports related to DR-CAFTA. The Project has focused on numbers of events, event participants, and special trade studies without systematic follow-up on how effective these have been in raising awareness and changing attitudes toward DR￾CAFTA engagement, or in improving practices so as to enhance benefits from Treaty participation. There is no evidence that DICOEX has pursued any systematic or consistent plan for following up on events, studies, and consultancies with activities to help participants to improve their operations (in the case of GODR entities) or trade and competitiveness (in the case of Dominican producers and exporters). • Private sector event participant follow-up. Some private sector circles were resentful of the GODR for failing to include them in consultations during DR-CAFTA negotiations. The project has worked to make up for that shortcoming by engaging the private sector in the range of events it sponsored on the characteristics of DR-CAFTA components, and on the schedules for implementing those components. Participants’ event evaluations uniformly point to the usefulness of the topics but also on the brevity with which they were covered. The most common request is for added information, orientation and training. This is particularly true for agri-business and agriculture-based enterprise owners who indicated the need for much more guidance, for example, on new food sanitation requirements, particularly for the US market. • Leveraging producer and trade association capacity. Trade association representatives interviewed indicated their dismay that DICOEX – and the Project by implication – did not recognize the information sources (meetings, trade newsletters, workshops etc.) that could be venues for the inclusion of a training unit – a two-hour, half-day, or full-day – that addresses DR-CAFTA concerns of particular interest to their membership. Trade and producer association representatives claimed willingness to make their communications organs and meeting venues available for sharing relevant DR-CAFTA information. These same representatives felt current communications mechanisms such as reports posted on the DICOEX website were not all that helpful. “Businessmen don’t read reports,” was the observation of the head of one trade association. “They want to see written procedures they can follow and regulations and standards they can understand.” A case in point is the rather voluminous reports on TRQs which would be more ‘digestible’ if prepared in brief brochures and inserted into and distributed with a trade association newsletter. 25 4. CONCLUSIONS AND RECOMMENDATIONS The evaluation’s conclusions and recommendations flow directly from the above evaluation findings and are organized to address: a) USAID Project administration; b) DR-CAFTA IP Implementation and Monitoring; and c) USAID Strategic Planning and New Initiatives. The evaluators do not attempt to recommend, however, whether or not the Mission should increase or decrease funding for DR-CAFTA implementation support in relation to funding priorities across its entire development assistance program portfolio because that goes beyond the scope of the evaluation. Rather, the recommendations identify factors for consideration in determining the future direction of DR-CAFTA IP activities and corresponding funding support in the context of evolving USAID mission priorities and new USG development initiatives. 4.1 USAID PROJECT ADMINISTRATION [Note: The first evaluation conclusion and recommendation is at the top of the list because the evaluation team considers it critical for the Mission’s prompt consideration.] 1. USAID Option Year Policy. Conclusion: Option Year funding built into development the Mission’s DR-CAFTA IP has been disruptive to project implementation by constraining implementing partners’ ability to plan programs and administer activities at the end of the budget year when there is no contractual agreement yet executed and funded. Recommendation: At the project level, determine immediately if it will exercise its option to fund DR-CAFTA IP for a fifth year, and if so, promptly sign a fifth year contract with the DR-CAFTA IP implementing partner so that work can continue – with what course corrections should be made – without disruption. The mission should not wait until the scheduled evaluation delivery date in September to make this decision because of the uncertainty that affects Project momentum and capacity to sustain progress toward fully achieving DR-CAFTA IP outcome and output targets. At the Mission and Agency levels, consider carefully the implementation challenges introduced when ‘option years’ are included as part of project design. Noteworthy challenges that can disrupt project implementation include the difficulty of retaining staff, the inability to commit resources over the long-term for expenditures sub-contracts to keep activities continuing. These can be serious constraints to maintaining project momentum and meeting implementation targets, even when verbal indications and encouragements are given. USAID procurement regulations mandate that contractors, for example, cannot arrange events or schedule studies, research or consultancies beyond the period of contracting. Currently, in the case of DR-CAFTA IP, the project is unable to provide the core of its technical assistance, which includes workshops and events, because these events require advance preparation and cannot be planned until the option year is exercised. 2. USAID Branding Policy. Conclusion. Branding has delayed and complicated the attainment of building DICOEX visibility and respectability as the go-to GODR agency for DR-CAFTA implementation guidance and support by often placing it in a subordinate position to USAID, an agency of the USG that in some DR circles is seen as having imposed DR-CAFTA on the country more for its own interests than for those of the Dominican Republic. USAID branding requirements have left GODR agencies – particularly, the DICOEX, in the shadows with the result of diminished visibility for its role as treaty compliance coordinator and communicator. As a result, the Project has fallen short of what it might have achieved in its institutional strengthening and public-private sector dialogue program activities. Recommendation: For DR-CAFTA IP, apply for waivers – as regards size and location of the USAID logo on all DR-CAFTA IP-supported 26 activities – for events, publications, studies, etc. in which a GODR agency, particularly DICOEX, is vehicle for dissemination. At the Mission and Agency levels, give sincere consideration to current branding policy to allow for modifications in that policy where it would otherwise affect USAID or USG overall goals. The extensive use of USAID branding policy that has hindered the capacity of GODR institutions to establish their identity and visibility, independent of the USG and its own trade agenda, is a case in point. Branding in such cases risks being counter-productive where US trade interests are perceived – rightly or wrongly – to be a ‘hidden agenda’ behind development assistance. For overall Agency project design practices, introduce levels of ‘branding visibility’ according to the degree of likely positive or negative impact on project objectives and inform USAID staff in how to select, justify and request the level appropriate to their project implementation needs. 3. USAID DR-CAFTA IP and USAID/RED Project Integration. Conclusion: More productive performance from DR-CAFTA IP and USAID/RED can result from better integrated USAID direction and management of the two projects. Recommendation: USAID should convene the DR-CAFTA IP and USAID/RED Team leadership to develop along with the COTRs a mutually reinforcing work plan for the remaining periods of project implementation. One or two common indicators toward which both projects would make a contribution should be included in the plan. The evaluation suggests one illustrative example of a common indicator in Annex E: Alternative Performance Indicator Option in which both Projects have an interest and can make a contribution. The indicator is based on FDA rejections of Dominican shipments due to SPS and other deficiencies and is a free good, costing next to nothing to collect from the FDA website; DICOEX already tracks and reports this indicator.14 4.2 DR-CAFTA IP IMPLEMENTATION AND MONITORING 4. USAID PMP Output and Outcome Results Indicator Measurement. Conclusion: Several Project PMP indicators have limited use in measuring outcome results because they essentially measure activities or outputs that may or may not produce desired outcomes. Recommendations: De-emphasize activity-level indicators; introduce fewer, more relevant measures of Project performance – e.g., ‘customer satisfaction sample surveys’ to assess impact of Project-sponsored events on participants’ knowledge, attitudes and practices. An example of a proposed performance indicator is included in Annex E. 5. Future DR-CAFTA Implementation Support Focus. Conclusion: Several interview respondents observed that there is sufficient momentum behind the implementation and enforcement of many DR-CAFTA chapters that have external support – or at least support from the Project no longer appears warranted. GODR treaty implementation, while uneven among the Treaty’s 22 chapters, has advanced enough that the Project can focus on those chapters with greatest deficiencies and/or rationale per USAID’s strategic objectives. Recommendation: During its remaining implementation period, the Project should redirect its resources toward those DR-CAFTA chapters and those GODR implementing, regulatory and enforcement agencies that are the furthest behind 14 See MIC/DICOEX. Análisis de Las Detenciones de Embarques Dominicanos por Parte de Estados Unidos: Periodo Enero Diciembre 2010. Sto Domingo, Marzo, 2011. 27 in Treaty compliance implementation and in national economic preparedness to take risks and invest in expanding exports or increasing domestic competitiveness against imports that will enter the country without tariff or non-tariff restrictions. Focus remaining Project resources on: Chapter 3: National Treatment & Market Access; Chapter 4: Rules of Origin (ROO); Chapter 6: Sanitary and Phytosanitary Standards (SPS); Chapter 7: Technical Barriers to Trade and Chapter 19: Trade Capacity Building (TCB). 15 6. Private Sector Communications and Dialogue. Conclusion: There is underutilized capacity and goodwill among private trade/producer associations for getting out DR￾CAFTA messages to private enterprises that is available for DICOEX to tap. Recommendation: Help DICOEX work with trade/producer associations on getting critical DR-CAFTA messages to their memberships. Engage agricultural, industrial, commercial, professional and other trade associations in a dialogue on how best to reach their members with information on the positive and negative impacts of DR￾CAFTA on their businesses and incomes. Look for ways, such as trade association meetings, newsletters, and websites to get out information about DR-CAFTA to their memberships; work with trade association leadership to tailor messages to be responsive to their memberships’ interests and needs. 7. DR-CAFTA Orientation for Legislators and Senior GODR Ministry Officials. Conclusion: The unequal awareness about and commitment to DR-CAFTA compliance among senior GODR officials has adversely affected critical coordination between line ministries at the technical level and has delayed legislating needed regulatory measures and legal reforms for Treaty compliance implementation. Recommendation: Develop and implement a strategy for briefing high-level GODR legislators and administrators – perhaps political candidates and legislators as well – on DR-CAFTA issues. 8. DR-CAFTA IP Awareness and Informational Events. Conclusion: The wide range of DR-CAFTA IP-sponsored event topics has challenged the Project, DICOEX and GODR implementing agencies in their ability to follow up with and support events participants seeking more help and guidance. Recommendation: The mission should direct the DR￾CAFTA IP to scale back and focus future event funding on Dominican ‘change agents’ rather than on just filling seats at seminars. These ‘change agents’ should be carefully selected for their skills and their leadership roles in key institutions – producer clusters and trade associations in the private sector and vice ministerial and directorate levels in the GODR – with the goal of supplementing the effectiveness of both GODR agency capacity building and private enterprise dialogue. 9. DICOEX Website Communications. Conclusion: The DICOEX website’s deficient navigability, structure and content accessibility limit its usefulness at present as a communications tool and constrain the Directorate’s role as the GODR leading coordinator and information sharing entity on DR-CAFTA compliance, challenges and opportunities. Recommendation: The Project should reinvigorate its work with DICOEX 15 We note here that the Project Team and its DICOEX partner may have a different view of which DR-CAFTA chapters would be best to continue supporting – IPR, for example, as well as dispute settlement. We will defer to the project and to the USAID Mission on which chapters but urge that chapter selection be based on sound and transparent criteria which go beyond simply meeting the DR-CAFTA compliance calendar but instead include such factors as: 1) what other donors are doing and what comparative advantage other donors have; 2) Mission rural development and related new FtF and GCC priorities; and 3) where the GODR is now capable of moving ahead without any outside help but urged on simply by its treaty commitments and its pride in having made so much progress to date. 28 not only in redesigning its website but also in building integrated and innovative systems that will strengthen DICOEX’s information and communication capacity. This would be timely as DICOEX has a new website design that has not yet been implemented because the Ministry of Industry and Commerce (MIC) is undergoing an assessment in order to upgrade its technology. MIC anticipates implementing the new website design as part of the assessment’s results, and is awaiting these results due to the potential need to make small changes, per the parameters of the new technology. In addition to the website, the project is working on implementing two systems that will assist DICOEX to strengthen its communication channels and monitor the administration and implementation of trade agreements such as DR-CAFTA. These systems will complement and strengthen DICOEX’s capacity to provide “real time” information, not only through its website but also through other means. 10.DICOEX Treaty Compliance Monitoring. Conclusion: DICOEX’s role in treaty monitoring remains a ‘work in progress’ with limited effectiveness as currently conducted by its legal department. Recommendation: Help DICOEX install and operate an electronic scorecard or equivalent tool for treaty compliance tracking and web-based reporting. 11.Project Participant Event Evaluations. Conclusion: The DR-CAFTA IP has not managed or utilized its participant event evaluation process to get feedback on progress made and challenges to meet in raising awareness, changing attitudes and encouraging practices to benefit from DR-CAFTA engagement. The Project is to be applauded for systematically collecting and archiving participant evaluation forms, but there is little documentation to show that this information has been used in allocating resources more effectively – in terms of which sectors, Treaty chapters or other emphasis to give to Project interventions. Recommendation: The process with which evaluations are solicited – often by last-minute distribution of evaluation forms as participants are preparing to leave an event, rather than earlier in the course of the event when ideas for improvement or topics can be covered or more thoroughly covered – should be improved so that participant evaluations can be more useful inputs for Project implementation. 3.3 MISSION STRATEGIC PLANNING & NEW INITIATIVES 12.Global Climate Change (GCC) Initiative. Conclusion: Energy conservation and alternative energy contribute directly to greater Dominican trade competiveness. The application of SPS to agricultural commodity production encourages the use of green – no chemical – cultivation practices that not only open up niche markets for Dominican organic produce but also help conserve bio-diversity through prudent management – or zero use – of agricultural chemicals. Recommendation: Support DR-CAFTA strengthening through the GCC where treaty compliance will foster investments in green production through use of alternative energy and conservation practices and zero use of agricultural chemicals, or use within SPS guidelines that are environmentally safe. 13.Feed the Future (FtF) Initiative. Conclusion: The DR-CAFTA has direct links to USAID’s Feed the Future (FtF) Hunger Initiative through access to food – by generating jobs and incomes; also, health is impacted by poor food safety standards and practices. Recommendation: Support DR-CAFTA SPS strengthening for greater rural export competitiveness, jobs and incomes and health conditions; support DR-CAFTA 29 compliance guidelines that give rural producers a marketing edge such as patent and trademark protection for producer and artisan cooperatives. 14.Education. Conclusion: There is a broad awareness and consensus in both public and private circles that Dominican trade competiveness is constrained by lack of a technically qualified labor force. Recommendation: Focus future USAID education resources on supporting GODR technical training programs in skill areas needed by private business. 5. LESSONS LEARNED AND BEST PRACTICES The evaluation derives the following lessons learned from DR-CAFTA IP implementation. From each of these lessons learned a corresponding best practice emerges. 1. International trade treaties can be useful tools for USAID’s development assistance kit. DR-CAFTA is more than a trade agreement as are almost all regional and bilateral trade agreements under the WTO today. DR-CAFTA also calls on the Dominican Republic to adopt a number of domestic fiscal and economic reforms that internally make the country more efficient and competitive. (See the USTR website for the DR-CAFTA chapters that call for internal domestic economic policy reforms in such areas as investment, competition and SPS standards.) Best practice. Design and implement economic growth development assistance programs to include compliance with appropriate trade treaty measures. For example: Export competitiveness of small agribusinesses in commodity production chains can be increased by fostering compliance with DR-CAFTA SPS certification and product trademarks. 2. Avoid option years as they can disrupt project momentum. Best Practice. Contract projects for their entire planned implementation period. While this may go against current USAID funding policy which advocates the use of option years, it is critical for Missions to make their voices heard that such a policy should not be applied blindly to all projects, particularly those like DR-CAFTA IP that have a long-term institution building agenda and limited resources for achieving their targets. In such cases sustainable momentum is critical to cost-effective use of government resources. 3. Exercise branding waivers when appropriate as they can lead to more cost￾effective use of USG funds for institution building. USAID has a mechanism for waiving or modifying its branding policy where justified. Best Practice. Train Mission staff on how to know when waivers are appropriate and how to justify and apply them. ANNEXES ANNEX A: TRI-Project Evaluation Scope of Work................................. i ANNEX B: Selected Documentation Sources Consulted....................xi ANNEX C: Organizations and Persons Contacted.............................xii ANNEX D: DR-CAFTA IP PMP Indicators, Targets and Actuals .......xiv ANNEX E: Alternative Performance Indicator Option........................xv ANNEX F: Key Informant Interview Form ........................................xviii ANNEX G: Participant Event Evaluation Form ..................................xxi i ANNEX A: TRI-Project Evaluation Scope of Work ii iii iv v vi vii viii ix x xi ANNEX B: Selected Documentation Sources Consulted Batista, Lynette, Matriz de Roles y Responsibilidades por Instituciones Prioritarias Para la Republica Dominicana en El DR-CAFTA. Chemonics. DR-CAFTA IP PMP (revised). October 2009. Chemonics. Year 1 Annual Progress Report, September 2007-September 2008. 15 October 2008. Chemonics. Year 2 Annual Progress Report, October 2008-September 2009. 15 October 2009. Chemonics. Year 3 Annual Progress Report, October 2009-September 2010. 15 October 2010. Chemonics USAID, Contract Number EEM-I-00-07-00008-00, DR-CAFTA Implementation Project. September 10, 2007. Hart, Donaldo, Elka Scheker and Patricia Mejia. Evaluacion Inicial de la Capacidad Institucional Dominicana para Implementar el DR-CAFTA. October 2007. Landman, Robert, and Elka Scheker. Evaluacion de Medio Termino de la Capacidad Institucional para Implementar del DR-CAFTA. April 2009. Mendez, Gastón. Revision del Plan de Gestión, October 2010. MIC/DICOEX. Análisis de Las Detenciones de Embarques Dominicanos por Parte de Estados Unidos: Periodo Enero Diciembre 2010. Sto Domingo, Marzo, 2011. Nuñez, Ruben D. and Danilo Cruz de Paula. DR-CAFTA: The Day After: A Layman’s Guide to What the Trade Agreement Means for the Dominican Republic. Chemonics, USAID Competitiveness and Policy Project, February 2007. Zuniga, Juan Luis. Guía para la Comunidad Comercial Republica Dominicana: Continentes Arancelarios de Importacion bajo el DR-CAFTA. March 2009. Zuniga, Juan Luis, Guía para la Comunidad Comercial Republica Dominicana: Salvaguardia Agrícola Especial (SAE) bajo DR-CAFTA. May 2009. USAID/DR. Dominican Republic Development Assistance Strategic Plan, 2005-2010, Sto. Domingo, 2005. USAID. Agency Evaluation Policy, January 2011. USAID. TIPS: Conducting Mixed-Method Evaluations, 2010. xii ANNEX C: Organizations and Persons Contacted USAID/DR Duty Greene, Director, Office of Economic Growth Luis González, COTR, DR-CAFTA IP Odalis Pérez, Mission Environmental Officer Conrado García, Mission Economist USDA/FAS – Carlos G. Suárez, Senior Agricultural Specialist DR-CAFTA IP Lynette Batista, Project Director Wendy Soto, M&E and Outreach Patricia Yanes, Chemonics DR-CAFTA IP Backstop USAID/RED Project Dr. Jesús de los Santos, Project Director Pilar Ramírez, Organic Agriculture Specialist DR-CAFTA IP Current/Past Consultants and Sub-Contractors DASA – Dr. Roberto Despradel, VP PARETO – Dr. Rolando Guzmán, Executive Director INTEGRA – Lic. Elka Scheker, Consultant and former GODR ‘services’ negotiator Consultant – Luis Ramon Rodriqguez, Agriculture and Agribusiness Consultant – Blanca Jimenez, Small and Micro-Enterprise Consultant – Natalia Polanco, Dispute Settlement Ministerio de Industria y Comercio-DICOEX Yahaira Sosa Machado, Director Alberto Duran Espaillat, Trade Capacity Buidling Coordinator Rene Tavernas, Trade Capacity Building Expert Lic. Haydée Fadul, North Regional Coordinator, Santiago Manuel Rodríguez, Economic Studies Coordinator Ministerio de Agricultura Leslie Marmolejo, Encargada, Oficina de Tratados Comerciales Agrícolas (OTCA) Raúl Peralta, Dirección de Inocuidad Agrícola Ministerio de Salud Pública y Asistencia Social Dr. Salvador Hiciano, Control de Riesgo en Alimentos Ministerio de Hacienda Lic. Yaraida Vólquez Helena, Analista Dirección General de Contrataciones Públicas ONAPI Ayalivis Garcia, Director Trade Marks and Patents xiii CNC Laura del Castillo Saviñón, Technical Sub-Director Gisela Delgado, International Assistance Expert Maria de Lourdes Nuñez, Coordinator, Logistics and Trade Promotion INDOTEL Lic. Julissa Cruz Abreu, Head of Regulations Department INTEC Dr. Julio Sánchez Maríñez, Vice-Rector for Research and Int’l Relations AMCHAM William Malamud FEDECAMARAS Dennis F. Suarez, Executive Director JAD Osmar Benitez Claudia Chez xiv ANNEX D: DR-CAFTA IP PMP Indicators, Targets and Actuals Indicator No. Result DR-CAFTA IP Indicators Baseline Target/Actual 2008 2009 2010 1 PO Percent of effectiveness of Dominican Republic government institutions (specifically DICOEX) to implement DR￾CAFTA 0 50/57 60/65 70/77 2 PE1 Number of legal, regulatory or institutional actions taken to improve implementation or compliance with international trade & investment agreements 16 6/11 6/12 3/5 3 PE1 Number of people (cumulative) trained to implement trade agreements (DR-CAFTA) 0 300/1889 300/1624 300/849 4 PSE1 Number of operational tools developed and established with partner institutions 0 20/44 20/37 20/22 5 PSE1 Number of communication mechanisms developed and established by partner institutions 0 10/9 10/13 5/6 6 PSE 3 Number of M&E tools developed with partner institutions 0 10/17 10/15 5/9 7 PE2 Number of events in which private sector institutions actively participate in DR￾CAFTA funded activities and are accessible to the private sector 0 20/39 20/38 15/28 8 PSE4 Number of awareness sessions organized to discuss DR-CAFTA related issues 0 25/44 25/ 25/ 9 PSE4 Number of training hours provided to project participants 0 200/271 200/254 150/139 10 PSE4 Number of local press articles on DR￾CAFTA related matters 0 100/64 100/71 50/77 11 PART Improved trade readiness (i.e. complying with WTO standards and protocols for production and export) of LAC presence countries, calculated as exports as % of GDP 17.5 17/14.8 17/11.7 17/11.7 12 Portman￾Bingaman Number of legal, regulatory or institutional actions taken to improve agricultural sector implementation or compliance with international trade and investment agreement 0 2/4 2/8 6/6 13 Portman￾Bingaman Number agricultural sector participants (cumulative) trained to understand and maximize the benefits of DR-CAFTA 0 200/877 300/891 300/146 14 Portman￾Bingaman Number of awareness sessions organized to discuss DR-CAFTA issues relating to agriculture 0 20/21 20/18 8/5 PO = Program Objective; PE = Program Element; PSE = Program Sub-element; PART = Program Assessment Rating Tool xv ANNEX E: Alternative Performance Indicator Option In the course of its data collection, the Evaluation discovered one source of secondary data that holds potential for more meaningful measurement of project impact. The evaluator believes there may be sources of information useful to measuring other project impacts as well. FDA reports on the number of shipments detained from the Dominican Republic contain statistics that appear promising as measures of the degree to which Dominican firms are adapting to new DR-CAFTA trade requirements for meeting sanitary and phyto-sanitary standards (SPS) and labeling, for example. Data are compiled and reported daily on Dominican shipments for these deficiencies. Included in the detention reports for each deficient shipment are the date of detention, type of commodity, port of embarkation, port of entry, reason for infraction, and responsible firm. The FDA detention reports are compiled and updated on a daily, weekly and month basis, which permits comparisons in trends for this data. For example, the following table shows trends in the number of detentions annually through 2010 and starting in 2007, the first year of DR-CAFTA application. Tables following that show the major commodities detained, the causes for detention, the regions of origin of the commodities and names of the responsible firms. Such data can be used in tracking changes in detentions as a measure of GODR efforts to apply DR-CAFTA SPS standards. They can also be used to plan assistance to responsible firms, regions and commodities related to the more frequent detention occurrences. The data tables and graphs are taken from a DICOEX study of FDA detentions, but the data can be downloaded and compiled in a number of fashions directly from the FDA website. xvi xvii xviii ANNEX F: Key Informant Interview Form RESPONDENTS ENGAGED IN OR AFFECTED BY DR-CAFTA IMPLEMENTATION AND/OR THE USAID-FUNDED DR-CAFTA IMPLEMENTATION PROJECT Talking points: • The US Government wants to learn how it can improve the effectiveness of its cooperation with the Dominican Republic in bringing the benefits of DR-CAFTA to Dominican business and consumers. • We are looking particularly at ways that to better support: o Legislative and regulatory reform needed for DR-CAFTA compliance o DICOEX in its DR-CAFTA trade policy coordination and information sharing roles o Other GODR institutions implementing DR-CAFTA agreement compliance o Dominican private sector understanding of DR-CAFTA and its benefits/challenges in the DR • We welcome suggestions about where else to look and whom else to contact for information about how to build on DR-CAFTA benefits to date and how to spread benefits more broadly. Interviewer: _____________________ Date: ____________________ PART 1: RESPONDENT BACKGROUND INFORMATION Name: Title/Function Organization: Contact Info: Relation (past/present)to DR-CAFTA: Period of Treaty or Project Involvement: PART 2: PAST DR-CAFTA IP PERFORMANCE 1. What have been the principle achievements from DR-CAFTA agreement? From the Project? For whom? • Outcomes: Policy, regulatory legislative reforms for DR-CAFTA compliance (specific examples); changes in producer – large and small – and consumer welfare, if possible to document - • Outputs: DICOEX and GODR agency institutional – administrative efficiency, inter-agency coordination and communications (transparency) capacity; in Private Sector awareness and treaty compliance 2. For DR-CAFTA – for ALL respondents: where has DICOEX and GODR agency treaty compliance xix implementation fallen short of expectations (desired results)? Where are its weaknesses (w) still? What are its strengths(s)? What corrections have (or need to be) to complete the Treaty compliance process? GODR/DICOEX Treaty Compliance regulation/legislation preparation/reform support GODR/DICOEX Treaty compliance implementation support GODR/DICOEX Inter-agency coordination GODR/DICOEX Communications and information sharing with private sector and public 3. For the “IP” Project – when applicable, if not go to #4): Where has implementation fallen short of expectations? Exceeded expectations? Where are its weaknesses (w) still? Strengths (s)? What corrections have (or need to be) for supporting full treaty compliance? What corrections have been or should be made? • Design (events, consultants/advisors, studies, other): • Implementation (events, consultants/advisors, studies) internally to project and its partners and externally with DICOEX, GODR and other USG agencies and programs: 4. What lessons have been learned or best practices identified during DR-CAFTA compliance implementation? xx In Project design/implementation? • • 5. Where and how can the USG best allocate its future TCB support in the DR? • In the short-term? • In the longer-term? 6. What scope is there for coordinating USG TCB support with: • Other donor organizations? • Other USG institutions? PART 3: RESPONDENT REFERRALS AND EVALUATION 7. Can you provide referrals and contact information to other ‘go-to’ trade and DR-CAFTA experts in the GODR, private sector and/or academia? Studies or sources of information? • • Respondent assessment: cooperative and helpful = y/n/so-so; informed and knowledgeable = y/n/so-so For continuation space, use reverse side of corresponding page and indicate question number. xxi ANNEX G: Participant Event Evaluation Form