SAFE AGRICULTURE/FOOD EXPORT (SAFE) PROJECT MIDTERM EVALUATION REPORT APRIL 30, 2019 SAFE AGRICULTURE/FOOD EXPORT (SAFE) PROJECT MIDTERM EVALUATION REPORT J.E. Austin Associates, Inc. produced this publication at the request of the National Cooperative Business Association CLUSA International (NCBA CLUSA) in response to request for proposals (RFP) ‘NCBA CLUSA International Safe Agriculture/Food Export (SAFE) Project Mid-Term Evaluation’. It was prepared independently by: David Rinck - Team Leader Antonio Cordella – Agricultural and Trade Economist Xerius Research & Consulting, S.R.L. – Quantitative Research Team DATE: April 30, 2019 Agreement No: FCC-517-2015/008-00 J.E. Austin Associates Inc. 1621 N. Kent Street Suite 601 Arlington, Virginia 22209, USA T 703.841.9841 F 703.841.9847 www.jeaustin.com Cover photo: Dairy producers delivering milk to a milk collection center in Arenoso, Duarte Province, Dominican Republic (Credit: David Rinck for J.E. Austin Associates, Inc.) DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States government. SAFE PROJECT – MIDTERM EVALUATION REPORT 2 CONTENTS Acronyms............................................................................................................................ 5 Acknowledgements ........................................................................................................... 7 Executive Summary .......................................................................................................... 8 Project Overview......................................................................................................................................... 8 Evaluation Overview.................................................................................................................................... 8 Conclusions ................................................................................................................................................... 9 Recommendations and Lessons Learned............................................................................................. 14 1. Evaluation Overview ...............................................................................................17 1.1 Project Description..................................................................................................................... 17 1.2 Evaluation Purpose...................................................................................................................... 18 1.3 Evaluation Questions .................................................................................................................. 19 2. Evaluation Methodology .........................................................................................21 2.1 Data Collection Methods .......................................................................................................... 21 2.2 Data Analysis................................................................................................................................ 27 2.2 Potential Biases and Limitations............................................................................................... 28 3. Findings and Conclusions........................................................................................29 3.1 Goal Achievement – Findings ................................................................................................... 29 3.2 Goal Achievement – Conclusions............................................................................................ 47 3.3 Relevance – Findings................................................................................................................... 50 3.4 Relevance – Conclusions ........................................................................................................... 52 3.5 Effectiveness – Findings .............................................................................................................. 53 3.6 Effectiveness – Conclusions ...................................................................................................... 59 3.7 Efficiency – Findings..................................................................................................................... 61 3.8 Efficiency – Conclusions ............................................................................................................. 63 3.9 Sustainability – Findings .............................................................................................................. 64 3.10 Sustainability – Conclusions ...................................................................................................... 66 3.11 Impact – Findings ......................................................................................................................... 67 3.12 Impact – Conclusions ................................................................................................................. 68 4. Recommendations and Lessons Learned .............................................................70 4.1 Recommendations ....................................................................................................................... 70 4.2 Lessons Learned .......................................................................................................................... 73 Annex B. Project Activities ............................................................................................75 Annex C. Evaluation Team.............................................................................................78 Annex D. Evaluation Timeline .......................................................................................80 Annex E. Citations and References ...............................................................................81 Annex F. Data Collection Tools.....................................................................................84 Annex G. Indicator Analysis ...........................................................................................99 Annex H. Survey Results...............................................................................................106 Annex I. List of Contacts ..............................................................................................119 Annex J: Staff Notes on Equivalence ...........................................................................122 SAFE PROJECT – MIDTERM EVALUATION REPORT 3 List of Tables Table 1: Evaluation Questions .................................................................................................................... 19 Table 2: Sample Frame by Province and Profession .............................................................................. 22 Table 3: Calculation of Sample Population by Confidence Level........................................................ 23 Table 4: Key Informant Interviews by Stakeholder Group .................................................................. 25 Table 5: Focus Group Discussions by Location...................................................................................... 26 Table 6: Site Visits by Location................................................................................................................... 26 Table 8: Goal Achievement Evaluation Questions................................................................................. 29 Table 9: RI 2-5 – Individuals Benefitting Directly as a Result of USDA Assistance ........................ 30 Table 10: RI 1: Percentage Increase in L/D/C ......................................................................................... 31 Table 11: RI 11-12 Hectares under Improved Techniques and Technologies................................. 35 Table 12: RI 13-19 Individuals who have Applied Improved Techniques.......................................... 35 Table 13: RI 17-19 Individuals who have Applied Specific New Techniques ................................... 36 Table 14: AIs Related to Application of New Improved Techniques................................................ 36 Table 15: AI 65 – Number of Loans Disbursed as Result of USDA Assistance.............................. 37 Table 16: RI 10 Farmers Making Decisions on Economic Considerations or Analysis.................. 37 Table 17: AIs Related to Private Sector Extension Services................................................................ 38 Table 18: Equivalence.................................................................................................................................... 39 Table 19: RI 20-23 POs and Enterprises Applying Improved Techniques................................... 40 Table 20: RI 24, 27 Increase in Grade “A” Milk Sold and New MCC Equipment..................... 41 Table 21: AI 34 People Reached through Public Information Campaign .......................................... 44 Table 22: Case Study - Quesería Rottis ................................................................................................... 44 Table 23: RI 28 29 New Beef Markets and Signed Agreements for Exports................................... 45 Table 24: Relevance Evaluation Questions .............................................................................................. 50 Table 25: Dominican Beef Exports under DR-CAFTA......................................................................... 51 Table 26: Quantitative Survey Results - Relevance of Interventions................................................. 51 Table 27: Effectiveness Evaluation Questions ......................................................................................... 53 Table 28: Quantitative Survey Results - Effectiveness of Interventions............................................ 54 Table 29: RI 1: Percentage Increase in L/D/C ......................................................................................... 54 Table 30: RI 11-12 Hectares under Improved Techniques and Technologies................................. 55 Table 31: AI 65 – Number of Loans Disbursed as Result of USDA Assistance.............................. 55 Table 32: Efficiency Evaluation Questions................................................................................................ 61 Table 33: Sustainability Evaluation Questions......................................................................................... 64 Table 34: Quantitative Survey Results - Sustainability of Interventions............................................ 64 Table 35: Impact Evaluation Questions .................................................................................................... 67 Table 36: RI 13-19 Individuals who have Applied Improved Techniques.......................................... 67 Table 37: RI 1: Percentage Increase in L/D/C ......................................................................................... 67 Table 38: RI 24, 27 Increase in Grade “A” Milk Sold and New MCC Equipment..................... 67 Table 39: RI 28 29 New Beef Markets and Signed Agreements for Exports................................... 68 Table 40: Evaluation Timeline ..................................................................................................................... 80 Table 41: Quantitative Survey Forms........................................................................................................ 84 Table 42: FGD KII Guide ............................................................................................................................. 97 Table 43: Result Indicators –Performance Against End of Project Targets....................................100 Table 44: Activity Indicators - Performance Against Mid-Point Targets.........................................102 Table 45: Overall Survey Results .............................................................................................................106 Table 46: Key Informant Interviews........................................................................................................119 Table 47: Focus Group Discussions........................................................................................................120 Table 48: Site Visits.....................................................................................................................................121 SAFE PROJECT – MIDTERM EVALUATION REPORT 4 List of Figures Figure 1: SAFE Project Geographic Focus................................................................................................ 17 Figure 2: EERC Haiti, Dominican Republic - Drought Situation Map ................................................ 32 Figure 3: Capacity Development Survey Results Disaggregated by Site ........................................... 62 Figure 4: Chain-Wide Approach to Quality Upgrades.......................................................................... 70 Figure 5: SAFE Project Food for Progress Results Frameworks......................................................... 74 Figure 7: Quantitative Survey Results..................................................................................................... 107 SAFE PROJECT – MIDTERM EVALUATION REPORT 5 ACRONYMS ADHA Asociación Dominicana de Hacendados y Agricultores AGANI Asociación de Ganaderos de Nisibón, Inc. AI Activity indicators(s) AI Artificial insemination AGASAJUMA Asociación de Ganaderos de San Juan de la Maguana ASOCARNE Asociación Nacional de Productores de Ganado de Carne BAGRICOLA Banco Agrícola CAFTA-DR Dominican Republic-Central America Free Trade Agreement CBSP Community-Based Service Providers CCA Central Competent Authority CCC Commodity Credit Corporation CDPMN Caribbean Drought & Precipitation Monitoring Network CDSO Crude degummed soybean oil CLUSA Cooperative League of the USA CONALECHE Consejo Nacional para la Reglamentación y Fomento de la Industria Lechera COOPAGASAJUMA Cooperativa de Ganaderos de San Juan, Inc. COP Chief of Party CRI Cooperative Resources International (GENEX) CVO Chief Veterinary Officer DFI Development Finance Institution DIGEMAPS Dirección General de Medicamentos, Alimentos y Productos Sanitarios DIGEGA Dirección General de Ganadería DOP Dominican Peso ERCC Emergency Response Coordination Center EQ Evaluation question ET Evaluation Team FAS Foreign Agricultural Service FEDA Fondo Especial para el Desarrollo Agropecuario FEDEGANO Federación de Ganaderos de la Línea Noroeste FEDAGARE Federación de Asociaciones de Ganaderos de la Región Este FEDEGANORTE Federación de Ganaderos de la Región Norte FEGACIBAO Federación de Ganaderos del Cibao FEGASUR Federación de Ganaderos de la Región Sur FFPr Food for Progress FFS Farmer Field School(s) FGD Focus group discussion(s) FSIS Food Safety and Inspection Service GoDR Government of the Dominican Republic GAP Good Agricultural Practices GMP Good Manufacturing Practices HA Hectare(s) HACCP Hazard Analysis and Critical Control Points IFAD International Fund for Agricultural Development IIBI Instituto de Innovación en Biotecnología e Industria INDOCAL Instituto Dominicano para la Calidad ITESIL Instituto Tecnológico San Ignacio de Loyola de Dajabón SAFE PROJECT – MIDTERM EVALUATION REPORT 6 JAA J.E. Austin Associates, Inc. JAD Junta Agroempresarial Dominicana KII Key Informant Interview(s) l/c/d Liters per cow per day LAD Laboratorio Agroempresarial Dominicano LANAR Laboratorio Nacional de Residuos (de Honduras) LAVECEN Laboratorio Veterinario Central LFS Livestock Field School(s) M&E Monitoring and Evaluation MCC Milk Collection Center(s) MT Metric tons MTE Midterm Evaluation NCBA National Cooperative Business Association OIRSA Organismo Internacional Regional de Sanidad Agropecuaria OTSCORP S.A. Optima, Technical Services PMP Performance Monitoring Plan PO Producer Organization(s) PPP Public-Private Partnership(s) PROGIS Progana Information System RFP Request for Proposals RI Results Indicator(s) SAFE Safe Agriculture/Food Export (project) SME Small and Medium-sized Enterprises(s) SoQ Seal of Quality SPS Sanitary and Phytosanitary SPS Sanitation Performance Standards SRT Self-Reporting Tool TA Technical Assistance TAF Technical Assistance Fund TAMU Texas A&M University (Borlaug Institute) TOR Terms of Reference US$ United States Dollars USDA United States Department of Agriculture USSEC United States Soy Export Council USG United States Government USGC United States Grains Council SAFE PROJECT – MIDTERM EVALUATION REPORT 7 ACKNOWLEDGEMENTS The J.E. Austin Associates, Inc. Evaluation Team wishes to acknowledge the invaluable assistance of NCBA CLUSA staff, both at their headquarters office in Washington DC and in the Dominican Republic, who worked with us collaboratively to complete this Midterm Evaluation of the Safe Agriculture/Food Export (SAFE) Project. In addition, the team wishes to recognize the invaluable assistance of staff at SAFE partner agencies, including the Dirección General de Medicamentos, Alimentos y Productos Sanitarios (DIGEMAPS) and the Dirección General de Ganadería (DIGEGA), as well as at Junta Agroempresarial Dominicana (JAD), the Borlaug Institute at Texas A&M University (TAMU), and Cooperative Resources International (GENEX), all of whom provided critical insights into the subject matter for this evaluation. The team also acknowledges the gracious support of the private business operators and employees, and producers and producer associations in the Dominican Republic that welcomed us into their business places, farms, and homes and shared their time and knowledge with us over the course of our fieldwork. Notwithstanding this support, this report presents an independent evaluation of the SAFE Project, and the team takes full responsibility for any errors, omissions, or oversights that appear within the report. SAFE PROJECT – MIDTERM EVALUATION REPORT 8 EXECUTIVE SUMMARY PROJECT OVERVIEW On September 29, 2015, the United States Department of Agriculture (USDA) Foreign Agricultural Service (FAS) awarded the Safe Agriculture/Food Export (SAFE) project (known locally as PROGANA) to the National Cooperative Business Association (NCBA) CLUSA International (CLUSA) through the Food for Progress (FFPr) program. SAFE is a five-year project in the Dominican Republic, ending November 29, 2020 (activities are currently scheduled to end August 31, 2020). The project is funded through monetization of donated commodities estimated to value US$14,282,200, US$1,753,087 in Commodity Credit Corporation (CCC) funds, and US$176,834 in anticipated interest earned. SAFE implementing partners include the Junta Agroempresarial Dominicana (JAD), the Borlaug Institute for International Agriculture and its Center for Food Safety at Texas A&M (TAMU), and Cooperative Resources International (CRI/GENEX). The project works with 14,406 direct beneficiaries over the life of the project, out of which 13,500 are farmers and about 1,300 are members of other stakeholder groups. Specific objectives fall into two areas: 1) Improve productivity in the livestock (beef and dairy) value chain; and 2) Expand trade of beef and dairy products (which CLUSA staff stress is the more important of the two). EVALUATION OVERVIEW The purpose of this Midterm Evaluation (MTE) was to develop concrete and feasible recommendations to advance and redirect intervention strategies and activities, if and where required. Between January and May 2019, J.E. Austin Associates, Inc. (JAA), an independent contractor based in Washington DC, deployed a two-person evaluation team (ET) to implement this MTE, including a Team Leader and a Dominican Republic￾based Agricultural and Trade Economist. JAA also sub-contracted a Quantitative Research Team based in Santo Domingo to carry out data collection, while home office JAA staff provided remote support to the ET. The MTE assessed the SAFE project in terms of 23 evaluation questions (EQ) under six topics: 1) goal achievement; 2) relevance of project activities; 3) project effectiveness; 4) efficiency of systems and processes; 5) sustainability; and 6) impact. The ET deployed a mixed methods approach, including quantitative and qualitative data collection methods to implement this MTE. Specific methodologies included document review and project indicator analysis prior to fieldwork, as well as quantitative surveys of a statistically significant sample of 365 project beneficiaries, key informant interviews (KIIs) with 72 individuals from six stakeholder groups related to the project, and 13 focus group discussions (FGDs) with functional project beneficiary groups. The ET recorded and analyzed data in real time using sequential and parallel methodologies in order to identify emerging issues for probing questions and to strengthen findings. Specific data analyses methodologies included triangulation, content analysis and trend analysis. SAFE PROJECT – MIDTERM EVALUATION REPORT 9 CONCLUSIONS EQ 1.1 To what extent have the project goals been achieved? The SAFE project has lagged in terms of the number of continuing beneficiaries reached largely due to challenges during initial project years, though the project is on track to meet or exceed targets with regard to the number of cumulative beneficiaries it has reached due to achieving higher numbers in later project years. However, the precise number of beneficiaries SAFE has reached is unclear due to inconsistencies between data sources including project summary reports and the Progana Information System (PROGIS) online M&E platform. Under ‘Increased Agricultural Productivity’ (FFPr SO1), the project is significantly behind in terms of its target of a 50% increase over the baseline of 7.5 l/c/d of milk amongst target beneficiaries (currently the increase stands at only 6%). This may be due to several factors, possibly including over-estimation of baseline value, as well as on￾going drought and early project implementation challenges in TA delivery. Under the second Strategic Objective of ‘Expanded Trade in Agricultural Products (Domestic, Regional, and International)’ (FFPr SO2), the project has met it targets in terms of dairy producer organizations (POs) applying new techniques and technologies, but these improvements have not resulted in significant increases in volumes of Grade A milk handled by these organizations. Also, under the second Strategic Objective, at the time of this MTE, the equivalence process has yet come to fruition i.e. the achievement of certification by the Food Safety Inspection Service (FSIS) that a system for beef inspection that is ‘equivalent’ to the U.S. system, is in place in the Dominican Republic. Notably, sanitation challenges at slaughterhouses have proven greater than anticipated at the time of project development. However, SAFE has implemented a number of key steps toward achievement of this goal, including facilitating an Equivalence Board which convenes key stakeholders in the process, facilitating a market study for Dominican beef products in Puerto Rican markets, facilitating visits by consultants that developed a ‘critical pathway for sanitary improvements at slaughterhouses, and supporting the identification of designated testing labs. The GoDR has responded by committing required investments into inspection services to monitor sanitary conditions and approving the selection of designated labs. As a result, stakeholders express optimism in the process will achieve a positive outcome during the life of the project. However, stakeholders also agree that the pace at which final steps are completed is dependent on the sustained political will of the GoDR to make investments into upgrades of inspection services. EQ 1.2 Which activities have been successful in meeting their targets? Infrastructure: Improve Beef and Dairy Value Chains – As of the end of FY18, SAFE has met targets for disbursing co-investments for beef processing facilities participating in the Equivalence Process through grants to MERCARNE and Suplidora de Carne A&B. Market Access: Facilitate Public-Private Partnerships – The project has met its annual target for this activity and is on track to meet its project target. SAFE PROJECT – MIDTERM EVALUATION REPORT 10 Training: Beef and Dairy Quality Standards – The project has met its annual targets for this activity through audits and training with dairy and beef processing companies. EQ 1.3 How can the change be described since the inception of the project? Initial challenges in technical assistance (TA) delivery resulted in significant lags in progress toward early annual targets in terms of reach and application of new techniques and technologies by individual dairy producers. However, adoption of the livestock field schools (LFS) approach to producer TA delivery and improved coordination of the equivalence process stakeholders since January 2018, has generated progress resulting in optimism for a successful outcome during remaining project years. EQ 1.4 Which activities failed to reach their targets and why? Include analysis of factors that delay/prohibit adequate progress towards results. Capacity Building: Develop Public Information Campaign and Improve Marketing and Branding No dairy processors have yet obtained the Seal of Quality (SoQ) to date; the project has not reached its targets for this activity due to lack of product supply. Two dairy processors are, however, close to obtaining the certification. Capacity Building: Private Sector Agricultural Extension Services – While the project is on track to ultimately meet targets for training of private sector extension agents (reaching about 60% of the project target), to date only one grant has been issued. In addition, recipients of the single grant to date have yet to be deployed as project staff consider whether producers organizations are willing to pay for services adequately. Capacity Building: Producer Associations & Groups –The project has been successful in reaching higher numbers of continuing, new, and cumulative beneficiary POs and has leveraged resources to implement a number of upgrades at milk collection centers (MCCs). However, despite these successes, training has not resulted in increases in delivery of Grade A milk from MCCs to processors. Capacity Building: Promote Improved Policy and Regulatory Framework – To date, the process has yet to reach fruition and no new markets or buyers have been reached at this point. However, in coordination with the Equivalence Board, the project has assisted the GoDR to realize a number of important steps in support of the equivalence process, such as making new budget and staff allocations, as well as with regard to assignation of testing laboratories and investments into inspection services. Financial Services: Facilitate Agricultural Lending and Private/Public Investment – The project lags far behind in terms of the number and value of loans disbursed through its financial institution partners. The underlying factor for the low levels of loans reflects poor rates of application for improved techniques and technologies, which may in turn have resulted from early challenges in targeting and TA delivery methodology. SAFE PROJECT – MIDTERM EVALUATION REPORT 11 Training: Farm Management – Despite the development and dissemination of SAFE Logbooks, or ‘production registers’, the project has lagged behind targets in terms of the number of farmers making decisions based on financial considerations or analysis. Market Access: Facilitate Buyer-Seller Relationships – With the lack of any awards of the SoQ to date, and the ongoing nature of the equivalence process, the project has not yet met targets for this activity. However, as noted, two dairy processors are close to obtaining the SoQ, and there is optimism that the equivalence process will be complete during the life of the project. Training: Improved Agricultural Production Techniques – The project has successfully met targets related to training delivery. It is, however, significantly behind targets for application of improved techniques and technologies, largely due to challenges with TA delivery in early project years. EQ 2.1 Are the project activities in alignment with GoDR priorities and strategies? SAFE project activities align with GoDR priorities with regard to the equivalence process and improvements in dairy productivity. Notably, the Ministry of Agriculture has demonstrated its commitment to project objectives by committing to invest in the expansion of milking parlors on a two-to-one basis in order to improve milk quality through better hand-udder sanitation. EQ 2.2 Do the project activities respond to the aspirations and needs of the project beneficiaries/participants? SAFE project activities have done a ‘fair’ job responding to the aspirations and needs of the project beneficiaries. The project could improve through greater focus on resilience to recurring drought conditions, which are undermining productivity. EQ 2.3 Are the project activities in alignment with USDA/US Government development goals, objectives, and strategies? SAFE project activities align with USDA/USG development goals, objectives, and strategies. However, the project could improve in this regard through increased marketing and branding related to informing the public that USDA is the project donor. EQ 2.4 Are the activities in alignment with the USDA award agreement? SAFE project activities under both objectives are aligned with the USDA award agreement. However, both SAFE and USDA staff pointed out that the pace at which the project will achieve some targets, especially those specifically related to the equivalence process and to dairy productivity targets, is also dependent on the sustained commitment of GoDR stakeholders and the dairy processors themselves. EQ 3.1 To what extent have different actors in the value chain been able to increase quantity e.g., yield/ha and minimize waste since project inception? SAFE measures increase in quantity primarily in terms of changes in l/c/d of milk. To date, the project is significantly behind in terms of its target of a 50% increase over the SAFE PROJECT – MIDTERM EVALUATION REPORT 12 baseline of 7.5 l/c/d of milk amongst target beneficiaries (currently the increase stands at only 6%). Increases have been undermined by recurring drought conditions, and initial challenges in TA delivery that resulted in low application of TA. EQ 3.2 To what extent have all actors in the value chain adopted quality standards? To date, no value chain actors have adopted quality standards. However, two dairy processors are close to meeting requirements for the SoQ. Likewise, the project, through its implementing partner Texas A&M University, has developed a critical path for improving slaughterhouse compliance with HACCP and sanitation standards, and two have actively applied recommended corrective actions. EQ 3.3 To what extent have value chain actors been trained in quality standards? The SAFE project has made ‘fair’ progress in training value chain actors in quality standards. However, progress has been challenging due to the unanticipated scale of sanitary problems in both the beef and dairy sectors, the practice of mixing milk of various standards and selling it as lower grade, and due to limited financial motivation of producers to invest in upgrades due to marginal returns for higher grade milk. EQ 3.4 Are partner organizations providing training to beneficiaries that is tailored to the local context and achieving increased use of agricultural techniques and technologies, increased use of improved post-harvest processing and handling practices, and improved marketing and branding? Most SAFE partner organizations are providing training that is tailored to the local context. Specifically, SAFE implementing partner TAMU focuses their training on addressing weaknesses in the inspection services and sanitary problems in order to advance the equivalence process. In addition, implementing partner CRI/GENEX has adapted their approaches to focus on delivering TA on ‘basic practices’ related to livestock breeding. The transition to the LFS approach, led by staff at SAFE implementing partner Junta Agroempresarial Dominicana (JAD), has improved producer satisfaction with TA and enhanced the capacity of POs to deliver TA to membership. EQ 3.5 Are local government extension officers offering technical support to farmers in new techniques and technologies? To date, most government extension officers, i.e. the MEGALECHE extension unit of the Dirección General de Ganadería, are reported to have generally performed poorly in terms of offering technical support to farmers. EQ 3.6 Have effective linkages been established between different project participants e.g., between producers and processors? Effective linkages have been established between project participants related to the beef value chain through the Equivalence Board, which has realized a number of important steps in the equivalence process through coordination between DIGEGA, DIGEMAPS, the slaughterhouses, and TA providers. SAFE PROJECT – MIDTERM EVALUATION REPORT 13 EQ 4.1 What internal and external factors have influenced the ability of the project to meet expected results and targets? Internal factors that have influenced the ability of the project to meet expected results and targets include staffing challenges, which ultimately resulted in replacement of key staff (notably, the former Chief of Party), as well as initial targeting and TA delivery problems. External factors that have influenced the ability of the project to meet expected results and targets included recurring droughts, which have undermined dairy productivity, have influenced the ability of the project to meet some expected results and targets, especially related to productivity. An additional factor is willingness of the GoDR and slaughterhouses to makes required investments into inspection services and sanitary conditions. EQ 4.2 Do project staffing, training activities and management structures efficiently support the achievement of expected project outcomes? Despite initial staffing challenges, which generated delays and challenges in partner relations, the current project staffing, training activities and management structures efficiently support the achievement of expected project outcomes. The transition to the LFS approach to TA delivery was a key improvement in training activities. EQ 4.3 Does the project resource allocation to activities and to the project’s geographic areas efficiently support achievement of project outcomes? There is no reason to believe that project resource allocation to activities and to the project’s geographic areas do not efficiently support the achievement of expected project outcomes. EQ 5.1 What factors are likely to contribute towards the sustainability of the following issues addressed by the SAFE project: policy environment, financial incentives (i.e., profit motive), access to inputs and information, market access, etc. The factor most likely to result in sustainability is the ability of the project to deliver value chain-wide improvements in productivity and trade expansion. Dairy processors and slaughterhouses that have received value chain-wide assistance are closest to being able to meet targets. EQ 5.2 What factors are likely to hinder sustainability of the following issues addressed by the SAFE project: policy environment, financial incentives (i.e., profit motive), access to inputs and information, market access, etc.? Conversely, the factor most likely to hinder sustainability is incomplete value chain-wide adoption of improvements in productivity and trade expansion. Specifically, incomplete adoption of quality enhancing improved techniques and technologies by dairy producers and MCCs leads to mixed Grade A milk with lower grades that sells at lower prices. Similarly, marginal price improvements for dairy producers, among other factors, are likely to impede sustainability of activities that enhance productivity. SAFE PROJECT – MIDTERM EVALUATION REPORT 14 As noted, both SAFE and USDA assert that a ‘key factor’ in the pace of the equivalence is continuous commitment by the GoDR to allocate resources to inspection services in such a way that they remain adequate to meet the requirements of the U.S. Food Safety and Inspection Service (FSIS). Likewise, slaughterhouses will need to maintain sanitary conditions in line with HACCP requirements. EQ 5.3 What project activities are likely to contribute towards sustainability and is there evidence that project activities or benefits will be sustained after the project ends? Association-wide activities that improve milk quality and thus price are most likely to contribute to sustainability. Furthermore, demonstrated improvements in milk prices are likely to lead to sustainable application of productivity enhancing improved techniques and technologies, and eventually to investments into their adoption. EQ 6.1 What impacts are the project activities having on the project participants, both positive and negative, especially in relation to the expected results and strategic objectives? The impact of project activities on the project participants have been limited in scale due to initial challenges in TA delivery, as well as incomplete value chain-wide adoption of quality enhancing improved techniques and technologies, which has undermined the realization of improvements in prices for milk. Likewise, despite the realization of key steps in the equivalence process, this remains incomplete and no dairy processors have yet obtained the SoQ. EQ 6.2 To what extent can impact be attributed to project activities? New commitments by the GoDR to SAFE objectives can be attributed to project activities that have advanced the equivalence process, as well (to a lesser degree) to the alignment of SAFE activities with GoDR economic development strategies, especially regarding export expansion. EQ 6.3 To what extent have target groups (farmers, processors, vendors, consumers, etc.) adopted promoted techniques that have economic benefits? The extent of target group adoption of promoted techniques that have economic benefits has been low and incomplete. Going forward, achieving illustrative successes in market expansion by processors and slaughterhouses, as well as more comprehensive value chain￾wide work will be key to improved adoption levels. RECOMMENDATIONS AND LESSONS LEARNED Based on the conclusions described above, the ET recommends NCBA CLUSA take the followings steps to advance and redirect SAFE intervention strategies and activities: 1. Resolve inconsistencies in the Progana Information System (PROGIS) platform. 2. Adopt a value chain-wide approach to dairy quality upgrades through five steps: SAFE PROJECT – MIDTERM EVALUATION REPORT 15 Overall, the project should work with partners to develop a phased approach to expanding milking parlors and other quality enhancing techniques and technologies starting initially with higher performing LFSs already linked to dairy processors in order to demonstrate the efficacity of the TA package in terms of price improvements (and cost savings). Subsequently, these partners should incrementally increase focus on resiliency enhancing techniques and technologies, such as improved pastures, in order to expand the quantity of l/c/d. The partners should deliver financing through a mixed approach, with a component of repayable loans combined with a grant component. Throughout, the process should align TA and financing for specific products i.e. the TA addresses borrower weaknesses to reduce risk, create bankable investments, and improve repayment rates. This integration of TA into finance to de-risk lending is known as “blended finance”, while an entity integrating these elements is a Technical Assistance Facility (TAF). The TAF can ultimately receive additional capitalization, including from private investors or development finance institutions (DFIs), in order to expand operations and/or support new activities. Likewise, this entity can also adopt equity models in order to finance upgrades to POs and enterprises. 3. Continue advocating for inspection services and slaughterhouses to progress on the critical path routes toward conformity with the FSIS standards in order to prepare for ‘mock’ audits in line with their recommended next steps in the equivalence process. Focus TA on advancing one slaughterhouse on this ‘critical pathway’ for improving compliance with HACCP and sanitation standards in order to demonstrate the viability of the process to the entire industry while also maximizing the chances of success. SAFE PROJECT – MIDTERM EVALUATION REPORT 16 4. Audit project marketing and branding strategies to better inform the public that USDA is the project donor. Over the course of this MTE, the ET identified the following lessons learned: 1. Dairy development projects need to take into account climatic conditions and focus on resiliency in cases where climate challenges impact the ability of producers to expand production. 2. In order to realize the benefits of upgrades, especially in terms of enhanced quality, dairy sector improvements need to take a value chain-wide approach. 3. In economies characterized by high levels of subsidies, implementers need to demonstrate alternative sustainable models for TA and finance to private and public sector actors, including revolving funds and blended finance. SAFE PROJECT – MIDTERM EVALUATION REPORT 17 1. EVALUATION OVERVIEW 1.1 PROJECT DESCRIPTION On September 29, 2015, the United States Department of Agriculture (USDA) Foreign Agricultural Service (FAS) awarded the Safe Agriculture/Food Export (SAFE) project known locally as PROGANA, in the Dominican Republic, through the Food for Progress (FFPr) program. The National Cooperative Business Association CLUSA International (NCBA CLUSA) is implementing this project over a period of five years ending November 29, 2020 (activities are scheduled to end August 31, 2020). 1 The project is funded through a Commodity Credit Corporation (CCC) donation of 2,120 MT of yellow grease, 2,200 MT of inedible tallow, and 16,100 MT of crude degummed soybean oil (CDSO) for monetization in the Dominican Republic estimated to value US$14,282,200, in addition to US$1,753,087 in CCC funds to assist administration and monitoring, and US$176,834 in anticipated interest earned. 2 Figure 1: SAFE Project Geographic Focus 1 ‘Cooperative Agreement Between the Government of the United States of American and National Cooperative Business Association for the Provision of Agricultural Commodities Through the Food for Progress Act (USDA Award FCC-517-2015-008-00)’ USDA FAS, Washington DC, October 24, 2015) 2 ‘Cooperative Agreement Between the Government of the United States of American and National Cooperative Business Association for the Provision of Agricultural Commodities Through the Food for Progress Act (USDA Award FCC-517-2015-008-00) – Amendment I’ USDA FAS, Washington DC, December 12, 2015 SAFE PROJECT – MIDTERM EVALUATION REPORT 18 SAFE implementing partners include Junta Agroempresarial Dominicana (JAD), which is the primary sub-recipient. JAD is responsible for the provision of extension services and building producer capacity. The Borlaug Institute for International Agriculture and its Center for Food Safety at Texas A&M (TAMU) provides technical support in Sanitary and Phytosanitary (SPS) measures and food safety systems. The Cooperative Resources International (CRI/GENEX) provides technical support in genetics. NCBA CLUSA and its partners implement the SAFE project in the provinces of Santiago Rodríguez, Dajabón, Independencia, San Juan, Monte Plata, La Altagracia, Hato Mayor, El Seibo, María Trinidad Sánchez, Duarte, and Puerto Plata. The project seeks to work with 14,406 direct beneficiaries over the life of the project out of which 13,500 are individuals trained in short-term agricultural sector productivity (farmers); about 1,300 others, including 200 public extension, 250 people in related firms, 40 civil society leaders, 280 extension workers, and 200 individuals trained in trade policy and advocacy, and 330 beef and dairy processor staff. The specific objectives of the project fall into two areas: 1. Improve agricultural productivity in the livestock (beef and dairy) value chain by increasing the use of improved techniques and technologies; improving farm management; increasing the availability of improved inputs and use of financial services; strengthening the capacity of government institutions and key groups; and increasing the leverage of private sector resources. 2. Expand trade of beef and dairy products by adding value to post-production; increasing the adoption of established standards; increasing access to markets; building linkages between buyers and sellers; improving post-harvest infrastructure; increasing the use and efficiency of post-production processes; improving the policy and regulatory framework; and strengthening the capacity of key organizations in the trade sector. To achieve these two main objectives, NCBA CLUSA identified twelve activities to address links in the Dominican beef and dairy livestock value chains, each of which was disaggregated into specific tasks and actions with discreet indicators measured throughout the five years of the project and their end targets (see Annex. A: Results Frameworks and Annex B. Project Activities). 1.2 EVALUATION PURPOSE This midterm evaluation (MTE) assessed the SAFE project in terms of the following topics derived from the USDA project framework: 1. Project accomplishments in relation to goal achievement; 2. Relevance of project activities; 3. Early signs of project effectiveness; 4. Efficiency of systems and processes; 5. Sustainability of project activities; and 6. Impact of project activities. SAFE PROJECT – MIDTERM EVALUATION REPORT 20 Efficiency • What internal and external factors have influenced the ability of the project to meet expected results and targets? • Do project staffing, training activities and management structures efficiently support the achievement of expected project outcomes? • Does the project resource allocation to activities and to the project’s geographic areas efficiently support the achievement of expected project outcomes? Sustainability • What factors are likely to contribute towards the sustainability of the following issues addressed by the SAFE project: policy environment, financial incentives (i.e., profit motive), access to inputs and information, market access, etc. • What factors are likely to hinder sustainability of the following issues addressed by the SAFE project: policy environment, financial incentives (i.e., profit motive), access to inputs and information, market access, etc.? • What project activities are likely to contribute towards sustainability and is there evidence that project activities or benefits will be sustained after the project ends? Impact • What impacts are the project activities having on the project participants, both positive and negative, especially in relation to the expected results and strategic objectives? • To what extent can impact be attributed to project activities? • To what extent have target groups (farmers, processors, vendors, consumers, etc.) adopted the promoted techniques that have economic benefits? Source: ‘NCBA CLUSA International Safe Agriculture/Food Export (SAFE) Project Mid-Term Evaluation (Terms of Reference)’ NCBA CLUSA International (approved by USDA), July 25, 2018 SAFE PROJECT – MIDTERM EVALUATION REPORT 21 2. EVALUATION METHODOLOGY Between January and May 2019, JAA deployed a two-person evaluation team (ET) to implement this MTE, which included in-country fieldwork from February 11-March 4, 2019. The ET consisted of the Team Leader who was responsible for overall supervision of the MTE implementation and deliverables. It also included a Dominican Republic￾based Agricultural and Trade Economist who analyzed indicator and quantitative survey data and assisted with in-country logistics. In addition, JAA sub-contracted a Quantitative Research Team based in Santo Domingo to carry out data collection. Home office staff at JAA headquarters in Washington, D.C. provided remote support to the ET throughout the evaluation process (see Annex C. Evaluation Team and Annex D. Evaluation Timeline). The ET deployed a mixed methods approach, including quantitative and qualitative data collection methods to implement this MTE. Analysis was sequential and parallel in order to both identify emerging issues and to probe and strengthen findings as they emerged. Following fieldwork, the ET triangulated M&E and survey data with qualitative findings to ensure the reliability of conclusions. 2.1 DATA COLLECTION METHODS The mixed methods approach of the MTE combined secondary and primary quantitative and qualitative data collection methods as follows: Document review: Initially, prior to deployment to the field, to inform data collection tools and probing questions, the ET implemented a desk review of SAFE project reporting data, including the SAFE Cooperative Agreement and amendments, the SAFE Baseline Report, Semi-Annual Narrative Activity Reports, success and ‘impact’ stories, as well as related reference works. The ET reviewed the SAFE baseline survey to assess how best to adopt data collection methodologies to ensure comparability of data between studies (see Annex E. Citations and References). Indicator analysis: Prior to departure for the field, the ET implemented an indicator analysis in order to evaluate whether the SAFE project is on track to reach activity and results targets at midpoint. At the time of this MTE, the SAFE project has 101 indicators for monitoring progress in achieving results, comprised of 70 Activity Indicators (AIs) and 31 Results Indicators (RIs) for which the project records data (see Annex G. Indicator Analysis). To implement this analysis, the ET utilized data drawn from PROGIS, an online M&E platform created by NCBA CLUSA in 2018, to provide real time measurement of indicator progress. The ET chose this data source for indicator analysis, as opposed to other project reports, because SAFE staff stated that PROGIS data reflected a more accurate representation of the current project status (SAFE staff had updated PROGIS data as recently as March 15, 2019). However, where suitable, progress reports are also cited. Analysis of relevant indicators is included in corresponding sections of findings below (see 3. Findings and Conclusions). SAFE PROJECT – MIDTERM EVALUATION REPORT 24 The SAFE team provided the ET with an updated beneficiary database on February 12, 2019. Subsequently, the database was cleaned and updated prior to calculation of the survey sample frame (for example, eliminating duplicate entries and updating contact numbers). This updated database included 9,494 beneficiaries, i.e.1,118 or 13% more than the initial database. However, this increase did not impact the validity of the survey sample size. Conversely, the revised data facilitated improved contact information for selected beneficiaries. In terms of purposive replacement, the ET included ten additional beef producers and five additional dual purpose (beef and dairy) producers in the evaluation sample, replacing 15 beneficiaries from the private sector and civil society stakeholder groups for a total sample size of 365. Analysis of relevant survey results are included in corresponding sections of findings below (see 3. Findings and Conclusions and Annex F: Data Collection Tools). Key informant interviews: Following analysis of the initial test surveys in the province of Monte Plata, the ET implemented key informant interviews (KIIs) with a purposively selected sample of members of all project stakeholder groups. The purpose of the KIIs was to better understand project challenges and success and identify lessons learned through probing results of the document review, indicator analysis, and quantitative survey. Based on information obtained through document review, the ET identified the following stakeholder groups for participation in KIIs: 1. SAFE project and partner staff, including NCBA CLUSA headquarters and field-based SAFE staff, as well as staff from SAFE partners Junta Agroempresarial Dominicana (JAD), Cooperative Resources International (CRI)/GENEX Inc., and Texas A&M University (TAMU) (Borlaug Institute); 2. Donor staff, including Washington DC and Santo Domingo-based USDA FAS officials; 3. Government of the Dominican Republic (GoDR) officials, including national and local staff at the Dirección General de Ganadería (DIGEGA) and Dirección General de Medicamentos, Alimentos y Productos Sanitarios (DIGEMAPS), and MEGALECHE extension staff; 4. Financial institution staff, including staff at the Consejo Nacional para la Reglamentación y Fomento de la Industria Lechera (CONALECHE) and Banco Agrícola; 5. Producer Organization (PO) staff, including staff from dairy and beef associations, cooperatives, and federations, and; 6. Enterprises, including milk collection centers (MCCs), dairy and beef processors (including cheese makers), and slaughterhouse staff. Purposive sampling of KII participants included selection according to the likelihood of significant knowledge of SAFE project activities, as well as convenience of access, so as to access the largest number of informants possible over the course of fieldwork within the limited time and personnel resources available to the ET. SAFE PROJECT – MIDTERM EVALUATION REPORT 27 Over the course of this MTE, the ET visited 13 sites including five MCCs, two milking parlors (salas de ordeño), two input and veterinary supply stores, one cheese processing facility, one laboratory, as well as associated beneficiary dairy farms and ranches. 2.2 DATA ANALYSIS For the quantitative survey, the Quantitative Research Team input data into tablets and conducted basic analysis on an ongoing basis throughout survey implementation to identify any emerging trends such as frequency distribution and sub-group comparison via cross-tabulation. Analyses were transmitted to the ET as they became available and were used to inform probing questions during subsequent KIIs, FGDs, and site visits. Upon completion, data was aggregated into an online Excel-based spreadsheet. For each question/statement, the Quantitative Research Team calculated the percentage of interviewed beneficiaries responding and the average score based on points (1–5) for each statement. The higher the score, the more strongly respondents agree with the relevant question. A score of 5 indicates that the interviewed beneficiary strongly agrees with the relevant sentence, while a score of 1 indicates they strongly disagree. For the qualitative data, The ET utilized triangulation of data obtained initially during indicator analysis and the quantitative surveys to develop parallel protocols with same or similar questions across KIIs and FGDs. Throughout the KIIs, FGDs, and site visits, the ET, with oversight from the Team Leader, recorded data directly into MS Word doc-based forms in real time. The ET analyzed feedback daily to identify emerging trends to aggregate findings around common themes and generate further probing questions. Following the conclusion of data collection, the ET aggregated data obtained from the KIIs and FGDs around common themes related to the six areas of EQs. The ET tabulated responses and disaggregated data by province and value chain (beef and dairy) to understand events and outcomes related to the six EQs areas. Data analysis methods used by the ET included: • Triangulation – Subsequent to fieldwork, triangulation enabled the ET to cross-verify and cross-validate findings that emerged from the data sources to identify correlations between findings related to the six EQ areas. Methodological triangulation also enabled the ET to strengthen potential linkages and accuracy of data where results obtained through one method were less conclusive than another method. • Content Analysis– Content analysis entailed the ET’s intensive review of KII and FGD data to identify and highlight notable examples of SAFE successes and challenges that contributed to or hindered progress against indicator targets identified through the indicator analysis. While preliminary data analysis was conducted throughout the evaluation intensive data analysis began immediately following the ET out-briefing at the SAFE office in Santo Domingo and continued until submission of the Draft Evaluation Report. Following submission, the survey datasets collected by the ET will be available to SAFE and USDA in formats as required. SAFE PROJECT – MIDTERM EVALUATION REPORT 28 2.2 POTENTIAL BIASES AND LIMITATIONS The SAFE MTE methodology has a number of potential biases and limitations that have implications for the types of findings and conclusions that can be drawn from this Midterm Evaluation Report. These, and the steps the ET took to mitigate them, include: Positive response (‘halo’) bias: Probing questions regarding income, growth, and outcomes may result in positive response bias i.e. the tendency of respondents to subjectively focus on positive outcomes. The ET mitigated this bias by probing for both successes and challenges throughout each method to develop the most holistic picture possible of SAFE achievements and shortcomings. Selection bias: Selection bias is an inherent risk when implementers help to facilitate contact with project beneficiaries. The ET worked closely with SAFE staff to organize KIIs and FGDs with project beneficiaries. However there remains a risk that SAFE staff selected the most active, responsive, or engaged beneficiaries meaning that the ET may only have heard from key informants or trainees anticipated to report positive experiences. To mitigate the risk of selection bias, prior to launching this MTE, the ET requested that SAFE provide a universal list of stakeholders in advance. Subsequently, the ET identified individuals and sites from the initial sample of partners and beneficiaries provided by SAFE staff. However, the final list of field sites was selected in collaboration with SAFE staff to take into account logistics challenges and maximize exposure to beneficiaries and informants within the limited resources available to the ET. Indirect effects: The MTE methodology and timeframe mean that the ET was obliged to focus primarily on direct project beneficiaries which are the finite list of entities and individuals within each stakeholder group identified. Nonetheless the ET recognizes that there may be additional indirect beneficiaries of certain activities including PO staff, associated and unassociated producers, and enterprises; the evaluation methodology did not allow the ET to evaluate indirect impacts. Sampling limitations: Due to time and resource constraints, the ET was only able to conduct KIIs and FGDs with a purposively selected sample of stakeholders. While the ET worked closely with the SAFE staff to identify a reasonably representative sample set of stakeholders this does not provide a fully comprehensive picture of SAFE performance and results but rather serves to probe around findings derived from the statistically significant quantitative survey. Subjective measurements: Qualitative approaches can result in performance analysis being dependent on the professional opinions and experience of the ET which may leave findings, conclusions, and recommendations open to their subjective interpretation. The ET mitigated this bias through systematic triangulation of findings across stakeholder groups and methods and drew evidence-based conclusions and recommendations based on the data rather than on their professional experiences. In addition, where possible the ET sought out the professional opinions of relevant skilled personnel to collaborate and review findings and conclusions to improve their accuracy and soundness. SAFE PROJECT – MIDTERM EVALUATION REPORT 32 During the baseline study in 2016, a survey of about 500 farms, about two thirds of which are now SAFE beneficiaries, found that dairy productivity ranged between 2.1- 12.3 l/c/d with an average of 7.5 l/c/d. However, perversely, the following year, 2016 productivity averaged just 6.4 l/c/d, 15% below the baseline as a result of reduction in pasture due to drought. Subsequently, by the end of FY 2018, productivity increased to about 7.725 l/c/d, a 3% increase over baseline representing a 47% shortfall below the project target. In FGDs with dairy producers, participants cited ‘normal’ productivity figures between about 2-7 l/c/d. However, when asked about challenges to productivity KII and FGD respondents almost unanimously cited drought related to El Niño as a dominant long￾term exogenous condition, significantly reducing productivity every two years. 10 Background research confirmed these statements. The Emergency Response Coordination Center (ERCC) regional climate maps showed ‘severe’ and ‘extreme’ dry conditions throughout SAFE project areas in 2016, and the Caribbean Drought & Precipitation Monitoring Network (CDPMN) described ‘long-term drought conditions’ throughout the island of Hispaniola in 2018-19.11 One PO leader stated that drought conditions were “the most urgent issue (facing dairy farmers). If we managed the resources, we would put them all into assistance for affected producers.”12 Figure 2: EERC Haiti, Dominican Republic - Drought Situation Map 10 For example, FGD, dairy producers, San Juan de la Maguana, February 22, 2019 11 ‘Caribbean Drought Bulletin’ Volume V, Issue10, Caribbean Drought & Precipitation Monitoring Network (CDPMN), March 2019 12 KII, PO staff, San Juan de la Maguana, February 22, 2019 SAFE PROJECT – MIDTERM EVALUATION REPORT 33 The SAFE project addresses ‘Increased Agricultural Productivity’ (FFPr SO1) through two Results. The first is ‘Increased Use of Improved Agricultural Techniques and Technologies’ (FFPr 1.2), which includes three Sub-Results and activities as follows: FFPr 1.2.1: Increased Availability of Improved Inputs; Activity: Inputs: Develop agro-dealers, and/or other input suppliers. FFPr 1.2.3: Increased Use of Financial Services; Activity: Financial Services: Facilitate Agricultural Lending and Private and/or Public Investment. FFPr 1.2.4: Increased Knowledge by Farmers of Improved Agricultural Techniques and Technologies; Activity: Training: Improved agricultural production techniques. SAFE staff stated that activities under this result present producers with assistance ‘to improve productivity and access required inputs and financing to obtain inputs’.13 Specifically, the project promotes a techniques, technologies, and practices for pasture and farm management, milking and milk handing, reproduction and genetics, and animal health and well-being. Through the grants program and budget, the project also offers producers a programa de coinversión productivo (PCP), which is a specific set of inputs supplied to qualifying participants of the LFS based on regular attendance. The LFSs promote the entire TA package, but what the farmers receive is an input package of pasture seeds, fertilizers, jerrycans, Embrapa kits (a Brazilian-designed unit used to clean teats and improve hygiene during milking), and other inputs with a total value of US$500-700 per producer. The project delivers this TA through its partnership with the MEGALECHE extension team, a program under DIGEGA dedicated to the development of the livestock sector created by the GoDR in 2001. MEGALECHE links producers to relevant research through a network of farm schools and extension agents distributed throughout the country. In addition, CONALECHE provides loans at heavily subsidized rates to associated small-scale dairy producers (6% annual interest for eight to ten-year loans of up DOP 1.5 million). CONALECHE also facilitates pre-approval of co-financing with Banco Agrícola for larger loans at 8% annual interest.14 The GoDR established BAGRICOLA in 1945 to support primary production. The bank offers similarly subsidized loans (8% annual interest for production loans and 9.5% for marketing loans with loan sizes up to DOP 380 million). Under ‘Increased Availability of Improved Inputs’, the project implements the ‘Inputs: Develop agro-dealers, and/or other input suppliers’ activity through coordination of training events on the use of seeds for improved pasture and veterinary supplies by inviting agro-dealers located in SAFE areas to participate in trainings to promote adequate use of inputs. In some cases, agro-dealers are affiliated with specific POs and operate on the premises of association-operated MCCs. However, the number of these affiliated agro-dealers is ‘limited’ and SAFE staff report that participation by independent agro-dealers has often been ‘passive’.15 13 KII, SAFE project and partner staff, Santo Domingo, February 22, 2019 14 KII, financial institution staff, Santo Domingo, February 28, 2019 15 KII, SAFE project and partner staff, Santo Domingo, February 22, 2019 SAFE PROJECT – MIDTERM EVALUATION REPORT 34 Under the sub-result ‘Increased Use of Financial Services’, SAFE implements the activity ‘Financial Services: Facilitate Agricultural Lending and Private and/or Public Investment’ through its facilitation of co-investments with CONALECHE or other institutions. It deploys a US$1.5 million grant fund (which includes targets of US$50,000 for slaughterhouse SPS improvements, US$264,000 for MCC infrastructure improvements, and US$270,000 for Private Sector Extension services. The remainder is for miscellaneous grant expenses from which the PCP was created in October 2018 to foster application of technologies. 16 To date, the project has also co-financed cooling tanks and two sets of solar panels at PO-operated MCCs to reduce operating costs derived from dependence on the subsidized Dominican power grid, which is prone to outages in rural areas. Staff at BAGRICOLA reported that co-investments are ‘useful’ in that they facilitate loans in excess of their maximum loan amount.17 Finally, under ‘Increased Knowledge by Farmers of Improved Agricultural Techniques and Technologies,’ the project implements ‘Training: Improved agricultural production techniques’ through its partnership with the MEGALECHE extension team. During the initial two project years, TA was implemented in classrooms-type environments. However, in KIIs and FGDs, respondents described numerous problems with MEGALECHE performance related to ‘lack of resources’ (especially fuel for vehicles) and ‘motivation’. 18 Several respondents stated that some MEGALECHE staff delivered TA ‘only by phone’ and described ‘forging signatures’ on training reports.19 One respondent stated that ‘there is a lot of dead wood’ at MEGALECHE’.20 In response, in July 2018 the project adopted farmer field schools (FFSs) as ‘livestock field schools’ (LFS) to prioritize ‘practical’ approaches to TA delivery (see 3.5 Effectiveness).21 SAFE Farm Follow-Up Reports in February 2019 surveyed approximately 900 beneficiary producers across SAFE project areas. These surveys showed that producers applying one or more of the techniques promoted by SAFE had achieved higher average productivity rates of 7.97 l/c/d as compared to producers not adopting the package (5.9 l/c/d).22 However, SAFE staff note that there are ‘significant regional variations’ that must be taken into account when prescribing the specific use of agricultural techniques and technologies, especially types of improved pasture and the intensity of paddock rotation. These variations must also take into consideration herd size and breed types as well as optimal types of pasture under various drought conditions. As a result, producers will continue to rely on recommendations from qualified technicians for site-specific application of TA topics.23 16 KII, SAFE project and partner staff, Santo Domingo, March 1, 2019 17 KII, financial institution staff, Santo Domingo, February 28, 2019 18 For example, FGD, association staff, Arroyo Ancho, February 19, 2019, KII, financial institution staff, Santo Domingo, February 28, 2019, and USDA staff, Santo Domingo, February 28, 2019 19 For example, FGD, association staff, Arroyo Ancho, February 19, 2019 20 KII, financial institution staff, Santo Domingo, February 28, 2019 21 See ‘Farmer Field School (FFS) Manual’ FAO, Special Framework of Assistance, St. Lucia, 2006 22 ‘Visitas a Fincas y Técnicas; October 18-February 19, 2019’ NCBA CLUSA, Santo Domingo, Dominican Republic, March 2019 23 KII, SAFE project and partner staff, Santo Domingo, March 22, 2019 SAFE PROJECT – MIDTERM EVALUATION REPORT 42 Finally, ‘Financial Services: Facilitate Agricultural Lending and Private and/or Public Investment’ registered poor performance under AI 14 ‘Number of loans disbursed as a result of USDA assistance’ with only 3% of the target number of loans was reached at project mid-point (see above). However, the project reports several promising achievements during the most recent project period with regard to co-investments with enterprises and POs. For example, In KIIs with SAFE staff respondents reported that during the second part of FY18 the project began implementing a previously established MOU with IFAD under which it facilitated PRORURAL funding for 11 livestock POs for infrastructure improvements with the POs providing a match worth a total of US$18,852.56. In addition, SAFE implemented a co-investment with four POs that provided matches worth an average of US$6,720. Finally, two slaughterhouses, AGROCARNE and MERCARNE, provided US$4,348 in financing as part of a new agreement signed with the Ministry of Public Health for support related to the equivalence process, in addition to US$727,584 in infrastructure investments they made in 2018, largely in relation to the ‘critical routes’ developed by SAFE implementing partner TAMU (these are required for compliance with equivalence standards for an anticipated FSIS audit of the inspection system). The second Sub-Result under ‘Expanded Trade in Agricultural Products (Domestic, Regional, and International)’ (FFPr SO2) is ‘Increased Access to Markets to Sell Agricultural Products’ (FFPr 2.2), which has one related activity: FFPr 2.2.2: Improved Linkages Between Buyers and Sellers Activity: Market Access: Facilitate Buyer-Seller Relationships. The latest Performance Report Summary describes this activity as meeting 70% of its annual target value of sales and 37% of its project target. Only 9% of its target was met for volume of sales (7.6 million liters of milk) as a result of producer, MCC and processor training, and facilitation of buyer-seller meetings. 38 This includes a meeting between an international buyer (Cooperativa Dos Pinos) and two POs during FY18.39 However, the SoQ and equivalence process remain in progress (see below). In a related activity in FY18, the project facilitated a market study of the Puerto Rican beef market to obtain insights on beef export requirements and potential demand for Dominican beef. This was conducted by ASOCARNE and Professor of Texas A&M University (TAMU). Early conclusions of this study found that Dominican producers have an advantage in the Puerto Rico because Nicaragua exports frozen beef. In contrast, the Dominican Republic can export ‘manufacturing trimmings’ (which would not compete with U.S. prime cut exports) at a lower cost via a scheduled container ferry service between Santo Domingo and San Juan.40 38 ‘Performance Report Summary – April 1st, 2018 – September 30, 2018’ NCBA CLUSA, Santo Domingo, Dominican Republic, November 15, 2018 39 Ibid 40 KII, enterprise, February 13, 2019 SAFE PROJECT – MIDTERM EVALUATION REPORT 43 However, in KIIs, experts at TAMU stated that developing beef quality that could meet demand would require improved farm management as there is currently no quality incentive for prime cuts and beef producers are paid by weight. 41 TAMU staff stated that to develop quality cut exports the Dominican Republic would also eventually need a quality certification regime; SAFE staff does not foresee this in the near future (for further details, see Annex J: Staff Notes in Equivalence).42 The SAFE project also implements three Foundational Results, and four related activities, under ‘Expanded Trade of Agricultural Products (Domestic, Regional, and International)’ (FFPr SO2) primary related to the SoQ and equivalence process: FFPr 2.4.2: Improved Policy and Regulatory Framework; Activity: Capacity Building: Promote Improved Policy and Regulatory Framework Activity: Market Access: Facilitate Private and/or Public Partnerships FFPr 2.4.4: Improved Capacity of Key Organizations in the Trade Sector Activity: Capacity Building: Producer Groups FFPr 2.4.5: Increased Leverage of Private-Sector Resources; Activity: Market Access: Facilitate Private and/or Public Partnerships Under these Sub-Results, key RIs related to the activity ‘Capacity Building: Promote Improved Policy and Regulatory Framework’ include ‘Number of policies, procedures, regulations and/or administrative procedures in each of the following stages of development (stage 5)’ and ‘Number of regulatory policy and regulatory framework meetings for debate and consultation with stakeholders’. The most recent Performance Report Summary states that progress toward annual project targets stands at ‘133% and 131% of annual project targets respectively’.43 The report also states that for FY18 the project exceeded annual targets for the indicators ‘Number of training sessions in trade policy analysis and advocacy provided for individuals’ by 150% and ‘Number of individuals trained in policy analysis and advocacy’ by 274%, ‘representing 50% and 72% of LOP targets, respectively’.44 In KIIs with SAFE staff, respondents reported that the focus of policy work has been related to the equivalence process. With no certifications to date, the project has failed to meet the target of five processors dairy processors obtaining the SoQ by the end of FY 2018. However, the project has conducted a number of audits of conditions at dairy processing plants and the Dominican Ministry of Health conducts regular inspections, though these regularly find conditions at the majority of facilities falling far short of existing requirements.45 41 KII, SAFE project and partner staff, Santo Domingo, March 1, 2019 42 Ibid 43 ‘Performance Report Summary – April 1st, 2018 – September 30, 2018’ NCBA CLUSA, Santo Domingo, Dominican Republic, November 15, 2018 44 Ibid 45 For example, see Aquino, Felix ‘Informe Inspección a Queso Don Manuel (Higüey)’ DIGEMAPS, Santo Domingo, August 28, 2018 SAFE PROJECT – MIDTERM EVALUATION REPORT 46 In addition, KII respondents also stated that identifying LANAR as the lab conducting chemical residue testing has resolved a critical impasse in the equivalence process. During initial project years, TAMU staff described challenges working with LAVECEN. For example, consultant recommendations regarding required improvements were not adopted, resulting in delays. According to TAMU staff, there were significant delays due to DIGEMAPS ‘waiting on a loan from the Inter-American Development Bank (IADB)’ before acting, while ‘updates must be made on a daily basis’.53 With LAVECEN’s role now limited to forwarding samples to LANAR in Honduras via OIRSA, through a new technique developed by LANAR staff, this problem has been circumvented. However, TAMU staff also noted that ‘these labs now need to be prepared for an audit’ and they have ‘not yet looked at the other labs’.54 In KIIs, TAMU staff reported similar challenges working with the DIGEMAPS inspection services responsible for assessment of slaughterhouse sanitation, describing them as ‘very defensive’.55 These respondents noted that ‘a lot of trainings were cancelled, often because the DIGEMAPS did not approve the trainers’. However, they stated that ‘this changed beginning in February 2018’, and now consultants are ‘better able to support improvements and provide capacity building, including remotely via WhatsApp’. For example, in January 2019, a consultant contracted by TAMU visited the Dominican Republic and provided inspectors with assistance ‘as if he was an FSIS auditor’. This consultant also helped them adopt the inspection forms so they would be more concise and similar to the ones used by FSIS in the U.S. (a challenge now is for inspectors to correctly implement the new procedures and forms developed with assistance from TAMU consultants to better understand the FSIS regulations, and to interact with the quality control staff of the corresponding establishments). TAMU also stated that initially the number of inspectors was insufficient. However, in 2018, the GoDR provided DIGEMAPS with funding to hire more than 30 new staff to work in the slaughterhouses, including medical veterinarians and inspectors. These people are already in place and SAFE subsequently provided them with training on the use of the programs and forms required by FSIS.56 In KIIs, TAMU staff stated that “changes took a while and time was lost. It took about a year and a half to get things going” noting that this was largely because “nobody understood the severity of the (prevailing sanitary) issues at the time the (SAFE) proposal was designed”. 57 However, visited the Dominican Republic in January and February 2019 to evaluate the organization of the DIGEMAPS central office, and conduct trainings for official inspectors on how to evaluate inspection reports for non-conformities. SAFE now plans ‘mock audits’ three weeks before the FSIS official audit is scheduled. 58 53 KII, SAFE partner staff, Skype, March 1, 2019 54 Ibid 55 Ibid 56 KII, Government of the Dominican Republic, Santo Domingo, February 28, 2019 and KII, SAFE staff, Santo Domingo, March 1, 2019 57 KII, SAFE partner staff, Skype, March 1, 2019 58 KII, SAFE staff, Skype, March 1, 2019 SAFE PROJECT – MIDTERM EVALUATION REPORT 47 In KIIs with SAFE and DIGEMAPS staff, respondents recognized that there was considerable skepticism at slaughterhouse management that the equivalence process will come to fruition. The last attempt to establish equivalence ended unsuccessfully in 2011. Therefore, respondents felt that going forward the strategy should be to focus on one to three slaughterhouses, so that they can ‘demonstrate a success story’. They believe this will motivate other slaughterhouses to make required investments. 59 In contrast, in KIIs TAMU staff stated that, although they have developed a seven-month ‘critical route’ for the three slaughterhouses to improve HACCP and sanitation programs the board should focus on only one slaughterhouse ‘because if there are too many, FSIS will be more likely to find a problem’.60 SAFE and DIGEMAPS staff stated that two slaughterhouses are close to ready to export (MERCARNE in Santo Domingo and AGROCARNE in La Romana) but ‘others need assistance’. In summary, they stated that “if one can export, then the others will be confident that we have the capacity to export, and the other slaughterhouses will follow (making the investments needed)”.61 3.2 GOAL ACHIEVEMENT – CONCLUSIONS EQ 1.1 To what extent have the project goals been achieved? The SAFE project has lagged in terms of the number of continuing beneficiaries reached largely due to challenges during initial project years, though the project is on track to meet or exceed targets with regard to the number of cumulative beneficiaries it has reached due to achieving higher numbers in later project years. However, the precise number of beneficiaries SAFE has reached is unclear due to inconsistencies between data sources including project summary reports and the PROGIS online M&E system. With regard to the first goal of ‘Increased Agricultural Productivity’ (FFPr SO1), the project is significantly behind in terms of its target of a 50% increase over the baseline of 7.5 l/c/d of milk amongst target beneficiaries (currently the increase stands at only 6%). Given the low application rates of the TA package, and that beneficiaries that have applied the TA package have achieved average rates of only 7.97 l/c/d, and under the prevailing drought conditions, it is unlikely that the project will achieve this goal over the remaining life of the project. Under the second Strategic Objective of ‘Expanded Trade in Agricultural Products (Domestic, Regional, and International)’ (FFPr SO2), the project has met it targets in terms of dairy POs applying new techniques and technologies, but these improvements have not resulted in significant increases in volumes of Grade A milk handled by these organizations. Likewise, despite the implementation of a public information campaign to promote the SoQ for dairy products, no processors have yet obtained this seal due to reluctance to make required investments (though at least two cheese manufacturers are likely to obtain the seal during the remainder of the project life). 59 KII, Government of the Dominican Republic, Santo Domingo, February 28, 2019 and KII, SAFE staff, Santo Domingo, March 1, 2019 60 KII, SAFE partner staff, Skype, March 1, 2019 61 KII, Government of the Dominican Republic, Santo Domingo, February 28, 2019 and KII, SAFE staff, Santo Domingo, March 1, 2019 SAFE PROJECT – MIDTERM EVALUATION REPORT 48 Finally, also under the second Strategic Objective, despite the completion of a market study that finds opportunities for Dominican beef products in Puerto Rican markets, and several positive steps involving submission of the SRT, identification of labs, and strengthening of inspection services, HACCP systems and sanitation programs at participating slaughterhouses, the equivalence process has yet to come to fruition, and no new markets for beef exports have yet been entered. Next steps in this process entail preparation for audits of inspection services and slaughterhouses. EQ 1.2 Which activities have been successful in meeting their targets? Infrastructure: Improve Beef and Dairy Value Chains – As of the end of FY18, the SAFE project has met its targets having disbursed co-investments totaling US$20,580.79 for beef processing facilities participating in the Equivalence Process to MERCARNE and Suplidora de Carne A&B slaughterhouses. In addition, the co-investment with MERCARNE and investments made by AGROCARNE Central Romana as part of the critical path route developed by SAFE implementing partner TAMU, represent key steps in the Equivalence Board strategy to support at least one slaughterhouse to meet FSIS inspection requirements in order to provide a success story that will motivate others to make required investments. Market Access: Facilitate Public-Private Partnerships – The project has met its annual target for this activity and is on track to meet its project target, as a result of progress toward the equivalence process related to identification of participating laboratories to which SAFE provides assistance. Training: Beef and Dairy Quality Standards – The project has met its annual targets for this activity through audits and training with dairy and beef processing companies. Given the likelihood that one or two companies will successfully obtain the SoQ and the equivalence process will reach fruition, it is likely the project will successfully reach project targets as well. EQ 1.3 How can the change be described since the inception of the project? Initial challenges in TA delivery resulted in significant lags in progress toward early annual targets in terms of reach and application of new techniques and technologies by individual producers. Likewise, challenges in TA delivery to laboratories and inspection services related to the equivalence process resulted in delays to the completion of this process. Both of these factors undermined progress toward targets related to both beef and dairy processer companies and POs. However, adoption of the LFS approach to producer TA delivery and improved coordination of equivalence process stakeholders since staff changes in January 2018, including investments in inspection services by the GoDR, has generated optimism for improved results over the remaining project years. Likewise, intensive value chain work with both producers and processors has generated progress toward the development of the SoQ. However, the impact of recurring drought conditions and the limited remaining project timeframe present challenges to the full realization of project targets. SAFE PROJECT – MIDTERM EVALUATION REPORT 49 EQ 1.4 Which activities failed to reach their targets and why? Include analysis of factors that delay/prohibit adequate progress towards results. Capacity Building: Develop Public Information Campaign and Improve Marketing and Branding – With no dairy processors obtaining the SoQ to date, the project has failed to reach targets for this activity due to lack of product supply. Furthermore, the high cost of improvements and lack of recourse by DIGEMAPS against processors that fail to meet sanitary standards has generated reluctance to make improvements. Nonetheless, near completion of SoQ requirements by at least two companies has demonstrated that intensive value chain work with processors can result in progress toward targets. Capacity Building: Private Sector Agricultural Extension Services – While the project is on track to ultimately meet targets for training of private sector extension agents (reaching about 60% of the project target), to date only one grant has been issued under this activity, and agents are yet to be deployed due to PO capacity to provide required financial resources for the maintenance of agents. The transition to an LFS approach may improve PO demand for private TA in subsequent project years. Capacity Building: Producer Associations & Groups –The project has been successful in reaching higher numbers of continuing, new, and cumulative beneficiary POs and has leveraged resources to implement a number of upgrades at MCCs. However, although the project has reached 109% of its target for the number of suppliers delivering grade A milk to the MCCs, training has not resulted in increases in delivery of Grade A milk to processors. This is likely the result of early challenges in meeting targets for individual beneficiary numbers, and low application rates of improved techniques and technologies by these individuals. An additional factor may also be poor financial motivation due to low marginal benefits of quality improvements, which undermines economic incentives. Capacity Building: Promote Improved Policy and Regulatory Framework – As noted, in coordination with the Equivalence Board, the project has realized a number of important steps in equivalence process. However, to date, the process has yet to reach fruition and no new markets or buyers have been reached. Financial Services: Facilitate Agricultural Lending and Private/Public Investment – The project lags far behind in terms of the number and value of loans disbursed through its financial institution partners. The underlying factor for low levels of loans reflects poor rates of application for improved techniques and technologies, may in turn have resulted from early challenges in targeting and TA delivery methodology. Training: Farm Management – Despite the development and dissemination of SAFE Logbooks, or ‘production registers’, the project has lagged behind targets in terms of the number of farmers making decisions based on financial considerations or analysis. This lag reflects the low number of individuals reached in early project years, as well as low application rates of improved techniques and technologies. An additional consideration is producer concern that records will lead to higher rates of taxation. However, the interest of farmers that have correctly utilized the registers suggests that producer-led training on use through LFSs may lead to improved future adoption. SAFE PROJECT – MIDTERM EVALUATION REPORT 52 Additionally, in several FGDs with dairy producers, respondents specified that their priority was effectively dealing with recurring drought conditions, which had decimated productivity due to reduction in pasture and water for cattle, and stated that the project should do more to assist them in this regard (although producers have resisted investments). 70 Additionally, in two FGDs with federation and PO staff, respondents stated that they “knew what the priorities were” i.e. federation and PO-associated businesses, and they would have preferred resources were ‘provided directly to them’.71 In KIIs with Santo Domingo-based USDA staff, respondents confirmed that SAFE project activities align with USDA/US Government development goals, objectives, and strategies, especially with regard to export promotion related to the equivalence process, which demonstrates the effectiveness of CAFTA-DR.72 One respondent stated that the project was important in that it provides ‘leverage in achieving policy objectives with the GoDR’.73 However, these respondents also criticized the project as not having done enough to represent SAFE as a USDA project, especially in early years, and requested that SAFE improve marketing and branding to better ‘inform the public that USDA is the project donor’.74 The same respondents confirmed that the project activities align with the project award agreement with USDA, stating that they had sufficient input into design during the project development phase, as well as ongoing interaction with SAFE staff over the course of subsequent implementation, including amendments.75 However, they also stated that interaction in initial project years was ‘difficult’, though this had ‘improved in recent years’, especially subsequent to ‘staff changes in 2018’.76 Likewise, these respondents noted that ‘SAFE can only do so much’ and progress (with regard to the equivalence process) is ‘really up to the GoDR’, which ‘ultimately determines the pace’ through decisive decisions (in areas such as final selection of laboratories and investments in upgrades to the inspection service). 77 3.4 RELEVANCE – CONCLUSIONS EQ 2.1 Are the project activities in alignment with GoDR priorities and strategies? SAFE project activities align with GoDR priorities with regard to the equivalence process and improvements in dairy productivity, and the Ministry of Agriculture is prepared to invest in the expansion of milking parlors on a two-to-one basis in order to improve milk quality through better hand-udder sanitation. 70 Ibid 71 FGD, federation staff, Montellano, February 19, 2019 and PO staff, San Juan de la Maguana, February 22, 2019 72 KIIs, donor (USDA) staff, Santo Domingo, February 28, 2019 73 Ibid 74 Ibid 75 Ibid 76 Ibid 77 Ibid SAFE PROJECT – MIDTERM EVALUATION REPORT 56 The project has implemented training on standards by focusing on GMPs and HACCP principals for livestock, as well as training and audits of sanitary conditions at cheese manufacturing facilities and slaughterhouses. By the end of the most recent reporting period, SAFE reported implementing training for ‘20 dairy processor staff on GMP and 176 beef processor staff on HACCP principles, as a prerequisite for Equivalence Standards.83 During the same period, the project reported carrying out 11 audits of dairy processing facilities (including Quesería Rottis), related to the SoQ in partnership with DIGEMAPS. 84 Finally, the project reported implementing ‘103 sessions across the (targeted) 11 provinces training 2,898 farmers in milk handling and milk handling practices representing 111% of the annual target’. In topics related to quality standards, quantitative survey respondents ranked activities ‘fair’, including ‘milk and milk handling’ (score: 3.80), ‘use of milking parlors (sheds)’ (score: 3.32), ‘sanitary measures’ (score: 3.88), ‘export compliance measures’ (score: 3.46), ‘milk handling practice (or beef carcass handling)’ (score: 3.77), and ‘GAP, GMP and HACCP’ (score: 3.44). However, numerous KII respondents described prevailing sanitary conditions as more challenging than initially thought. As noted, in KIIs with SAFE partners, TAMU staff reported that ‘“nobody understood the severity of the (prevailing sanitary) issues at the time the (SAFE) proposal was designed”. 85 Likewise, DIGEMAPS inspections have found that conditions at the majority of facilities fall far short of existing health certification requirements.86 Finally, as described by SAFE staff, MCCs mix Grade A milk with other milk qualities and sell at the lower grade,87 while financial motivation remains a challenge, with markets providing only marginal increases in price for Grade A milk (see 3.1 Goal Achievement – Findings). Nonetheless, on a positive note, in FGDs, dairy producers described the use of milking parlors, which is fundamental to sanitary improvement in the dairy value chain, as resulting in several notable and significant positive improvements, including ‘increasing productivity’ and ‘ease of milking’.88 As noted, recognizing this fundamental step in improving dairy quality, the Minister of Agriculture stated that his office was prepared to provide a ‘two-to-one match’ for SAFE co-investments into the construction of producer milking parlors (see 3.1 Goal Achievement – Findings).89 Finally, in KIIs, several project stakeholders described the SAFE project as motivating the emergence of a ‘culture of food safety’.90 These respondents stated that processors, especially in the dairy sector, were aware that consumers were increasingly taking safety into consideration in their purchasing decisions. 83 ‘Performance Report Summary – April 1st, 2018 – September 30, 2018’ NCBA CLUSA, Santo Domingo, Dominican Republic, November 15, 2018 84 Ibid 85 KII, SAFE partner staff, Skype, March 1, 2019 86 For example, see Aquino, Félix ‘Informe Inspección a Queso Don Manuel (Higüey)’ DIGEMAPS, Santo Domingo, August 28, 2018 87 KII, SAFE project and partner staff, Santo Domingo, February 14, 2019 88 FGD, dairy producers, Villa Riva, February 18, 2019 and FGD, dairy producers, Montellano, February 19, 2019 89 KII, Government of the Dominican Republic, Santo Domingo, March 4, 2019 90 For example, KII, SAFE partner staff, Skype, March 1, 2019 SAFE PROJECT – MIDTERM EVALUATION REPORT 57 The SAFE project has provided TA directly, as well as through partner organizations. Key partner organizations that provide TA include TAMU and CRI/GENEX, as well as the MEGALECHE extension unit and partner beef and dairy POs, cooperatives, and producer federations. Specifically, SAFE has mobilized TA to beef and dairy processors, as well as to DIGEGA and DIGEMAPS, for improved post-harvest processing and handling practices, and improved marketing and branding specifically focused on inspection services and producer capacity to meet standards. TAMU provides TA through consultants affiliated with the Borlaug Institute, which was established at TAMU in 1984 and is specialized in design and implementation of agricultural development and training programs. Under its mandate, the institute has implemented similar projects in a number of countries in Latin America and the Caribbean, including El Salvador, Guatemala, and Panama.91 TAMU focuses on beef exports (equivalency), but also is mandated to work on dairy standards. As noted, in KIIs with TAMU staff, respondents reported that, “nobody understood the severity of the (prevailing sanitary) issues at the time the (SAFE) proposal was designed” and initially recommendations were not implemented resulting in a loss of time, though this has improved since 2018 as labs were identified and DIGEMAPS received supplemental funding for additional inspectors (see 3.1 Goal Achievement – Findings).92 Likewise, TAMU consultants initially assessed conditions at several cheese processors and ‘found deficiencies in the cold chain’ amongst other significant problems with sanitary conditions, which producers were reluctant to correct because of cost and because ‘DIGEMAPS did not take recourse against them’. 93 As noted, these respondents pointed that the project agreement also mandates a focus on the dairy sector standards, but is behind in this regard.94 In KIIs, CRI/GENEX staff described themselves as a private company focused on ‘support of genetic development for livestock cooperatives.’95 The company’s role in the SAFE project is promoting the increased use of agricultural techniques and technologies related to livestock breeding. Respondents at CRI/GENEX stated that they initially implemented an assessment of conditions to put together a training program on farm management focused on reproductive issues (especially timing of reproduction). This training was targeted to MEGALECHE extension agents but has ‘now transitioned to working more directly with farmers.’ Although training was originally carried out by ‘master and PhD-level consultants’, which is ‘how CRI/GENEX usually works’, “training has transitioned to more basic practices with farmers, such as looking for ‘heat’ every 22 days, and understanding differences between dairy, beef, and dual-use cattle, which is not really a job for high-level staff”.96 The present trainer, who began following staff changes at SAFE in 2018, was originally a Peace Corps volunteer in the Dominican Republic. 91 https://borlaug.tamu.edu 92 KII, SAFE partner staff, Skype, March 1, 2019 93 Ibid 94 Ibid 95 KII, SAFE partner staff, Skype, March 1, 2019 96 Ibid SAFE PROJECT – MIDTERM EVALUATION REPORT 58 KII respondents at CRI/GENEX stated that ‘initially, there were a lot of stops and starts as SAFE restructured’ and likewise, ‘poor logistics negatively impacted the initial trainings (for example, at some trainings there were no cows available on which to demonstrate practices)’.97 However, these staff stated logistics has improved and ‘about 146 producers received TA last summer over two series of trainings in four different areas’ and a more recent training had ‘about 40 people at two locations (80 total)’.98 They found that working with associations to set up ‘artificial insemination (AI) circuits with multiple routes has been effective’.99 CRI/GENEX promotes each association having stored semen and an AI technician. The ‘idea of a route is that each farm has 10-20 cows, and each association has a technician that can be called in when a cow needs to be bred’. These staff stated that ‘association leadership is pretty motivated, though ‘at the association-level, this is in early stages.’100 The CRI/GENEX staff were not familiar with LFS and has rather worked with demonstration farms. In the quantitative survey carried out by the ET, respondents ranked the effectiveness of activities associated with ‘genetics and animal reproduction’ as ‘fair’ (score: 3.41), with 93% of beneficiaries responding. Reflecting noted challenges in TA delivery with the MEGALECHE extension unit, the most recent project report states that SAFE has reached agreement with DIGEGA to begin replacement of under-performing staff (see 3.1 Goal Achievement – Findings), i.e. ‘One ongoing challenge of the project has been the lack of performance from the public extension workers (MEGALECHE). After a series of performance evaluations, it was determined in conjunction with MEGALECHE management that underperforming ones will be replaced’. 101 In FGDs with beef and dairy PO and cooperative staff, as well as with producer federation staff, the majority of respondents stated that their organizations provide TA to their membership through regular ‘meetings’, though the quality of this assistance could not be verified.102 With regard to TA delivery, in FGDs, PO membership described positively the transition to the LFS approach, recognizing its focus on ‘practical’ training. 103 A number of respondents specifically appreciated the way in which LFSs allowed them to receive training at ‘the times that worked best for them (according to their schedules)’.104 However, as noted, SAFE staff reported that, to date, there is concern about their ‘capacity to provide financial resources for private extension services’, and the project has lagged in terms of implementing co-investments with these organizations to develop these services (see 3.1 Goal Achievement – Findings).105 97 Ibid 98 Ibid 99 Ibid 100 Ibid 101 ‘Performance Report Summary – April 1st, 2018 – September 30, 2018’ NCBA CLUSA, Santo Domingo, Dominican Republic, November 15, 2018 102 For example, FGDs, federation staff, Villa Riva, February 18, 2019 and association staff, Arroyo Ancho, February 19, 2019 103 For example, dairy producers, San Juan de la Maguana, February 22, 2019 104 Ibid 105 KII, SAFE staff, Santo Domingo, March 1, 2019 SAFE PROJECT – MIDTERM EVALUATION REPORT 59 In terms of coordination and linkages between project stakeholders, with regard to the beef value chain, as noted, SAFE has facilitated frequent meetings of an Equivalence Board, which has realized a number of important steps in the equivalence process through coordination between DIGEGA, DIGEMAPS, the slaughterhouses, and TA providers (see 3.1 Goal Achievement – Findings). However, coordination and linkages between stakeholders in the dairy sector have faced several challenges. First, as noted, a challenge is that most MCCs mix Grade A milk with other milk qualities and sell at the lower grade, thus undermining the realization of price increases in line with quality improvements (see 3.1 Goal Achievement – Findings). In addition, in KIIs with SAFE staff, respondents stated that initial targeting was ‘haphazard’, with the project often selecting unrelated producers, MCCs, and processors, for TA and co￾investments.106 These respondents confirmed that this resulted in challenges in realizing value chain-wide improvements, with some improvements lost as a result of partial adoption of improved techniques and technologies i.e. in some cases, improved quality milk was mixed with inferior quality milk at both the MCC and processor sites. In response, in July - August 2018, SAFE began implementing assessments of milk quality at MCCs that commercialize their output to large processing plants, initially focusing on three collection centers that receive milk from 163 farmers across the northwest of the Dominican Republic. Because milk quality testing is not frequently conducted at farm￾level by individual producers and those that do test quality do not keep farm records, assessments analyzed milk samples at MCCs in order to identify those producers that fail to comply with standards so that ‘improvements can be implemented through working sessions and visits to the problem farms’.107 According to SAFE staff, by the end of October 2018, the project had implemented these assessments at 41 MCCs and selected 20 for development of investment plans, with the intention of completing investment plans at all 41 in order to ‘know where to concentrate efforts to increase milk quality at the producer level, for their respective value chains’.108 3.6 EFFECTIVENESS – CONCLUSIONS EQ 3.1 To what extent have different actors in the value chain been able to increase quantity e.g., yield/ha and minimize waste since project inception? SAFE measures increase in quantity primarily in terms of changes in l/c/d of milk. To date, the project is significantly behind in terms of its target of a 50% increase over the baseline of 7.5 l/c/d of milk amongst target beneficiaries (currently the increase stands at only 6%). Increases have been undermined by recurring drought conditions, and initial challenges in TA delivery that resulted in low application of TA. Nevertheless, the transition to an LFS approach in 2018 has improved application of TA, though investment into farming systems remains problematic, possibly as a result of recurring subsidies undermining the perceived need to make financial investments. 106 For example, KII, SAFE staff, Santo Domingo, February 13, 2019 107 Costa, Dr. Mildred ‘Informe Sobre Monitoreo Realizado en La Línea Noroeste; El Cajuil (Loma de Cabrera), Santiago de la Cruz (Dajabón), El Chaquey y Sabana Larga (Dajabón)’ NCBA CLUSA, Santo Domingo, Dominican Republic, July 16, 2018 108 For example, KII, SAFE staff, Santo Domingo, March 1, 2019 SAFE PROJECT – MIDTERM EVALUATION REPORT 60 EQ 3.2 To what extent have all actors in the value chain adopted quality standards? To date, no value chain actors have adopted quality standards. Two dairy processors are close to meeting requirements for the SoQ and can be anticipated to obtain it before the end of the SAFE project. Likewise, SAFE implementing partner TAMU has developed a critical path for improving compliance with HACCP and sanitation standards, and two are close to ready. EQ 3.3 To what extent have value chain actors been trained in quality standards? The SAFE project has made ‘fair’ progress in training value chain actors in quality standards. However, progress toward value chain-wide improvements of standards has been challenging due to the unanticipated scale of sanitary problems in both the beef and dairy sectors, the practice of mixing milk of various standards and selling it as lower grade, and possibly due to the limited financial motivation of producers to invest in upgrades due to marginal returns for higher grade milk. EQ 3.4 Are partner organizations providing training to beneficiaries that is tailored to the local context and achieving increased use of agricultural techniques and technologies, increased use of improved post-harvest processing and handling practices, and improved marketing and branding? Most SAFE partner organizations are providing training that is tailored to the local context. Specifically, TAMU consultants affiliated with the Borlaug Institute have focused their training on addressing weaknesses in the inspection services and sanitary problems revealed through audits of dairy processing facilities and slaughterhouses. SAFE implementing partner CRI/GENEX has adapted their approaches to focus on delivering TA on ‘basic practices’ related to livestock breeding directly to producers and developing association capacity to manage AI routes. The transition to the LFS approach has improved producer satisfaction with TA and enhanced the capacity of POs to deliver TA to membership. However, initially the project lagged in terms of application of achieving increased use of improved agricultural techniques and technologies, increased use of improved post￾harvest processing and handling practices, and improved marketing and branding, especially due to initial challenges in TA delivery. Nonetheless, the transition to the LFS approach, and investments in inspection services by the GoDR, seem to have improved project performance in this regard since 2018. EQ 3.5 Are local government extension officers offering technical support to farmers in new techniques and technologies? To date, government extension officers, i.e. the MEGALECHE extension unit, have performed poorly in terms of offering technical support to farmers in new techniques and technologies. SAFE PROJECT – MIDTERM EVALUATION REPORT 62 In terms of external factors, as noted, throughout fieldwork for this MTE, KII and FGD respondents consistently cited drought related to El Niño as a dominant long-term exogenous condition significantly reducing productivity every two years, while background document review confirmed drought conditions in 2016 and 2018-19 (see 3.1 Goal Achievement – Findings).113 Figure 3: Capacity Development Survey Results Disaggregated by Site Source: SAFE MTE Quantitative Survey 113 ‘Caribbean Drought Bulletin’ Volume V, Issue10, Caribbean Drought & Precipitation Monitoring Network (CDPMN), March 2019 SAFE PROJECT – MIDTERM EVALUATION REPORT 63 In terms of allocation of resources across activities and geography, analysis of the responses to questions ranking capacity development activities related to productivity, processing, and managerial skills, showed moderate co-variation when disaggregated across the 12 provinces surveyed. For example, the order of preference for rankings i.e. 1-5, was significantly uniform across all provinces (with preference for ‘fair’ scores completely uniform across all provinces). Likewise, in FGDs and KIIs, respondents did not generally raise issues with resource allocation, and were not able to cite recommendations or lessons learned in this regard. The sole exception to this was one KII with financial institution staff, in which the Director of Credit at Banco Agrícola stated, relative to the performance of SAFE with regard to facilitating lending, “previously, the project facilitated regular meetings with us and other banks, which helped us to plan our lending activities and strategize. Since the restructuring (of the staff), these meetings have not been held. They were useful, SAFE should re-start them”. 114 3.8 EFFICIENCY – CONCLUSIONS EQ 4.1 What internal and external factors have influenced the ability of the project to meet expected results and targets? Internal factors that influenced the ability of the project to meet expected results and targets included staffing challenges, which resulted in delays and problems in coordination and relations between project partners. The critical external factor that has influenced the ability of the project to meet expected results and targets is recurring drought conditions, which have undermined dairy productivity. EQ 4.2 Do project staffing, training activities and management structures efficiently support the achievement of expected project outcomes? Despite initial staffing challenges, which generated delays and challenges in partner relations, following a restructure in 2018, the current project staffing, training activities and management structures efficiently support the achievement of expected project outcomes. A restructuring of SAFE staff in 2018, including replacement of key staff, has improved efficiency in supporting the achievement of expected project outcomes. The transition to the LFS approach to TA delivery was a key improvement in training activities. EQ 4.3 Does the project resource allocation to activities and to the project’s geographic areas efficiently support the achievement of expected project outcomes? There is no reason to believe that project resource allocation to activities and to the project’s geographic areas do not efficiently support the achievement of expected project outcomes. Previously regular meetings between SAFE and financial institution partners, were reportedly helpful, but ceased in 2018. 114 KIIs, financial institution staff, Santo Domingo, March 1, 2019 SAFE PROJECT – MIDTERM EVALUATION REPORT 65 In KIIs with dairy processors, staff at two companies that had made significant progress toward obtaining the SoQ through SAFE support on improvements in sanitary conditions, including improvements in milk quality through work with supplier milk producers, described this assistance as supporting their marketing objectives, which they stated includes expanding their share of domestic markets for higher quality products (both of these processors agreed with the assessment that there is increasing domestic demand for higher quality products, as evidenced by the portion of inventory in local supermarkets consisting of imported dairy products).117 As noted, both of these processors anticipate obtaining the SoQ in the near future (see 3.1 Goal Achievement – Findings). Also, as noted, in KIIs with GoDR staff, the Minister of Agriculture cited improvements in milk quality as fundamental to improving the Dominican dairy sector, and confirmed that his office was prepared to provide a ‘two￾to-one match’ for SAFE co-investments into construction of producer milking parlors (see 3.1 Goal Achievement – Findings and 3.5 Effectiveness - Findings).118 However, as noted, in KIIs, a number of respondents explained that the GoDR habitually offers subsidized assistance to dairy farmers for political reasons, negating the need for individuals to make farm investments.119 These respondents pointed out that, in previous drought years, DIGEGA has ‘bailed’ out producers, and as a result, many simply wait for assistance as opposed to making resiliency enhancing investments into their production systems (see and 3.5 Effectiveness - Findings).120 In KIIs with beef processors, staff at slaughterhouses that had received significant SAFE support and have been identified as strategic to demonstrating success toward the equivalence process requirements confirmed that the equivalence process conformed with their marketing objectives. For example, in a KII with staff at AGROCARNE, the president stated, “we are trying to establish quality standards to improve the reputation of Dominican beef. There is currently no standards regime for beef (in the Dominican Republic), and demand for higher quality processed meat products is very limited, but it is growing (as evidenced by imports of higher quality products). Nicaragua does well because standards are enforced effectively.”121 Likewise, in a KII with MERCARNE staff, a respondent stated ‘we are trying to meet the standards for export to the U.S.’, noting specifically that ‘SAFE provided a co-investment of about DOP 2 million of which about DOP 500,000 was to improve equipment’ (leveraging almost DOP 1.5 million from MERCARNE) and that ‘MERCARNE is waiting for the inspectors to come to do the inspections’.122 However, this respondent also stated that ‘the co-inversion has taken a long time, and we are still waiting for the final installment of DOP 180,000 for the contractor’ and she is ‘not really aware of other activities. SAFE did not provide any information about the overall objectives of the project’ (this informant was brought on board in mid-2018).123 117 KIIs, enterprises, Montellano and Puerto Plata, February 20, 2019 118 KII, Government of the Dominican Republic, Santo Domingo, March 4, 2019 119 For example, KII, SAFE project and partner staff, Santo Domingo, March 1, 2019 120 Ibid 121 KII, enterprise, Santo Domingo, February 13, 2019 122 KII, enterprise, Santo Domingo, February 16, 2019 123 Ibid SAFE PROJECT – MIDTERM EVALUATION REPORT 66 3.10 SUSTAINABILITY – CONCLUSIONS EQ 5.1 What factors are likely to contribute towards the sustainability of the following issues addressed by the SAFE project: policy environment, financial incentives (i.e., profit motive), access to inputs and information, market access, etc. While SAFE has varying degrees of success in meeting targets with regard to the issues the project addresses, the factor most likely to result in sustainability is the ability of the project to deliver value chain-wide improvements in productivity and trade expansion. Dairy processors and slaughterhouses that have received value chain-wide assistance are closest to being able to obtain SoQ certification or equivalence requirements, and hence motivate additional enterprises to follow suit by providing examples of success. EQ 5.2 What factors are likely to hinder sustainability of the following issues addressed by the SAFE project: policy environment, financial incentives (i.e., profit motive), access to inputs and information, market access, etc.? Conversely, the factor most likely to hinder sustainability is incomplete value chain-wide adoption of improvements in productivity and trade expansion. Specifically, incomplete adoption of quality enhancing improved techniques and technologies by dairy producers and MCCs leads to mixed Grade A milk with lower grades that sells at lower prices. Similarly, marginal price improvements for dairy producers, among other factors, are likely to impede sustainability of activities that enhance productivity, including making investments into improved production systems. Likewise, improvements in dairy processor quality are unlikely to result in their meeting domestic market expansion objectives if overall supplier milk quality does not improve. Dairy processors are unlikely to obtain SoQ certification, and thus provide examples of success to additional enterprises, if problems persist in their supply chains. EQ 5.3 What project activities are likely to contribute towards sustainability and is there evidence that project activities or benefits will be sustained after the project ends? Association-wide activities that improve milk quality and thus price are most likely to contribute to sustainability. Furthermore, demonstrated improvements in milk prices are likely to lead to sustainable application of productivity enhancing improved techniques and technologies, and eventually to investments into their adoption. However, given limited SAFE resources, the project will require innovative sustainable approaches to delivering improvements at the required scale, i.e. association-wide, as opposed to previous GoDR-subsidized approaches. The Minister of Agriculture has confirmed that his office is prepared to match SAFE investments into construction of milking parlors, which the Ministry regards as fundamental to improving milk quality, on a two-to-one basis, and SAFE staff has recently implemented assessments of MCCs and related farmers to determine where problems lie, and to facilitate steps to address these problems. However, to date there is no evidence that project activities will be sustained after the project ends. SAFE PROJECT – MIDTERM EVALUATION REPORT 69 However, subsequent to improvements in project implementation in 2018, the GoDR has demonstrated commitments to SAFE objectives, as expressed in the obligation of funding to DIGEMAPS for expansion of inspection services, and commitments by the Minister of Agriculture to provide matching funds for expansion of milking parlors as a fundamental step to improving milk quality. EQ 6.2 To what extent can impact be attributed to project activities? New commitments by the GoDR to SAFE objectives can be attributed to project activities that have advanced the equivalence process, as well (to a lesser degree) to the alignment of SAFE activities with GoDR economic development strategies, especially regarding export expansion. EQ 6.3 To what extent have target groups (farmers, processors, vendors, consumers, etc.) adopted the promoted techniques that have economic benefits? To date, the extent of target group adoption of promoted techniques that have economic benefits has been low and incomplete. Going forward, achieving illustrative successes in market expansion by processors and slaughterhouses, as well as more comprehensive value chain-wide work will be key to improved adoption levels. SAFE PROJECT – MIDTERM EVALUATION REPORT 70 4. RECOMMENDATIONS AND LESSONS LEARNED This section presents recommendations, which the ET derived from the conclusions presented above, followed by lessons learned identified by the ET through this MTE. 4.1 RECOMMENDATIONS 1. Resolve problems and inconsistencies in the PROGIS database: The ET found several problems in the data entered into the PROGIS online M&E platform, including incorrect or missing beneficiary contact data, as well as duplication of beneficiaries and inconsistencies in the overall number of project beneficiaries. SAFE staff should audit beneficiary contact data and update entries where required. Staff should also use this audit to resolve inconsistencies in overall beneficiary numbers and disaggregate by beneficiary type. 2. Adopt a value chain-wide approach to dairy quality upgrades: The ET found that SAFE has lagged in terms of meeting targets related to adoption of improved techniques and technologies that improve dairy productivity and quality of milk. This is due to exogenous factors, especially recurring drought conditions that reduce water and pasture available to livestock, as well as due to poor financial motivation for producers to adopt improved practices, especially through investments into their production systems. Likewise, the ET found that poor financial motivation is related to MCCs mixing Grade A milk with lower grades and selling it as lower quality milk, as well as due to ‘incomplete’ adoption of quality upgrades i.e. partial adoption by associated producers by MCCs, and by supplier MCCs selling to processors, both of which undermine the realization of improved prices. Therefore, the ET recommends adopting a value chain-wide approach through five steps, as follows: Figure 4: Chain-Wide Approach to Quality Upgrades SAFE PROJECT – MIDTERM EVALUATION REPORT 71 2.1Assess PROGIS data to re-assess productivity targets and identify ‘champion’ producers to lead LFS: Following resolution of inconsistencies in the PROGIS online M&E platform, SAFE staff should stratify dairy producer beneficiaries according to ‘full’, ‘partial’ and ‘non’ adoption of the TA package and assess the correlation between adoption levels and productivity increases. Based on this assessment, the project should re-assess targets in order to set realistic performance objectives related to improvements in l/c/d of milk with the context of recurring drought conditions in the project target areas. Analysis of this data should also inform specific expert TA recommendations to producers based on regional variations across the project zones. Finally, through this assessment, the SAFE staff should identify ‘champion’ farmers that can lead LFS, utilizing their production systems as effective demonstration sites. Specifically, participation in ‘champion’ LFS should address lagging project performance in terms of famers making decisions based on economic considerations or analysis by making use of PROGANA logbooks (including cost of production records) a mandatory prerequisite for participation in order to facilitate development of producer financial records that will be subsequently required for financing, as well as useful in enhancing M&E data. 2.2Facilitate comprehensive quality upgrades at ‘milk tank’ scale to improve milk prices: The SAFE project should identify high performing LFS that deliver Grade A quality milk at the scale of MCC tanks (generally 3,000- 5,000 liters). The project should facilitate separate handling of Grade A milk by MCCs to processors at corresponding prices, realizing a profit to supplier producers in order to facilitate upgrades in their product quality. 2.3Facilitate comprehensive quality upgrades at ‘supplier’ scale to improve milk prices and ensure processor quality: The project should facilitate relationships between MCCs capable of supplying separately handled Grade A milk and processors seeking to upgrade their products in order to meet SoQ requirements, focusing on ‘champion’ processors (closest to obtaining the SoQ). Facilitation should entail face-to-face meetings between MCC leadership in order to negotiate additional linkages that may improve milk prices, such as timing and scale of delivery. 2.4Facilitate and promote processor linkages to supermarkets for SoQ products to address supply and demonstrate demand: SAFE staff should facilitate further supply chain linkages between ‘champion processors’ (closest to obtaining the SoQ) and supermarkets in order to better understand market requirements for higher quality dairy products and, after obtaining the SoQ, initiate and promote supply to consumers. Likewise, the project should promote the SoQ to additional processors through demonstration of verified market demand in order to motivate expansion of the standards. SAFE PROJECT – MIDTERM EVALUATION REPORT 72 2.5Facilitate ‘blended finance’ combining TA and investments for MCC￾wide quality upgrades to expand improved production: As noted, the project has lagged in terms of targets for agricultural lending, which is required to finance resiliency upgrades in production systems. The project should re-launch regular meetings with financial institution partners and integrate DIGEGA into these meetings in order to develop a comprehensive approach to implementing DIGEGA’s commitment to match SAFE grant funds for the expansion of milking parlors. Likewise, the project should expand focus on farm financial management in TA delivered through LFS in collaboration with the MEGALECHE. Overall, the project should work with these partners to develop a phased approach to expanding milking parlors and other quality enhancing techniques and technologies starting initially with higher performing LFSs already linked to dairy processors, in order to demonstrate the efficacity of the TA package in terms of price improvements. Subsequently, these partners should incrementally increase focus on resiliency enhancing techniques and technologies, such as improved pastures, in order to expand the quantity of l/c/d. For example, an important technique to ensure milk production during dry seasons is feed conservation such as haylage or silage. This seasonal preparation technique during ideal weather conditions could have possibly mitigated some of the effects of the drought occurring now and is not currently practiced at a wide-scale in the country. The partners should deliver financing through a mixed approach, with a component of repayable loans combined with a grant component. Throughout, the process should align TA and financing for specific products i.e. the TA addresses borrower weaknesses to reduce risk, create bankable investments, and improve repayment rates. This integration of TA into finance to de-risk lending is known as “blended finance”, while an entity integrating these elements is a Technical Assistance Facility (TAF). The TAF can ultimately receive additional capitalization, including from private investors or development finance institutions (DFIs), in order to expand operations and/or support new activities. Likewise, this entity can also adopt equity models in order to finance upgrades to POs and enterprises. 3. Advocate for inspection services and slaughterhouses to progress on the ‘critical path’ routes toward conformity with the FSIS standards: Given the anticipated response to the SRT by FSIS, the project should advocate for inspection services and slaughterhouses to progress on the critical path routes toward conformity with the FSIS standards in order to prepare for ‘mock’ audits by TAMU consultants in line with their recommended next steps in the equivalence process. Advocacy should include the highest levels of Dominican government agencies. Also, the project should focus TA on advancing one slaughterhouse on this ‘critical pathway’ for improving compliance with HACCP and sanitation standards developed by TAMU in order to demonstrate the viability of the process to the entire industry while also maximizing the chances of success. SAFE PROJECT – MIDTERM EVALUATION REPORT 73 4. Audit project marketing and branding strategies to better inform the public that USDA is the project donor: SAFE staff should implement an audit of the project’s marketing and branding strategies in order to improve the ability of the project to inform the public that USDA is the donor. 4.2 LESSONS LEARNED Dairy development projects need to take into account climatic conditions and focus on resiliency in cases where climate challenges impact the ability of producers to expand production: Increased climatic challenges related to global climate change will increasingly challenge dairy producers to improve productivity due to limiting water and pasture available for livestock. An additional negative consequence of global climate change will be increased sanitary challenges with livestock in tropical and semi-tropical regions experiencing greater incidence of disease due to warmer weather (though this evaluation did not find this to be the case in the Dominican Republic specifically at the time of this evaluation). Therefore, projects that address dairy sector improvements need to prioritize increased resiliency through adoption of climate smart practices. In order to realize the benefits of upgrades, especially in terms of enhanced quality, dairy sector improvements need to take a value chain-wide approach: Partial adoption of improvements in dairy production undermine the realization of improved prices due to MCC challenges in separately handling improved quality milk at the MCC level. Likewise, comprehensive improvements in quality determine the ability of processors to upgrade product quality. Therefore, dairy value chain projects need to take a value chain-wide approach that takes into account supply chain scale at the MCC and processor levels. In economies characterized by high levels of subsidies, implementers need to demonstrate sustainable models for TA and finance to private sector actors: As noted, the GoDR has provided subsidized assistance to private sector actors through a number of programs implemented by public sector institutions, including subsidized power through the electric company, loans through Banco Agrícola and CONALCHE, as well as recurring drought assistance to livestock producers through DIGEGA. This assistance, while useful, undermines sustainable producer investments into production systems that could lead to improved resiliency. In cases where economies are characterized by high levels of subsidies, implementers need to demonstrate sustainable models for TA and finance to private sector actors, including revolving loans funds tied to TA, or TAFs and ‘smart’ subsidies that increase demand for services by offsetting their cost. SAFE PROJECT – MIDTERM EVALUATION REPORT 74 ANNEX A. RESULTS FRAMEWORKS Figure 5: SAFE Project Food for Progress Results Frameworks SAFE PROJECT – MIDTERM EVALUATION REPORT 75 ANNEX B. PROJECT ACTIVITIES The project Theory of Change (TOC) outlines the following 12 project activities:127 ACTIVITY 1: TRAINING ON IMPROVED AGRICULTURAL TECHNIQUES The project focuses on the adoption of improved agricultural techniques through training and technical assistance on the planting of improved pastures and their pasture management techniques with training sessions on the following: animal nutrition, animal feed; milking and milk￾handling practices, animal health (use of pharmaceuticals, herd vaccinations, tick prevention and animal handling), sanitary standards for dairy and beef processing facilities i.e. Good Manufacturing Practices (GMP), Hazard Analysis and Critical Control Points (HACCP), sanitary registration of products, and training on the implementation of sustainable livestock management. ACTIVITY 2: TRAINING ON FARM MANAGEMENT This activity refers to the implementation of training on agribusiness and training and application of tools and techniques to improve farmer capacity in terms of farm administration and management in an orderly and documented way by recording the relevant information of the farms in terms of production i.e. quantity of milk, number of cows, reproductive records), management of personnel and payroll, and farm economic management (income and expenses records). The documentation and registration of activities on the farm is a tool for making administrative and productive decisions that allow for a business approach to farm management thereby improving the levels of herd profitability. ACTIVITY 3: FACILITATE AGRODEALERS AND INPUT SUPPLIERS The project coordinates technical assistance activities with input suppliers to design and establish demonstration farms for management, milking parlors, and other modules of improved practices. These suppliers include Semillas Latinas, ABODOM, Ferquido, Vitrogan, Enda Caribe, and others. It has signed MOUs with Nestle, and Rica. Inputs include seeds, fertilizers, pharmaceuticals, etc. The project links input suppliers which are interested in supplying inputs to producers in the target provinces, providing technical training to extension agents on the use of their inputs, discounting and sharing the cost of the required inputs for establishing demonstration plots and providing specific technical assistance (TA) to farmers. ACTIVITY 4: IMPROVE THE BEEF AND DAIRY VALUE CHAIN The project provides matching grants for dairy processors and Producer Organizations (POs) to strengthen milk collection centers (MCCs) and beef processors supporting the Equivalence process. Infrastructure improvements for dairy include milk handling equipment for farmers, chilling equipment for MCCs and laboratory equipment for milk testing facilities. Infrastructure improvements in beef processing includes Sanitary Performance Standards control systems identified by the regulatory assessment and agreed to by the GoDR and private sector partners. SAFE works with the Ministry of Agriculture, Ministry of Health, the Ministry of the Presidency, and slaughterhouses to achieve Equivalence. The project has signed MOUs with APROLECHE, CONALECHE, INDOCAL, Asociación de los Mataderos, Asociación Dominicana de Hacendados y Agricultores (ADHA), and also works Asociación de Ganaderos de Carne (ASOCARNE). 127 ‘NCBA CLUSA International Safe Agriculture Food Export (SAFE) Project Mid-Term Evaluation (Terms of Reference’ NCBA CLUSA International (approved by USDA), July 25, 2018 SAFE PROJECT – MIDTERM EVALUATION REPORT 76 ACTIVITY 5: CAPACITY BUILDING OF PRIVATE AND PUBLIC SECTOR AGRICULTURAL EXTENSION AGENTS/ SERVICES The project will provide more support to key private sector value chain actors to establish and improve private extension systems. To date, 38 students were trained at the Technological Institute of San Ignacio de Loyola (ITESIL) as new extension agents, of which two are working on farms around the area. Also, SAFE worked with project partners to provide training to private and public agricultural extension workers. Private sector companies included ABODOM, HESATECH, NUGEGA, MULTILAB, Nestle, and Rica. Public sector organizations included Ministry of Agriculture MEGALECHE extension unit, Vitrogan, and the Autonomous University of Santo Domingo (UASD). ACTIVITY 6: CAPACITY BUILDING OF PRODUCER GROUPS There are five livestock federations collaborating with the project: FEDEGANO; FEDEGANORTE; FEGACIBAO; FEDEGARES, and; FEGASUR. The project signed coordination agreements with the first two federations and with three associations; AGAMPTA; San Francisco de Macorís Livestock Farmers Association, and; Asociación de Hato Mayor. The five federations aggregate around 90% of associated livestock farmers within the eleven target provinces. The obligations of both parties are to share resources, mainly technical support and expertise to contribute to the Project´s goals. One of the SAFE Project’s goals is to strengthen the capacities of livestock POs focusing on organizational, administrative, financial and technical aspects. ACTIVITY 7: CAPACITY BUILDING FOR PROMOTING IMPROVED POLICY AND REGULATORY FRAMEWORK Under this activity, the project is supporting the development of necessary infrastructure overseeing the control and implementation of the sanitary standards equivalent to those demanded in the U.S. for the beef sector that are required to export beef to the U.S. market in Puerto Rico. Three main actors of the beef value chain are essential to attaining Equivalence: DIGEMAPS (General Directorate of Medicines, Food and Health Products of the MoH), the official public agency responsible for the sanitary inspection system in the Dominican Republic; LAVECEN, the reference laboratory to be certified and the official laboratory to be audited by the FSIS; and, the three preselected slaughterhouses, ideally the most advanced will be selected and audited by FSIS, to determine whether the MoH is operating a beef inspection system for beef that is equivalent to the U.S. inspection system. SAFE is providing technical assistance to slaughterhouses and beef farmers supplying slaughterhouses on a proposed course of action needed to achieve those standards. ACTIVITY 8: FINANCIAL SERVICES - FACILITATE AGRICULTURAL LENDING AND PRIVATE AND PUBLIC- SECTOR INVESTMENT The project provides training to financial institutions (FIs) on the financing needs of dairy farmers and processors. The SAFE project collaborates with FIs to conduct outreach to identify credit-worthy dairy producers and processors. Investment plans for small farms are developed as part of the support of the project. The project links these small farmers with financial institutions to facilitate producers’ and processors’ access to credit based on the result of the small investment plans supported by the project These include Banco Agrícola, Coopesur, CONALECHE, and the government-supported Fondo Especial para el Desarrollo Agropecuario (FEDA). Additionally, the SAFE project facilitates the participation of farmers and processors at exposition, trade, and finance fairs to promote available financial products. SAFE PROJECT – MIDTERM EVALUATION REPORT 77 ACTIVITY 9: TRAINING - BEEF AND DAIRY QUALITY STANDARDS SAFE provides training to small and medium dairy manufacturers throughout the Dominican Republic. These trainings guide processors on how to improve the quality of cheese and other dairy products by applying GMP such as the use of sanitation materials, avoidance of adulterating milk with peroxide, the use of proper manufacturing clothes (hats, boots, etc.), and avoiding product cross contamination. For beef, the project trains beef processing staff on GMP, specifically related to SPS gaps. Additional training on HACCP will be aimed at beef processors and slaughterhouses. ACTIVITY 10: MARKET ACCESS: FACILITATE PRIVATE AND PUBLIC PARTNERSHIPS SAFE signed a general agreement with DIGEGA that was made operational by the articulation of 26 MEGALECHE agents responsible for providing TA to targeted farmers and POs through the public extension system. In addition, SAFE works with a Consultative Technical Council (CTC) and the Equivalence Group to achieve the Equivalence certification to export beef to the USA (Puerto Rico). The approach establishes public-private partnerships to focus on the Equivalence certification (SAFE’s primary result) with FSIS standards. ACTIVITY 11: CAPACITY BUILDING - DEVELOP PUBLIC INFORMATION CAMPAIGN AND IMPROVE MARKETING AND BRANDING The project implements a comprehensive approach to develop the use of the Seal of Quality in dairy products. To improve the quality of milk at the consumer level, the project works from the beginning of the milk value chain, training milk producers on hygienic and safe milking techniques that will improve the quality of the milk, seeking to produce Grade “A”. Next, the Project trains milk processors on the requirements for the attainment of the Seal of Quality. Finally, the project accompanies milk processors in the attainment of the sanitary registration from the Ministry of Health and the Instituto Dominicano para Calidad (INDOCAL), in addition to posterior attainment of the Seal of Quality through technical assistance. Several visual materials are developed and distributed along the value chain with messages about adequate production and processing practices. There were public information campaigns at the beginning of the project through trade fairs, which will continue in 2019. ACTIVITY 12: MARKET ACCESS - FACILITATE BUYER-SELLER RELATIONSHIPS In the dairy sub-sector, the SAFE project aims to facilitate linkages at four levels: 1) between POs/MCCs and processors; 2) between processors and individual farmers; 3) between equipment and service providers and processors; and 4) between producers and equipment and service providers. The project organizes meetings between MCCs and processors with nearby independent dairy farmers. Also, the project works on linkages between cheese producers and U.S. buyers, and beef producers exploring the Puerto Rico market for Dominican exports once equivalence is achieved. SAFE PROJECT – MIDTERM EVALUATION REPORT 78 ANNEX C. EVALUATION TEAM David Rinck (Team Leader) was responsible for overall supervision of the ET and MTE implementation and deliverables. Mr. Rinck has more than 20 years of experience in monitoring and evaluation of economic growth initiatives, specializing in food security, agriculture and agro-enterprise development, market systems and economic policy analysis. In addition, he brings over 20 years of experience as Team Leader for complex project evaluations. Previously, Mr. Rinck was Chief of Party for a State Department￾funded project to enhance agricultural trade and investment in North Africa (Morocco, Algeria, Tunisia and Egypt). In Rwanda, he was the USAID focal point for the Comprehensive African Agricultural Development Plan (CAADP) compact. His publications include “Implementing Sustainable Private Sector Development: Relief to Development in Crisis-Affected Economies” ILO BDS Reader, September 2006 and “Market Development in Crisis- Affected Environments: Emerging Lessons for Achieving Pro-Poor Economic Reconstruction” Small Enterprise Education and Promotion Network (SEEP), 2007. Mr. Rinck holds a master’s degree in Social Sciences from the University of Chicago, and a degree in Agricultural Economics from the University of California at Davis. He has also studied economics, languages and sociology at many locations worldwide, including Beijing University, as well as in the Middle East at Damascus, Beirut, and Sana'a. He speaks English, Spanish, French, and Portuguese. Antonio Cordella (Agricultural and Trade Economist) is an international expert of Agricultural economics, Agricultural Policy and Trade Policy, with expertise in development issues in n African, Caribbean and Pacific (ACP) countries and emerging economies, including agriculture, commodities, trade, rural development, micro-credit, sector consolidation, poverty reduction, environment and gender issues, public-private partnership. He has extensive expertise in ex-ante and ex-post evaluations of technical cooperation projects, including trade policy, agricultural policy, rural development policy, and private sector competitiveness. Dr. Cordella also carried out numerous reviews/evaluations of food/commodity marketing systems and mapping of economic market chains; agribusiness; and backward and forward farm links at regional and national level. Recently, Dr. Cordella performed the mid-term evaluation of the ‘Exporting Quality and Safety’ project implemented by International Executive Service Corps (IESC) in the Dominican Republic. The project is focusing on greenhouse and oriental vegetables, avocado, cacao, and pineapple value chains. Specifically, he designed and facilitated a questionnaire survey of 335 direct beneficiaries of the project and facilitated various KIIs and FGDs with key stakeholders. Dr. Cordella also assessed the performance of the 36 result and activity indicators and presented results of the evaluation. He is fluent in Italian, Spanish, English, and French. SAFE PROJECT – MIDTERM EVALUATION REPORT 79 Xerius Research & Consulting, S.R.L. (Quantitative Research Team) is a firm based in Santo Domingo. Xerius carried out data collection by providing enumerators, supervisors, logistics and coordination for data collected in provinces of the Dominican Republic. This enabled the ET to implement the assignment quickly and efficiently. The ET supported training of enumerators and supervised testing of data collection forms. Evaluation Management: In addition to the ET members described above, JAA Headquarters staff supported the ET throughout the evaluation process. JAA was responsible for overall contract management and administration and ensuring that deliverables were submitted on time and met JAA and NCBA CLUSA’s quality standards. JAA was also responsible for all supervision, data management, quality control, and analysis. SAFE PROJECT – MIDTERM EVALUATION REPORT 81 ANNEX E. CITATIONS AND REFERENCES ‘2016 - Self Reporting Tool - Version 5 (Equivalence)’ USDA FAS, January 10, 2017 ‘Agreement Between the Government of the United States of American and National Cooperative Business Association for the Provision of Agricultural Commodities Through the Food for Progress Act (USDA Award FCC-517-2015-008-00) – Amendment I’ USDA FAS, Washington DC, December 12, 2015 ‘Agreement Between the Government of the United States of American and National Cooperative Business Association for the Provision of Agricultural Commodities Through the Food for Progress Act (USDA Award FCC-517-2015-008-00) – Amendment II’ USDA FAS, Washington DC, October 18, 2018 Aquino, Félix ‘Informe Inspección a Queso Don Manuel (Higüey)’ DIGEMAPS, Santo Domingo, August 28, 2018 ‘Caribbean Drought Bulletin’ Volume V, Issue10, Caribbean Drought & Precipitation Monitoring Network (CDPMN), March 2019 “Compliance – Project Status Comparison Outcome – April 2015-October 2018’ USDA FAS, Washington DC, January 16, 2019 ‘Cooperative Agreement Between the Government of the United States of American and National Cooperative Business Association for the Provision of Agricultural Commodities Through the Food for Progress Act (USDA Award FCC-517-2015-008-00)’ USDA FAS, Washington DC, October 24, 2015 Costa, Dr. Mildred ‘Informe Sobre Monitoreo Realizado en La Línea Noroeste; El Cajuil (Loma de Cabrera), Santiago de la Cruz (Dajabón), El Chaquey y Sabana Larga (Dajabón)’ NCBA CLUSA, Santo Domingo, Dominican Republic, July 16, 2018 ‘Dominican Republic Seeks to Export Meat to the U.S. in 2018 (success story)’ NCBA CLUSA, Santo Domingo, Dominican Republic, October 13, 2017 ‘Farmer Field School (FFS) Manual’ FAO, Special Framework of Assistance, St. Lucia, 2006 ‘Food for Progress and McGovern-Dole Indictors Definitions’ USDA/FAS, Washington DC, July 15, 2014 Hallie Preskie, Tessy Tzavaras Catsambas ‘Reframing Evaluation through Appreciative Inquiry’ Sage Publications, 2006 SAFE PROJECT – MIDTERM EVALUATION REPORT 82 ‘Intro and Strategic Analysis’ NCBA CLUSA, Washington DC, 2015 Jennings, Paul G. ‘Optimizing the Nutritional Management of the Dairy Cow Grazing Improved Tropical Pasture’ Marjan Consortium Ltd., St. Catherine, Jamaica, 2017 (reprinted) King, Gary, Robert Keohane, and Sydney Verba ‘Designing Social Inquiry: Scientific Inference in Qualitative Research” Princeton University Press, 2016Research’ Princeton University Press, 2016 ‘Milk quality monitoring key to improving Dominican Dairy industry (impact story)’ NCBA CLUSA, Santo Domingo, Dominican Republic, May 26, 2017 Moran, John ‘Business Management for Tropical Dairy Farmers’ Landlinks Press, Collingwood, Australia, 2009 (reprinted) ‘Monitoring Milk Quality in the Dominican Republic (success story)’ NCBA CLUSA, Santo Domingo, Dominican Republic, August 21, 2018 ‘NCBA CLUSA International Safe Agriculture Food Export (SAFE) Project Mid-Term Evaluation (Terms of Reference)’ NCBA CLUSA International (approved by USDA), July 25, 2018 ‘NCBA CLUSA Trains Small and Medium Dairy Processors in Dominican Republic (success story)’ NCBA CLUSA, Santo Domingo, Dominican Republic, December 28, 2017 ‘Performance Monitoring Plan’ NCBA CLUSA, Santo Domingo, Dominican Republic, December 3, 2018 ‘Performance Report Summary – April 1st, 2018 – September 30, 2018’ NCBA CLUSA, Santo Domingo, Dominican Republic, November 15, 2018 ‘Review and update of Performance Monitoring Plan (PMP)’ Optima, Technical Services (OTSCORP S.A.) Santo Domingo, Dominican Republic, June 6, 2016 (rev. Sept. 2016) ‘Safe Agriculture/Food Export (SAFE) Program- Dominican Republic Project Baseline Survey (Agreement No: FCC-517-2015/008-00): Baseline Study - Performance Report Summary April 1st, 2018-September 30, 2018’ NCBA CLUSA, Santo Domingo, Dominican Republic, September 2018 ‘Safe Agriculture/Food Export (SAFE) Program- Dominican Republic Project Baseline Survey (Agreement No: FCC-517-2015/008-00): Baseline Study - Status of the Livestock Sector in Eleven Provinces of the Dominican Republic’ Optima, Technical Services (OTSCORP S.A.) Santo Domingo, Dominican Republic, October 13, 2016 SAFE PROJECT – MIDTERM EVALUATION REPORT 83 ‘SAFE/PROGANA Project Holds Panel Discussion on Best Practices for Cost-Effective Livestock Farming and Improving Value Chain for Export Purposes (success story)’ NCBA CLUSA, Santo Domingo, Dominican Republic, July 7, 2017 ‘SAFE/PROGANA Signs Agreement with Farmers to Establish Demonstration Farms (success story)’ NCBA CLUSA, Santo Domingo, Dominican Republic, August 21, 2017 ‘SAFE Semi-Annual Reports (all-in-one)’ NCBA CLUSA, Santo Domingo, Dominican Republic, March 15, 2017 ‘SAFE Marches Towards its Goals with the Continuous Support of its Technical Advisory Council (success story)’ NCBA CLUSA, Santo Domingo, Dominican Republic, June 5, 2017 ‘Semi-Annual Narrative Activity Report October 1st, 2017 to March 31th, 2018’ NCBA CLUSA, Santo Domingo, Dominican Republic, March 2018 ‘Semi-Annual Narrative Activity Report April 1st, 2017 to September 30th, 2017’ NCBA CLUSA, Santo Domingo, Dominican Republic, September 2017 ‘Training Dominican Farmers on Livestock Artificial Insemination (success story)’ NCBA CLUSA, Santo Domingo, Dominican Republic, April 24, 2018 ‘USDA-Dominican Republic Visits NCBA CLUSA SAFE Project (success story)’ NCBA CLUSA, Santo Domingo, Dominican Republic, December 20, 2017 ‘Visitas a Fincas y Técnicas; October 18-February 19, 2019’ NCBA CLUSA, Santo Domingo, Dominican Republic, March 2019 SAFE PROJECT – MIDTERM EVALUATION REPORT 97 Table 41: FGD KII Guide 1. FGD/KII Guide – Consent Statement Interview date and location: Interviewer: Consent Statement: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called J.E. Austin Inc., a company that is based in the United States. Our team is in the Dominican Republic to conduct a study about the SAFE project, which was funded by the United States Department of Agriculture Food for Progress program. We would like to conduct a brief interview today to learn about your perspectives on project performance. This information will be used in report for USDA that will be publicly available. It is important to understand that while we would like your help in this study, you do not have to take part if you do not want to, and you do not have to answer any questions if you do not feel comfortable doing so. The objective of this research is to improve the performance of projects like SAFE. The information may be used by other organizations as well. The interview is expected to take about 60 minutes. Do you have any questions? You may ask questions at any time. If you have questions or concerns about the research after we leave today, you can contact [insert ET name and contact information]. By saying “yes,” and participating in this study, you are indicating that you have heard this consent statement, had an opportunity to ask any questions about your participation and voluntarily consent to participate. Will you participate in this interview? You may answer yes or no. ◻ Yes, I will participate ◻ No, I will not participate SAFE PROJECT – MIDTERM EVALUATION REPORT 98 SAFE MTE Dominican Republic KII FGD Guide NAME: ORGANIZATION: POSITION: NAME: ORGANIZATION: POSITION: 1. Please describe the history of your organization and its function: 2. Please describe your experience and role in the SAFE project: 3. Please describe the SAFE activities and indicators you have experience with: 4. Please describe the effectiveness / impact of these activities: 5. Please describe changes and modifications to the activities you have experience with (if any): 6. How did these changes and modifications improve (or not) the impact of the SAFE project in terms of productivity / trade: 7. Please describe successes and challenges you encountered in your role with the SAFE project: 8. Please describe lessons learned (if any) you have gained during your experience with the SAFE project: SAFE PROJECT – MIDTERM EVALUATION REPORT 99 ANNEX G. INDICATOR ANALYSIS The ET has analyzed performance against each indicator, so that data collection can provide information on the following questions: 1. What have been the main reasons for progress achievements against activity and objective indicator targets? 2. What are recommended course corrections that will improve the likelihoods of the project reaching its activity and objective indicator targets? 3. For targets not on track to be met, are there changes to initial assumptions or within the operating context that warrants review, and which are recommended (this may include a recommendation for a revision of the targets)? To implement this review, the ET requested that NCBA CLUSA transfer required documents to the ET via Dropbox, which the agency did prior to seasonal holidays in December 2018. Subsequently, in early February 2019, the SAFE team provided an update of progress achieved during FY 2017 and FY2018, and the ET accordingly used these new data for analysis, though data for some indicators still did not add up correctly. Finally, the SAFE team supplied a new set of data and suggestions for target modifications on March 15th. Therefore, this attachment lists the Program Performance Indicator data as presented in Modification II Attachment E, which became effective on October 18th, 2018. This Modification Amendment lists a total of 101 indicators including 31 Result indicators and 70 Activity Indicators. It also lists End of Project Targets for each Result Indicator (RI) and annual (i.e. end of each FY) ad life of project targets for each Activity Indicator (AI). This analysis evaluated the performance of each RI by estimating the degree to which they are on track to reach end of project targets. As for AIs, this analysis evaluated performance against annual (i.e. FY) targets as well against mid-point targets. Although both Modification II of Attachment E and Project Status Comparison Outcome data differentiate between Result and Activity Indicators, neither of them ordered these as progressive numbers (i.e. RI 1, RI 2, and so on). Therefore, the ET maintained the ordering provided by the PROGIS system to avoid arbitrary classification of the indicators. However, PROGIS does not formally distinguish between ‘Result’ and ‘Activity’ indicators. However, the ET has done so to improve clarification and understanding of the analysis. Moreover, since the PROGIS system only includes 88 Indicators, the ET has classified missing indicators taking in consideration existing classification criteria. Moreover, indicator performance data provided by the PROGIS system is generally not up-to-date and/or incorrect. As for Project Status Comparison Outcome, this still includes a significant portion of indicators that were eliminated in subsequent amendments due because the USDA FAIS platform does not allow their elimination if previous data was reported. SAFE PROJECT – MIDTERM EVALUATION REPORT 108 SAFE PROJECT – MIDTERM EVALUATION REPORT 109 SAFE PROJECT – MIDTERM EVALUATION REPORT 110 SAFE PROJECT – MIDTERM EVALUATION REPORT 111 SAFE PROJECT – MIDTERM EVALUATION REPORT 112 SAFE PROJECT – MIDTERM EVALUATION REPORT 113 SAFE PROJECT – MIDTERM EVALUATION REPORT 114 SAFE PROJECT – MIDTERM EVALUATION REPORT 115 SAFE PROJECT – MIDTERM EVALUATION REPORT 116 SAFE PROJECT – MIDTERM EVALUATION REPORT 117 SAFE PROJECT – MIDTERM EVALUATION REPORT 118 SAFE PROJECT – MIDTERM EVALUATION REPORT 122 ANNEX J: STAFF NOTES ON EQUIVALENCE ET Note: This narrative was prepared by SAFE staff in order to provide information on post-evaluation activities and future plans related to the equivalence process. It is included here at their request. Milestones in the Equivalence process As several KI expressed to the ET, chances are the Equivalence will be granted to the DR beef inspection system during the lifetime of SAFE. USDA’s relations with the Dominican government and the excellent articulation of SAFE´s main local partner, the JAD, did establish the foundation of support for change in the policy and regulatory framework regarding the beef inspection system, and also laid the groundwork for changes to the application of regulations from the ministries of Health, and Agriculture, supported by the Presidency of the Republic through the Administrative Ministry of the President. The advances have been remarkable, as stated by the , who as the Central Competent Authority (CCA) leads the whole process. SAFE started conducting an analysis of previous USDA FSIS evaluations as well as the overall beef and dairy value chains, and joined in 2017 the originally called “Equivalence Round Table” that leading producer organization ADHA had put together in 2016, composed of interested stakeholders from within the public and private sectors, including beef producers, the national processing plant association (ADM), leading processing plants (Mercarne, Agrocarne, and Suplidora de Carne A&B), and other relevant agencies. SAFE strengthened this platform that evolved as the “Technical Equivalence Working Group”, were technicians from key entities discuss the required changes and provide follow-up. As time passed by it became clear for SAFE that the political decision-making level was not always present at the technical debates, thus slowing the required reforms, and through , was able to influence for the creation of the “Presidential Commission for Beef Exportation”, with participation of the Administrative Ministry of the Presidency, the Ministry of Agriculture, and the Ministry of Public Health, along with other high level authorities. The meetings began in 2017 at the Presidential Palace and since August 2018 are being held as needed at the Ministry of Agriculture. Based on the analysis of previous FSIS evaluations and with the guidance of experts from SAFE/TAMU, “critical path” routes were developed for each of the key actors in the equivalence process to address and correct any nonconformities with FSIS standards prior to the next official on-site audit including, but not limited: the national beef inspection system’s documentation, management, and procedures; beef processing plant infrastructure, HACCP systems and Sanitary programs; and microbiological and residue testing laboratory systems. I. Chemical Residues and Microbiological testing programs One of the most important decisions was taken in September 2018, to address the barrier imposed by LAVECEN´s lack of progress: in the light of one of the options given by the FSIS to the Central Competent Authority (CCA / DIGEMAPS), a network of supporting laboratories has been implemented, comprising three (3) laboratories from the public and private sectors, two (2) domestic and one (1) international: Instituto de Innovación e Biotecnología Industrial (IIBI) to analyze Salmonella and Generic E.coli; and the Laboratorio Agroempresarial Dominicano (LAD) for E.coli 0157:H7 and Non-0157 STECs, both located in the Dominican Republic; and the Laboratorio Nacional de Residuos (LANAR) located in Tegucigalpa, Honduras, to test chemical residues. The samples are being sent to LANAR-Honduras through OIRSA/LAVECEN. SAFE PROJECT – MIDTERM EVALUATION REPORT 123 For the local laboratories, SAFE fostered commitments from the Dominican government and the private sector to cost share the required laboratory testing during the equivalency process: while the project and the participating beef processing plants co-share the cost of the Microbiology testing, the Ministry of Agriculture is covering the chemical residues ones being done in Honduras. Therefore, the National Pathogen Control Program and the National Chemical Residues Control Program have been fully implemented since October 2018. The decision of the CCA (DIGEMAPS) to create a network of supporting laboratories represented a huge step forward in the Equivalence process, as described by (former FSIS, SAFE/TAMU’s consultant for labs’ compliance component). By the end of March 2019, 328 tests have been done for Microbiology, and 502 for Chemical Residues. Meanwhile, observing sustainability considerations and per the Minister of Agriculture request, OIRSA keeps working to strengthen LAVECEN. The idea is to incorporate this laboratory as part of the supporting network, once it gets ready. OIRSA has hired a consultant that is helping to close non-conformities based on SAFE/TAMU reports. They also hired a firm for the disposition of residues/risky materials and are planning to start inter-laboratory-tests. The Head-Quarters of OIRSA already sent a positive response to the invitation of the Minister of Agriculture to determine if they wanted to manage the entire LAVECEN and they agreed as long as they get 100% control and management of the lab. Next steps are being prepared by the Legal Counselor of the Ministry. Experts from SAFE/TAMU provide follow-up to this process. Following a recommendation provided by SAFE/TAMU, the CCA stablished a team of three (3) external auditors from (a national reference lab that belongs to the Ministry of Health), that in November 2018 where trained by (SAFE/TAMU) on how to prevent conflicts of interest and to conduct internal audits to verify third-party labs’ practices on FSIS compliance. The auditors have already performed independent evaluations of the two local labs (IIBI and LAD/JAD) and, observing an official request sent by the Minister of Health in order to advance in implement FSIS procedures for the laboratories that are part of the Equivalence process, SAFE is supporting a mission of the three (3) auditors to travel to Honduras to evaluate LANAR in June 2019, as it has been schedule between DIGEMAPS and OIRSA/Honduras (since they run LANAR). II. Beef Processing Plants Regarding the three (3) beef processing plants actively participating in the Equivalence process, in April 2018 a critical path route was developed to improve their HACCP and Sanitary programs. The timeline included two (2) HACCP trainings and several related to SPS and FSIS regulations. Two evaluations were conducted in June and September 2018, respectively, to measure progress made, and a third/final one in November 2018 with an FSIS-like audit carried out by two SAFE PROJECT – MIDTERM EVALUATION REPORT 124 complementary missions integrated by (for the slaughterhouses and the CCA central office & inspection services), and (for the laboratory services), that allow for the CCA to decide on concentrating efforts in the two (2) facilities that have advanced the most, as shown in the following table: On November the 20th, the CVO/CCA presented the results to the Presidential Commission for Beef Exportation (USDA DR was present). The three plants do show real progress, although at different rates. At the moment, the validation of the HACCP system is taking place in Mercarne and Agrocarne, along with closing non-conformities related to SPS in their infrastructures. During the week of January 28 – February 2, 2019, a hands-on training of DIGEMAPS inspection personnel and an evaluation of the two beef slaughterhouses preselected in November 2018: Mercarne S.R.L. and Agrocarne (Central Romana Corporation) were conducted jointly by as part of the project to export raw beef products from the Dominican Republic to the United States (U.S.). The training primarily dealt with DIGEMAPS inspection personnel learning how to evaluate the various food safety programs of the establishments and how to use the new inspection verification forms developed by Dr. Santoni A problem that is currently being faced relates to the facilities taking more time than expected to improve their infrastructures and implement the recommendations provided by the CCA, so, to tackle this, on March 27th, 2019, SAFE participated in a joint visit by the Ministers of Public Health and Agriculture to the Agrocarne facility, in Central Romana, to motivate the streamlining of the implementation of the recommendations for closing non-conformities with respect to FSIS requirements, in the framework of the Equivalence process. A similar visit is being planned by both ministers to Mercarne for similar purposes, and most likely to the third facility, Suplidora A&B. SAFE PROJECT – MIDTERM EVALUATION REPORT 125 III. THE General Directorate of Medicines, Food and Health Products (DIGEMAPS) of the Ministry of Health (MoH), as the Central Competent Authority (CCA) On May 15, 2017, DIGEMAPS submitted the SRT with supporting documentation to FSIS via FSIS’ Public Health Information System, which is a web-based application. On November 5, 2018, new documentation was sent, regarding the network of local and international laboratories to be used (instead of LAVECEN) to do the testing: LANAR-Honduras, IIBI, and LAD/JAD. As per FSIS request, the documentation was translated into English and uploaded to their PHIS on December 27, 2018 (at the beginning of 2019, Dr. Aaron did review the translation to make them more understandable for the FSIS inspectors, and those were also provided to the CCA for the corresponding action). [DIGEMAPS expects for the SRT to be reviewed by FSIS any time soon and therefore either ask for clarification on some of the responses or go ahead and schedule the FSIS verification audit for later this Summer 2019]. SAFE has also provided key support to improve the organization of the central office, along with the offices inside the participating facilities. Guidance has also been provided to the CCA by SAFE/TAMU consultants to improve the performance of the official inspectors along with their articulation with the Quality Control staff of Mercarne and Agrocarne, taking into consideration this has become a number one priority to guarantee beef inspection services are implemented following FSIS criteria. This is being complemented with on-site technical assistance to implement an inspector’s performance evaluation mechanism, a practical versus theoretical approach to staff training, and standardized in-plant evaluations. has reported that the resolution regarding the resting period for animals before the slaughter is having difficulties to be implemented in some facilities, due to the restriction on transportation (form the National Livestock Directorate -DIGEGA-) from 6:00 pm - 6:00 am to prevent robbery (a problem arises when a producer cannot take his animals to one of the three slaughterhouses participating in the Equivalence process due to time restrictions, so they sell them to another plant; stated that he cannot enforce the resolution in all the other big facilities since he doesn´t have enough inspectors – but DIGEMAPS is planning to hire more inspectors extensive to 9 facilities). The salary of the official inspectors has been unified and the beef processing plants no longer pay the extra time to the inspectors but deposit the corresponding amount in a bank account of the Ministry of Health (although a bit slow, so new authorities are reviewing this internal process). It is expected that a set of tests for the official inspectors will be developed by the CVO/CCA following guidelines provided by (SAFE/TAMU), and that for the first-time ever inspectors will be evaluated in a regular basis, stating in May 2019. Finally, considering the need to keep improving the production of beef in the Dominican Republic, SAFE helped activate the DR Beef Producers Association (ASOCARNE) that had been dormant since 2001, and is setting the basis along with its new Value Chain Specialist to characterize the primary link that supplies to the participating slaughterhouses, including fostering links between producer groups and input suppliers such as the U.S. Grains Council (USG) and successful U.S. companies like Organic Valley to encourage changes in the formulation of animal feeds that comply with FSIS standards, without prohibited substances, including substituting chicken manure for Dried Distiller’s Grains produced in the USA. This is being planned along with key partners like JAD, ADHA, and ASOCARNE, and a follow-up of the beef market study in Puerto Rico will take place in mid-2019. SAFE PROJECT – MIDTERM EVALUATION REPORT 126 IV. Notes on sustainability: While SAFE is providing guidance so the DR gets the Equivalence to reinstate exportation of raw beef products to USA (Puerto Rico), no doubt the process is strengthening the country Beef Inspection System to the point that a real food safety culture is being fostered among key stakeholders. The Meat Department of the Ministry of Public Health’s General Directorate for Medicine, Food, and Sanitary Products (DIGEMAPS) has now a staff of thirty-one (31) inspection officers, of which eleven (11) happen to be veterinarians [21 have been hired in the last three (3) years]. The central office has three (3) permanent veterinarians, including the CVO/CCA and new policies, procedures and inspection formularies have been developed and put in place observing FSIS criteria. SAFE has been encouraging the Central Competent Authority (CCA) to hire more inspectors to strengthen the national inspection system, so at the moment inspection services are being expanded to the nine (9) main slaughterhouses of the country (that provide around 60% of the beef) and the local laboratories are increasing their capacities also following FSIS criteria. Despite the fact that LANAR-Honduras has been included as part of the supporting network of labs for the Equivalence process, SAFE has been supporting OIRSA to keep strengthening LAVECEN to be part of such a network in the near future. SAFE has been key at training processing plants staff and official inspectors from the Central Competent Authority (CCA)/DIGEMAPS to improve their performance on : risk identification and food safety; evaluating the processing plant’s HACCP plans; conducting ante mortem (general requirements, humane handling) and post mortem (General Requirements, Sanitary Dressing Procedures, SRM removal, Zero Tolerance) inspections based on FSIS standards; and sanitation (Sanitation Performance Standards, Sanitation Standard Operational Procedures both pre- and post-operational). Therefore, actions are in-line to providing the DR with a permanent beef inspections service and beef processing capacities that will certainly prevail beyond the project. V. Results: Change in Policy/Regulations: • Assisted the government to complete and submit the required Self Reporting Tool (SRT) for FSIS to audit the equivalency of its Beef Inspection System, that certainly includes SPS compliance. • Strengthened the Technical Equivalence Working Group composed of businesses, producer groups, and relevant governmental technicians from the ministries responsible for implementing government policy, to guide the regulation reform process. • Helped establish a Presidential Commission for Beef Exportation to the USA, were political will was motivated to provide the required support from the highest authorities • Contributed to the development of new inspection procedures and formularies, and adapted inspection documentation to conform with USDA FSIS requirements. • Trained producers and producer associations as well as public & private sector actors on Trade Policy Analysis. • Facilitated the reform of HACCP systems, SPS programs, and regulations for processing plants, animal welfare, inspectors’ working hours, and overtime payment mechanism (to avoid conflicts of interest). In the framework of the Equivalence process, the ministries SAFE PROJECT – MIDTERM EVALUATION REPORT 127 of Health and Agriculture have been able to advance on the following regulations / procedures along with key stakeholders: o Ministry of Health national resolution n.º 000011- 2018 that regulates slaughter working hours and set the basis for animal welfare (DIGEMAPS) o Development of agreements to pay overhead to inspectors working in slaughterhouses and transfers to be made to a MoH account (DIGEMAPS) o Animals showcased for sale at regional fairs must be certified Brucellosis and TB free (Minister of Agriculture) o National Directorate of Internal Taxes’ General Standard No. 09-2018 permitted the extended the provisions contained in General Standard No. 01- 2008 to designate beef processing plants participating in the Equivalence process as agribusinesses relaxing their schedule for tax payments (motivated by former JAD Executive President) o Procedure established opening of a unique window in the General Customs Directorate that will be used for exports of beef and beef derivatives (along with DIGEMAPS) o Ministry of Health´s new team of auditors for third party laboratories trained and stablished along with their set of procedures for the official evaluation and follow￾up of the laboratories that perform microbiological and residues analytics on beef in the framework of the Equivalence process (DIGEMAPS) o Completion of the new, more simplified official inspection forms (DIGEMAPS). o Completion of the 2019 National Residues Program, by the CVO/CCA, as part of the documents to be shown to FSIS (DIGEMAPS) o Improvement of the new procedure for the evaluation of laboratories conducting beef analytics as part of the Equivalence process (DIGEMAPS) o New DIGEMAPS Protocol for Validation of the HACCP Systems of the Meat Processing Plants participating in the Equivalence process o First evaluations developed by the CVO/CCA regarding the Implementation of Official Functions and Tasks by the Inspection Personnel Assigned to an Authorized Establishment (DIGEMAPS) o Created new protocol to collect and send meat samples to be tested in a network of three domestic and international laboratories including Instituto de Innovación e Biotecnología Industrial (IIBI) to analyze Salmonella and generic E.coli; and the Laboratorio Agroempresarial Dominicano (LAD) for E.coli 0157:H7 and Non-0157 STECs, located in the Dominican Republic; and chemical residue testing by the Laboratorio Nacional de Residuos (LANAR) located in Tegucigalpa, Honduras (DIGEMAPS) o Created new protocol for the validation of Hazards Analysis and Critical Control Points (HACCP) systems and SPS programs in processing plants (DIGEMAPS) Develop Political Will for Risk-Based System • Trained processing plants staff and official inspectors from the Central Competent Authority (CCA), the Ministry of Public Health’s General Directorate for Medicine, Food, and Sanitary Products (DIGEMAPS) on risk analysis • Supported Quality Control staff from processing plants to perform risk analyses to develop new or improve their existing HACCP systems with improved CCPs and sanitation programs. SAFE PROJECT – MIDTERM EVALUATION REPORT 128 Engage Producer Groups • Supported beef producers to define their strategic priorities for exporting products • Rescued and strengthened a Beef Producers Association (Asocarne) that was dormant since 2001 and helped them developed their priorities, highlighting beef exportation opportunities based on a beef markets study the project performed in Puerto Rico • Trained beef producers to improve the quality of their products through activities that encourage quality competition such as prime cuts and marbling. • Trained producers on Good Agricultural Practices, Good Manufacturing Practices, Food Defense, and Best Management Practices. • Adapting the traditional technical assistance approach provided by the governmental extension system to include more practical demonstrations and hands-on learning while reducing the emphasis on theoretical TA has encouraged increased application of technologies and techniques that reduce costs, improve quality and production. • Facilitated links between producer groups and input suppliers such as the U.S. Grains Council (USG) and successful U.S. companies like Organic Valley to encourage changes in the formulation of animal feeds that comply with equivalency standards. without substances prohibited by the FSIS standards including substituting chicken manure for Dried Distiller’s Grains. Train Officials in Risk Management and Surveillance Systems • Trained government inspectors to improve their performance on - risk identification and food safety; evaluating the processing plant’s HACCP plans; conducting ante mortem (general requirements, humane handling) and post mortem (General Requirements, Sanitary Dressing Procedures, SRM removal, Zero Tolerance) inspections based on FSIS standards; and sanitation (Sanitation Performance Standards, Sanitation Standard Operational Procedures both pre- and post-operational) • Encouraged the Central Competent Authority (CCA) to hire more inspectors to strengthen the national inspection system • Persuaded the Dominican government (motivated by former JAD Executive President) to provide funding to calibrate and acquire laboratory equipment and purchase reagents for the National Central Veterinary Laboratory, LAVECEN. • Trained new food safety inspection auditors and the CCA on how to prevent conflicts of interest and to conduct internal audits to verify third-party labs’ practices on FSIS compliance. • Supported the CCA to implement an inspector’s performance evaluation mechanism, and practical versus theoretical approach to staff training, and standardized in-plant evaluations. Improve SPS Laboratory Systems • Fostered commitments from the Dominican government and the private sector to cost share the required laboratory testing during the equivalency evaluation process. • Supported the Government to create a network of public – private laboratories to support the national inspection system: IIBI (public), LAD (private), LANAR (public). • Provided technical assistance to laboratory staff on ISO 17025 standards, quality controls, microbiology & residues testing methods. • Conducted evaluations of labs and established “critical path” strategies to take corrective actions to address any equivalency non-conformities. SAFE PROJECT – MIDTERM EVALUATION REPORT 129 Identified Counterfeit/Poor Quality Inputs and Provided Farmers with Alternatives • Connected producer groups with equivalency compliant input suppliers such as the USGC to substitute chicken manure in animal feeds for Dried Distiller’s Grains. • Executed communication campaigns to mitigate the use of prohibited substances such as the anti-parasitic ivermectin among others. Private Sector Investments • Leveraged $1,132,399 in public-private partnerships including investments in processing plant infrastructure, laboratory equipment, microbiological and chemical residue testing, and others. • $421,298 in cost-share from infrastructure and input investments with producers and producer groups to improve quality and production