Dominican Republic Food for Progress Exporting Quality/Exporta Calidad Program Final Evaluation Submitted: January 3, 2022 Updated: February 16, 2022 ii USDA FFPr Exporting Quality Program Final Evaluation Program: Food for Progress Agreement Number: FCC-517-2015/015-00 Funding Year: Fiscal Year 2021 Project Duration: September 2015-December 2021 Implemented by: International Executive Service Corps (IESC) Evaluation Authored by: Institute for Development Impact Brian Foster, Molly Hageboeck, Elizabeth Issac iii Table of Contents List of Figures................................................................................................................................................iv List of Acronyms............................................................................................................................................ vi Executive Summary .................................................................................................................................... viii Program Background and Purpose ......................................................................................................... viii Evaluation Questions, Design, Methods and Limitations....................................................................... viii Findings, Conclusions, and Recommendations .........................................................................................1 Section 1: Introduction and Purpose.............................................................................................................1 1.1 Program Context..................................................................................................................................1 1.3 Results Framework ............................................................................................................................15 1.4 Purpose of the Evaluation..................................................................................................................16 1.5 Evaluation Questions.........................................................................................................................17 Section 2: Evaluation Design and Methodology..........................................................................................18 2.1 Sampling Methods.............................................................................................................................18 2.2 Data Collection Methods...................................................................................................................22 2.4 Data Analysis Methods......................................................................................................................32 2.5 Evaluation Limitations .......................................................................................................................34 Section 3: Evaluation Findings (F), Conclusions (C), & Recommendations (R)............................................35 3.1 Performance against EQ Program Performance Indicators ........................................................36 3.2 EQ Program Performance in relation to Activity Questions........................................................42 3.3 Broad Results of EQ Activity Participation- EQ Performance Related to Achieving Strategic Objective 1 and Strategic Objective 2 .....................................................................................................44 3.4 Activity Specific Evaluation Questions...............................................................................................58 Annex A: Bibliography..................................................................................................................................97 Annex C: Food for Progress Results Frameworks......................................................................................103 Annex D: Datasets- Submitted Separately.................................................................................................105 Annex E-1: USDA EQ Final Evaluation Work Plan and Instruments-Submitted Separately ......................105 Annex E-2: Additional Sampling Data........................................................................................................106 Annex G: Final Evaluation Terms of Reference .........................................................................................108 iv List of Tables Table 1: Estimated Number of Active Producers & Planted Area in the DR (in Target Value Chains)..........1 Table 2: Value of Global Exports of Ag Origin from DR (2016-2020) (in USD) ..............................................2 Table 3: Fruit and Vegetable Exports from the DR to the US: Value (in USD) of Top 10 Products (in $ 1000)..............................................................................................................................................................3 Table 4: USDA Exporting Quality Program Activities and Interventions (Source: EQ Semi-Annual Reports) .....................................................................................................................................................................11 Table 5: Sampling Distribution for EQ Producer Beneficiary Group, Disaggregated by Value Chain .........19 Table 6: Sampling Distribution for People in Firms (based on EQ Beneficiary List)....................................20 Table 7: Summary of Quantitative Data Sources ........................................................................................21 Table 8: Summary of Qualitative Data Collection Sources and Sampling Distribution for KIIs...................22 Table 9: Main Methods for Data Collection Across the Three Main Groups of Questions.........................22 Table 10: EQ Performance Indicator List, LOP Target, and Actuals Achieved.............................................37 Table 11: Number of Farmers Interviewed According to the Cost and Volume of Production Perceived .33 Table 12: Farmer Perception of Changes in Production Costs, by Value Chain ..........................................34 Table 13: Farmer Perception of Changes in Production Volume, by Value Chain ......................................34 Table 14: Number of Firms Staff Interviewed According to the Cost and Volume of Production Perceived .....................................................................................................................................................................35 Table 15: Certification Program Beneficiaries by Data Source, and Value Chain .......................................38 Table 16: Traceability System Program Beneficiaries by Data Source, and Value Chain............................40 Table 17: Percentage Increase in Yields for IPM Beneficiaries....................................................................45 Table 18: Top EQ Activities/Interventions, Outcomes, and Relation to FFPr Strategic Objectives ............88 Table 19: Final Evaluation Determination of Status of Midterm Recommendations.................................62 List of Figures Figure 1: Geographic Distribution of EQ Program Value Chains and Beneficiaries ....................................14 Figure 2: Farm Sample Distribution in the DR, by Value Chain ...................................................................33 Figure 3: Farm Sample Farm Size (in Hectares)...........................................................................................34 Figure 4: Farmer Sample, Household Size ...................................................................................................34 Figure 5: Year in which Firms First Started Getting Support from the EQ Program (Represented as % of firms)............................................................................................................................................................35 Figure 6: People in Firms Survey Sample, by Value Chain (in %).................................................................35 Figure 7: Online Mini Survey Respondents (Firm Leaders), by Firm Type ..................................................36 Figure 8: Online Mini Survey Respondents (Firm Leaders), by Value Chain ...............................................36 Figure 9: COVID-19 Pamphlet Developed by Rapid Asia for Enumerators .................................................37 Figure 10: EQ Support Received by Farmer Beneficiaries in Sample (by %) ...............................................43 Figure 11: EQ Support Received by Firm Beneficiaries in Sample (by %)....................................................44 Figure 12: EQ Support Received by Firm Leaders in Sample (by %)............................................................44 Figure 13: Farm Beneficiaries' Rate of Application of Improved Techniques (by %) ..................................45 Figure 14: Firm Beneficiaries' Rate of Application of Improved Techniques (by %) ...................................46 Figure 15: EQ Support's Impact on Farm Inputs and Output......................................................................48 Figure 16: EQ Support's Impact on Firm Inputs and Output.......................................................................49 v Figure 17: Farm & Firm Sales Value (in %), for EQ Beneficiaries in Sample ................................................49 Figure 18: Firm Sales Volume (in %), by Year, for EQ Beneficiaries in Sample............................................50 Figure 19: Farm Sales Volume (in %), by Year, for EQ Beneficiaries in Sample...........................................50 Figure 20: Farmer Field School Beneficiaries' Application of EQ-supported Improved Techniques...........63 Figure 21: Renovation and Its Effect on Overall Farm Productivity ............................................................56 Figure 22: IPM Beneficiary Famers in Sample, by Value Chain ...................................................................57 Figure 23: In-Kind Recipients (represented as a %) Improvements in Product Quality and Productivity ..67 Figure 24: In-kind Equipment Recipients' Raw Material Purchase Increase (in %).....................................67 Figure 25: Impact of COVID on Farm and Firm Operations.........................................................................74 vi List of Acronyms Acronym Full Term ADOEXPO Asociación Dominicana de Exportadores, Inc. APHIS Animal and Plant Health Inspection Service CAFTA-DR Central America and Dominican Republic Free Trade Agreement CEDAF Centro para el Desarrollo Agropecuario y Forestal DEC Development Experience Clearinghouse DR Dominican Republic EQ Exporting Quality/Exporta Calidad FAMU Florida Agricultural and Mechanical University FDA Food and Drug Administration FFPr Food for Progress FSMA Food Safety Modernization Act FY Fiscal Year GCCA Global Cold Chain Alliance GoDR Government of the Dominican Republic HA Hectare I4DI Institute for Development Impact IDB Inter-American Development Bank IESC International Executive Service Corps IPM Integrated Pest Management KII Key Informant Interview LOP Life of Project Medfly Mediterranean fruitfly MinAg Ministry of Agriculture MSC Most Significant Change MT Metric Ton RF Results Framework SO Strategic Objective SOW Statement of Work vii Acronym Full Term SPS Sanitary and Phytosanitary Standards USAID U.S Agency for International Development USDA U.S. Department of Agriculture WFLO World Food Logistics Organization viii Executive Summary Program Background and Purpose Exporting Quality (EQ) is a Food for Progress program in the Dominican Republic funded by the U.S. Department of Agriculture (USDA). Initiated in September 2015, this six-year, $16.8 million1 program, which will close at the end of 2021, was implemented by International Executive Service Corps (IESC). It aimed to increase the productivity and sales in domestic and export markets of the following high-value fruit and vegetable global value chains: avocados, cocoa, pineapple, and greenhouse and Asian vegetables, and was designed to improve product quality, increase production efficiency, increase the value of post-harvest products, and enhance marketing and market linkages. I4DI was competitively sourced and selected by IESC and the contract was approved by USDA on July 2, 2021, to conduct the EQ program’s final evaluation and impartially assess whether the project achieved the expected results and met the U.S. Government’s Food for Progress Strategic Objective (SO) 1 – Increase Agricffultural Productivity and Strategic Objective (SO) 2 – Expanded Trade of Agricultural Products (Domestic, Regional, and International). The scope fof the evaluation included: • Assessing the Exporting Quality (EQ) project’s design, implementation, management, and replicability; • Providing recommendations for USDA, EQ participants, and other key stakeholders for future projects; • Following-up on midterm evaluation questions2 , including assessing direct and indirect, intended and unintended, positive or negative impacts; and, • Analyzing the extent to which the EQ program addressed midterm evaluation recommendations and incorporated them into the program’s subsequent activities and operations. The findings of this final evaluation will help USDA design future programs that are reflective of the agricultural context in the DR and monitor similar active projects such as USDA’s FFPr Trade Safe Project; the USDA FFPr PROGANA Project, and future projects in the DR and the region. Findings from this evaluation will also contribute to efforts to address priority questions in the FFPr Learning Agenda: Trade Expansion and Agricultural Market Development in several areas: 1) Market Systems, 2) Quality of Standards, and 3) Risk and Uncertainty, as well as cross-cutting areas namely culture, demographics, and sustainability. The specific findings from this evaluation that contribute evidence to the FFPr Learning Agenda priority questions are referenced alongside the respective findings where they appear in the text. Evaluation Questions, Design, Methods and Limitations This evaluation design and methods were structured to answer three sets of questions posed by IESC about the EQ program. 1 The total budget including commodity and freight costs is $21,108,644. 2 EQ Program Baseline study was conducted in 2016, and midterm evaluation in 2018- both of which were led by Antonio Cordella. ix Evaluation Questions addressed by the evaluation focus, in turn, on: Program results in relation to targets for EQ program performance indicators identified by IESC. • Questions about performance and results achieved under five distinct EQ activities: - Activity 1: Capacity Building: Producer Groups/Cooperatives - Activity 2: Capacity Building: Trade Associations - Activity 3: Cold Chain Improvement - Activity 4: In-kind Grants: Equipment - Activity 5: Public Information Campaign: Disperse Improved Market Information • Questions in relation to a set of internationally accepted criteria for evaluating programs and projects including: relevance, effectiveness, efficiency, sustainability and impact. A detailed list of the evaluation questions in each of these groups can be found in Annex B: Evaluation Questions of this report. Evaluation Design and Data Collection Methods used to collect and analyze evidence needed to answer each of these questions were selected based on the types of data required and the sources from which they could be obtained. The mix of methods selected for this purpose included reviews and data extraction from EQ program documents, particularly the EQ program’s semi-annual reports; 46 key informant interviews (KIIs) with USDA, Ministry of Agriculture personnel in the DR, EQ program staff, and firm leaders; surveys of farmers who participated in the EQ program; # workers in firms and # leaders of firms that participated in the EQ program. In addition to interviews with 46 key informants and implementing partner representatives descripted in this report, the evaluation team collected information from 4753 EQ program beneficiaries selected randomly from farms and firm employees that were EQ beneficiaries. The lists of beneficiaries from whom survey samples were selected were compiled by IESC the program implementing organization. An additional list, this one a census of leaders of firms in the EQ value chain was also prepared by IESC. • The in-person farm survey obtained data from 332 respondents from a stratified random sample of farms across all five value chains constructed on a proportional to value chain population basis. This survey had a 100 percent response rate. • To gather data from employees of firms in the five value chains, the evaluation used a survey stratified by of respondent, with those categories including exporters, packing houses, processors and government. There were 124 respondents to this phone-based firm survey, or a response rate of 72 percent. • The final survey, which focused on the leadership of firms in EQ value chains was conducted as a census by email. Of the 100 senior leaders to which the evaluation team reached out with an email survey, around 20 percent responded. A low response rate is not unusual for email surveys which have much lower response rates than do in-person surveys. These firm leadership survey responses are supplemented, in this evaluation, by KIIs conducted with some among the leaders of firms that were EQ program beneficiaries. The list for KIIs was developed 3 This comprises 332 farmers, 124 people in firms, and 19 firm leaders x independently as part of the study’s key informant identification process which may have resulted in a limited number of leaders partaking in both KIIs and the online surveys. Data Analysis for this evaluation consisted of several approaches each of which was particularly suited for the types of data it was used to analyze. For closed ended survey questions, quantitative analyses included tabulations and cross-tabulations using SPSS and STATA. For open ended questions, including stories gathered using most significant change (MSC) questions about EQ activities, content and pattern analyses were conducted. The integration of quantitative and qualitative data occurred in collaborative sessions with evaluation members representing specific data sets, i.e., program documentation; survey results, and key informant interview results as the team collectively answered each evaluation questions integrating evidence from those methods. Contribution analysis was employed when working with answers to questions that forced respondents to rank the effectiveness of EQ activities or order data on weaker and stronger EQ activities. Study Limitations are identified in evaluation reports to transparently alert readers to issues that may have arisen that would compromise reader’s ability to put confidence in the findings, conclusions and recommendations from an evaluation. Several issues arose during this evaluation which required attention. None, however, were deemed to be major limitations to the validity and reliability of the study’s findings. The few limitations the evaluation team did encounter are summarized below and detailed further in the report: • Lack of Disaggregated Program Data. The EQ Program did not collect value-chain wise disaggregated data for project activities as the program is governed by the 2014 Food for Progress McGovern Dole Indicators and Definitions, which did not require such disaggregation at data collection. This posed a limitation to the evaluation team in not being able to compare value-chain wise data across different times. Secondary data was used where relevant and available to draw these disaggregated data comparisons. • Respondents Not Willing to Disclose Sales Data. Given the sensitivities around revealing sales data, both farm survey respondents and employees of firms frequently declined to answer this type of question, resulting in fewer responses for those items. • Hard-to-Reach Respondents. At the start of the survey of firm workers the evaluation team encountered difficulties connecting by phone with members of the survey sample, many of whom may have worked on jobs that kept them on the floor of their businesses and away from phones. The evaluation team was able to work through this issue. • Duplicate Interviews. As noted above, as a function of parallel yet somewhat distinct key informant interviewing and survey administration under this evaluation, a few individuals in the leadership of firms in the value chains on which the EQ focused may have provided data twice. However, the questions differed in these two collection methods. • Data Collection Sequencing. It can be desirable to develop and deploy evaluation research data collection instruments in a sequency so that findings from one type of data collection process informs the next. Options in this regard were discussed, but in the end the logistics of the study precluded their structured use. • Recall Bias. The specific concern for this evaluation stems from the fact that record keeping is not a common practice among farmers, especially those that do not apply improved practices. Generally speaking, this problem tends to greater for fine grain information, e.g., money. No specific instances xi of recall bias were brought to the attention of the evaluation team for this study by its survey research partner in the Dominican Republic. • Halo Bias. There is also a known tendency among respondents to under-report socially undesirable answers and alter their responses to approximate what they perceive as the social norm (halo bias). The extent to which respondents will be prepared to reveal their true opinions may also vary for some questions that call upon the respondents to assess the performance of their colleagues or people on whom they depend for the provision of services. While this and other bias possibility exist, no instances of this bias were identified to the evaluation team for this study. 1 Findings, Conclusions, and Recommendations This section of the Executive Summary presents the evaluation teams findings, conclusions, and recommendations in relation to each of the sets of evaluation questions listed above. 1. Program Results in Relation to Performance Indicator Targets Findings. To answer this question, the evaluation team drew upon EQ cumulative results records for 35 EQ performance indicators, 20 of which were results indicators and 15 of which were activity indicators. The EQ program exceeded targets for 31 of these indicators; met their targets for two, and failed to meet targets set for another two. Conclusions. The EQ program created a strong and positive record of achieving program indicator performance targets. Recommendations. None at this Macro Level 2. Results Under Five Distinct EQ Activities Activity 1: Capacity Building: Producer Groups/Cooperatives Findings. For firm beneficiaries, evaluation evidence from multiple data sources the team used (surveys, documents, KIIs) indicates that the EQ program’s market-driven certification activity and related trainings on FSMA and SPS helped improved firms’ agricultural certification status and ability to meet US or European production standards and expand exports. The market-driven certification program has been cited as one of the most beneficial interventions offered by the EQ program. Notably, a large number of respondents stated that they intended to maintain the certifications they achieved with EQ program support. Majority of the traceability System Activity beneficiaries reported that EQ’s support on traceability systems helped increase their firm’s access to markets. For farmer beneficiaries, trainings provided by farmer field schools (FFS), and related integrated pest management training provided by EQ helped contribute to increased production yields, production quality, and increased production costs. Revitalization/renovation programs for cocoa and avocado farmers also helped increase production yields for a vast majority of these value chain farmers. Conclusion. All interventions under this activity had their intended effects on farms and firms through, i.e. increased, productivity, quality, market access and exports for firms, and increased yields, and product quality for farms. Majority of farms and firms also saw an associated increase in their production costs and inputs, that can be attributed to increased productivity and volumes. Recommendation. Each of these interventions, have had their intended effects on beneficiaries and as such, sustainability partners that have been identified must make every effort to carry on and expand this Activity 1 suite of EQ support. See Section 3.5 of this report for more information on sustainability partners for the EQ program. Activity 2: Capacity Building: Trade Associations Findings. EQ’s trade show activities have helped open DR firms to a previously untapped foreign market. Respondents cited that EQ’s introduction of trade shows opened their firms to new clients and opportunities they didn’t have previously, giving them exposure to international marketing, and their 2 products new visibility. Virtual trade fairs however drew fewer participants than in-person events with the impact to market knowledge, market access, and exports were reduced. Respondents across multiple data sources reported that virtual trade show experiences are not a great substitute for face￾to-face encounters with potential clients. Findings from multiple data sources in this evaluation converged to indicate EQ interventions increased their exports4 , with key activities cited being EQ’s certification program, and trade shows and trade mission activity. Data sources, especially through KIIs with sector actors indicated there is growing recognition of Dominican ‘fine’ organic chocolate in the domestic market, but this growth did not yet have discernible impact to exports. Multiple data sources reported that the Young Exporters Network and Networking for Women in Trade Leadership programs helped foster a culture of women entrepreneurship and engagement with women and youth in the export sector. Conclusion: Overall, this activity saw high participation among firms, with vast majority reporting an increase in market knowledge, increase in market access, and an increase in exports, thus had its intended impact on expanding market access and exports for firms. EQ interventions had a multi￾dimensional positive effect on exports and value chains through its various interventions such as the market-driven certification program, and trade shows and trade mission activity. Recommendations. Although increased exports cannot be directly attributable to EQ program interventions alone survey results suggest that multi-dimensional approaches have had a positive impact and should be considered in future programming. Activity 3: Cold Chain Improvement Findings. EQ investments in training, technical assistance, study tours, in-kind equipment grants, and demonstrations, in collaboration with World Food Logistics Organization/Global Cold Chain Alliance (WFLO/GCCA), resulted in widespread awareness of the importance of maintaining cold chain infrastructure from the farm to the exporter to increase exports of fresh produce. A high percentage of EQ beneficiaries who participated in cold chain activities indicated product quality had increased resulting in increased exports. There was also recognition by the federal government about the importance of cold chain development; with EQ support, the GoDR has drafted a national decree aimed at strengthening cold chain infrastructure in the country. Notably, however, among firms that responded to evaluation surveys, participation in EQ’s cold chain activities was relatively low, with only six percent of people surveyed in firms having received any cold chain support. Those who did were concentrated in greenhouse vegetables, avocados, and Asian vegetables. Structurally, participation in cold chain activities was evenly distributed among packinghouses, processors, exporters, and government. A small proportion of firm leaders were also familiar with and worked on EQ’s cold chain pilot program. Respondents to the firm survey, as well as other sources all stated that cold chain training improved product quality. Conclusions. EQ’s development and dissemination of the cold chain pilot program were effective and led to compliance. In terms of this training’s sustainability, some key informants in cold chain indicated the continuation of chain training, and investment in cold chain improvements was likely. Although a 3 relatively small set of EQ interventions, it is likely that the cold chain trainings along with the development of a draft national cold chain strategy will produce outsized benefits to the country’s fresh produce export sector. From survey and KII results, the private sector is going to sustain these efforts, but support of the national strategy will require donor support to finalize the draft decree and bring about implementation of GoDR activities to continue to support cold chain development. Recommendations. The types of cold chain strengthening activities the EQ program supported should be supported in the future by Government and the private sector. In addition, the work done under EQ with the GoDR to develop a national cold chain strategy should be continued under other programming. Activity 4: In-kind Grants: Equipment Findings. The in-kind equipment grants activity provided farmers and firms with a wide array of equipment intended to help increase productivity, enhance efficiencies, and improve product quality. Across all data sources, including mini-survey, KIIs, and quantitative surveys, over two-thirds of respondents reported increased product quality. A high proportion of participants reported increased productivity. Around two-thirds of key informants from cocoa trade associations reported increased volume and immense value in receiving pruning equipment from EQ. All firm in-kind equipment recipients who reported increased productivity also reported an increase in raw material purchases, while over 90 percent of farm in-kind equipment recipients who reported increased productivity also indicated increased raw material purchases. Conclusion. EQ Program had its intended effect on productivity. The EQ program’s investment in in-kind tools and equipment not only leveraged firm resources, but it also yielded benefits to the economy by stimulating increased purchases of raw materials and inputs. Recommendations. To sustain activities that include an equipment grants component, it is recommended that farmer associations, commodity groups, firms, and Government engage to a greater extent, building on EQ’s successes. Additionally, the GoDR should consider policies (tax, investment, import) that support the identification, demonstration and access to equipment, tools and technologies that improve production, post-harvest handling and processing of agricultural commodities, the benefit of which was demonstrated by EQ. Activity 5: Public Information Campaign: Disperse Improved Market Information Findings. EQ utilized multiple media sources including national TV and radio outlets, as well as social media to reach consumers with food safety campaign messaging. EQ program documents report that 5.7 million consumers reached through the Consumer Food Safety Campaign. This shows that the program significantly exceeded EQ target of 1.5 million for this is activity (Activity indicator 5.1.) Key informants in trade associations were familiar with this campaign and 100 percent of them asserted that it reached consumers with very targeted messaging. However, around 83 percent of the key informants in the cocoa value chain interviewed did not know about this campaign. Conclusions. The methods utilized under EQ were effective in delivering the consumer food safety information to the public. However, while respondents associated information received with the Campaign, no specific behavior changes were attributed by them to the EQ Consumer Food Safety 4 Campaign. Without undertaking a follow-up consumer survey, this evaluation team has no means for determining the Campaign’s the impact on consumer behavior. Recommendation. Consumer awareness campaigns should include a follow up element (including a consumer survey) to determine their effectiveness. 3. EQ Performance on Internationally Accepted Program Evaluation Criteria Relevance Findings. The work undertaken by EQ through its five activities were well aligned and contributed to Food for Progress Strategic Objective (SO) 1 – Increase Agricultural Productivity and Strategic Objective (SO) 2 – Expanded Trade of Agricultural Products. At a higher level, the EQ activities also contribute to overall FFPr DR’s Learning Agenda focus areas namely 1) Market Systems, 2) Quality of Standards, and 3) Risk and Uncertainty, as well as cross-cutting areas namely culture, demographics, and sustainability. The program was also aligned in a limited capacity with USDA PROGANA program in adopting the farmer field school model, and collaborated briefly with USAID FAST program implemented by USDA and Food and Drug Administration (FDA) on programming some FSMA readiness activities. Strong collaboration between EQ and USDA/APHIS was evident as part of follow-up activities after the Medfly outbreak. More than 80 percent of respondents representing the entire spectrum of EQ beneficiaries, namely producers/farmers, people in firms and firm leadership representing processors, packinghouses, exporters, trade associations, and government, reported that EQ addressed the core issues of target beneficiaries. Some key informants indicated the EQ may have been too broad and not deep enough, i.e. trying to do too much for too many beneficiaries, with one interviewee calling it a “first come-first served” approach that was not strategic enough in proportioning out project resources. As mentioned previously, as the vast majority of respondents, directly responded that EQ was able to address their core issues, this indicates a high convergence across data sources and activities. Conclusion. EQ program reflects the partnership aspirations of the US Government’s 2016 commitment to global development through its collaboration, with a) other US government programs, and b) the DR’s development policies and strategies that aim to promote a profitable and competitive agricultural sector in national and international markets. Based on the convergence in responses from the majority of respondents representing various beneficiary groups, it is evident that the EQ program was well￾designed for addressing the core issues that target firm and farm beneficiaries perceived. Recommendation. EQ’s model of collaboration with different stakeholders, including key functionaries in the government is worth replicating for future USDA programming. The commitment of GoDR resources necessary to sustain some EQ activities into the future should be a condition of continued USG support. Effectiveness Findings. EQ met or exceeded program targets on all but 2 indicators. Virtually all interventions led by EQ yielded demonstrable effects on program and FFPr strategic objectives: increased productivity and increased exports. Some interventions, such as the renovation programs, networks for youth and women, were not mature enough to yield demonstrable effects. 5 Based on findings from various data sources, review of program documentation, and contribution analysis, the main intervention that provided the most benefit to farmers and firms alike are EQ’s various training programs. Multiple survey respondents and key informants reported that they would like to receive more training due to the positive effects it has had on their operations. Table 26 in Section 3.5 additionally details a disaggregated view of the top activities and/or interventions based on beneficiary responses across various data sources. Table 26 also outlines the key outcomes of those activities, and the relevant results framework pathway taken by those activities to achieve FFPr strategic objectives (SOs) 1 and 2. See Annex C: Results Framework for more details. Conclusions. Program was well-designed and implemented to address program goals and objectives in light of the core challenges the target beneficiaries. Recommendations. Recommendations specific to activities under EQ are included in the body of the evaluation report. Efficiency Findings. The responses from several different key informant interviews indicate a consensus that this program performed well with the resources allocated to it. None of the respondents across data sources suggested that a higher or lower program budget for EQ would have produced more effective outcomes more efficiently without adversely affecting the impact and reach of the program. No conclusions or recommendations has been provided for this criterion. Sustainability Findings. One key Government interviewee said, “EQ left behind significant social capital” in the country. That is high praise and may be the most important contribution to the sustainability of the many activities undertaken by EQ. EQ certainly “laid the groundwork” (in the words of a private sector manager) for many improvements in fresh produce quality that are critical for increasing exports. Sustainability planning has been part of the EQ process. The evaluation team received a sustainability action plan from the EQ program team, that identified initial sustainability partners for Activities 1, 2, and 3. This indicates, that sustainability planning was pursued for all activities. Yet, as the program closes out a variety of well-regarded activities, as noted in the report, do not appear to have found permanent homes or follow-on sponsors. Conclusions. Sustainability challenges post-EQ are significant, especially where Government entities and reliance on Government resources are involved. Activities such as farmer training, pest surveillance, SPS monitoring, support for certifications and development of traceability systems, among others, will suffer due to the lack of engagement by the GoDR, especially the Ministry of Agriculture. It was reported that 95 percent of all training in cocoa over the past five years was supported by EQ. Activities that engaged the private sector are more likely to be carried on after EQ; cold chain actors uniformly indicated they will continue training and capacity building in cold chain management as well as increase their investments in cold chain infrastructure. And in the cocoa/chocolate sector, there are indications that industry may be able to come together to carry on the Dominican fine chocolate promotion and product placement activities, including the chocolate festival and competition. As 6 mentioned above, greenhouse vegetable growers are working on plans to carry on the production of insect bio-control agents as well as launch their own laboratory for quality testing. Regarding the development of sustainable partners, a few public sector partners have been identified to sustain some activities for example, ADOEXPO’s role in sustaining the women and youth networks; CEDAF’s commitment to sustain the work of the farmer field schools and supporting firms with traceability systems; certified avocado production through Eurofresh Dominicana, Min of Ag’s extensionists, and the AGROFAMILIA5 Sierra de Neyba project; IPM activities through Pre-inspecciona and IPM Min of Ag’s programs and the TraSa Project. For other interventions, no clear sustainability partners were identified through the evaluation process. From data gathered by the evaluation, the team concluded that finding sustainable partners in the government is less likely, despite expressions of interest, because of uncertainty of available government resources. Recommendations. For interventions that require a sustainability partner, the process for identifying and solidifying partners needs to begin early in the program with the objective of institutionalizing activities, sub-activities and program elements to the extent they are feasible and appropriate. To better understand and foster prospects for sustainability, USDA should incorporate a requirement for activity-level sustainability plans and exit strategies, and mid-term reviews of progress in relation to each of them, inclusive of modifications that may be needed to account for contextual changes. To facilitate a thorough review of sustainability plans and processes, USDA might consider organizing a sustainability/program close-out conference in the last six months of future programs. The objective of such an event would be to review sustainability planning, share lessons learned from program implementation, and engage Government and other local institutions in future program planning. Impact Findings. Some of the key effects that emerged from the findings of this evaluation are outlined below: Immediate-term effects: Medfly surveillance and eradication re-opened the US market; in-kind equipment grants quickly led to improvements in productivity, product quality and process efficiencies Medium-term effects: Farmer training that increases productivity; development of new markets for tropical avocado; introduction of SPS training and IPM modules that will reduce pesticide use & residues; introduction to trade fair participation leading to increased exports; certifications, FSMA￾readiness and traceability systems all contribute to increased market access and exports Long-term effects: Cold chain training and private sector infrastructure have changed the cold chain situation in the DR; fine chocolates industry will grow in volume and sophistication; increased networking among women and youth will lead to their increased participation in the ag economy 5 AGROFAMILIA is an agricultural productivity program with a global scope implemented by the Aliade Global Corporation with headquarters in Spain. Since April 2021 the program has assisted avocado producers in San Juan, Azua and Elías Piña provinces. FFPr Learning Agenda Priority Question #17: How can market driven public/private partnerships help ensure long-term sustainability of programs? It is through working early and consistently with local partners like CEDAF, ADOEXPO, and others right from the early stages of the program that EQ was able to be find reliable partners who can carry on the various EQ interventions even after the program closes. 7 No high-level recommendations or conclusions have been cited for this criterion. Inclusivity Findings. EQ final evaluation surveys and interviews indicated widespread concern about the next generation of producers, packers, and exporters as well as the generally low participation of women throughout the five targeted value chains. EQ focused several activities to support and engage women and youth in agriculture, including support for networking by the Women in Trade Leadership and the Young Exporter Network groups. Conclusion. Participants noted that both groups resulted in an increased awareness of the issues around inclusivity in agriculture, and resulted in valuable networking contacts. The Young Exporter Network intends to continue post-EQ on an informal basis, and the Women in Trade Leadership group is being supported by ADOEXPO. Recommendations. None. Additional details on findings, conclusions and recommendations are included in Section 3 of this report. 1 Section 1: Introduction and Purpose 1.1 Program Context Overview of the Dominican Republic’s Trade, Exports, and Value Chains In recent decades, the Dominican Republic (DR) has made significant economic progress by embracing a flexible and open trading system. This has resulted in impressive economic growth rates, with the DR outperforming most other countries in the Latin American and Caribbean region6 . The country's entry into the regional Central America and Dominican Republic Free Trade Agreement (CAFTA-DR) with the U.S. in 20077 resulted in a phase-out of most tariffs on fruits and vegetables which enhanced two-way trade between the DR and U.S. Significantly for the DR, as many of the most important agricultural exports are fresh fruits and vegetables, CAFTA-DR required all trading parties to apply science-based disciplines of the World Trade Organization (WTO) Agreement on Sanitary and Phytosanitary (SPS) Measures” and to work to resolve measures that restrict trade. Today, agricultural exports from the DR to all markets constitute about 18 percent of the country’s total merchandise exports, or about $2 billion per year. The leading agricultural exports from the DR are cigars, cocoa, bananas and plantains, rum, and avocados8 . USDA’s Exporting Quality (EQ) program, initiated in 2016, aims to further enhance DR exports under CAFTA-DR. The program is notable for its attention to product value chains (avocados, pineapples, cocoa, and greenhouse and Asian vegetables that are produced by small-scale farmers as well as commercial producers. The markets for these targeted products include domestic consumers, the country's large tourism industry, and exports. A summary of the number of producers, area cultivated, and area per producer for the five targeted value chains from the EQ program’s 2016 baseline study is reproduced in Table 1 with data from 2014. At that time, the average size of these farms ranged from 4.7 hectares for pineapples to 0.2 hectares for greenhouse vegetables. Table 1: Estimated Number of Active Producers & Planted Area in the DR in 2014 (Target Value Chains) Value Chain Number of Farmers Area (in Hectares) Average Area/Farmer Avocado 15,000 38,000 2.5 Pineapple 700 3,300 4.7 Cocoa 36,000 125,000 3.5 Greenhouse Vegetables 4,800 870 0.2 Asian Vegetables 5,000 2,500 0.5 Total Target Value Chains 61,500 169,670 2.8 6 Trading development or developing trade? The Dominican Republic's trade, policies, and effects in historical perspective." Historia Caribe (2013) 7https://ustr.gov/archive/assets/Trade_Agreements/Regional/CAFTA/Briefing_Book/asset_upload_file119_7211.pdf 8 Harvard Atlas 2 Changes in global exports for some of these target commodities over the life of the EQ program (2016 to 2020) are summarized in Table 2. The export trend over this period is strongly positive for most of these products in most years, despite the Covid-19 epidemic in 2020. The performance of avocado (greater than 100 percent increase) and cocoa (65 percent increase) exports over that period are especially strong. Table 2: Value of Global Exports of Ag Origin from DR (2016-2020) (in USD9 ) Product 2016 2017 % change 2018 % change 2019 % change 2020 % change % change 2016- 2019 Value (USD) Value (USD) Value (USD) Value (USD) Value (USD) Avocado 33,442,138 48,859,776 46% 51,145,821 5% 61,415,732 20% 67,612,167 10% 102% Pineapple 5,109,445 4,353,964 (15)% 4,113,638 (6)% 5,502,319 34% 5,954,801 8% 16.5% Cacao³ 116,882,016 146,337,847 25% 214,017,277 46% 195,029,391 (9)% 192,570,095 (1)% 65% Eggplant 1,634,697 1,490,920 (9)% 3,485,878 134% 1,278,944 (63)% 2,147,290 68% 32% Chili peppers 16,046,125 17,690,983 10% 18,648,276 5% 19,897,666 7% 25,092,489 26% 56% Tomato 14,492,070 17,642,089 22% 18,430,211 4% 17,775,764 (4)% 21,782,467 23% 50% The five EQ program target value chains represent more than a fifth of the Dominican Republic’s agricultural crop value, with cocoa beans being the most significant (9 percent of crop value and 6 percent of agro-GDP), followed by avocados (6 percent of crop value), pineapples (5 percent), greenhouse produce (1 percent) and oriental vegetables (0.3 percent). These products also contribute significantly to the commercial balance of the country, representing some 15.4 percent of agro-food export value, with cocoa beans accounting for an 11 percent share, greenhouse produce for 2.5 percent, avocados for 1.2 per cent, oriental vegetables for 0.6 percent and pineapple for 0.2 percent.10To illustrate the importance of the U.S. market to Dominican producers and food exporters, Table 3 shows the country’s top ten fruit and vegetable exports (by value) to the U.S. from 2016 to 2020. Of the top 10 food exports to the U.S., the top four (avocados, tomatoes, peppers, and cucumbers) as well as number seven (fresh vegetables) were targeted by EQ program interventions. The leading export to the U.S., avocado, has more than doubled in value from 2016 to 2020 to over $42 million per year; the U.S. represents 62 percent of Dominican avocado export sales. Tomato exports to the U.S. increased 64 percent over the same time period and the U.S. represents 80 percent of Dominican exports. 9 Source: Directorate General of Customs (DGA), Department of Statistics; Prepared by Ministry of Agriculture of the Dominican Republic, Department of Agricultural Economics and Statistics 10 IESC: “Exporting Quality and Safety Program – Baseline Study” (February 11, 2016) 3 Table 3: Fruit and Vegetable Exports from the DR to the US: Value (in USD) of Top 10 Products (in $ 100011) Product 2016 2017 2018 2019 2020 1 Avocados 20,077 32,344 36,660 36,599 42,172 2 Tomatoes, Fresh 10,675 12,545 14,696 13,432 17,474 3 Peppers, Fresh 13,048 13,780 11,809 14,325 16,914 4 Cucumbers, Gherkins 6,761 6,741 7,150 6,838 7,604 5 Sweet Potatoes 6,438 7,957 8,873 8,371 5,762 6 Guavas, Mangoes 810 1,198 1,966 3,307 3,942 7 Vegetables, Other, fresh or chilled 2,044 1,372 1,187 2,151 3,294 8 Coconuts, Other 1,340 933 2,231 2,338 2,581 9 Coconuts (Desiccated) 3,340 3,876 2,402 2,230 2,490 10 Pumpkins, squash, fresh or chilled 2,813 2,852 1,683 2,149 2,397 Avocado Value Chain. Avocado production is a significant business in the DR, with much of the output consumed domestically. Exports play an essential role for commercial growers due to the high prices obtained during the winter season in the U.S. market. Avocado plantings in the DR have gradually increased over the decades, from around 2,000 hectares in the 1970s to more than 38,000 hectares in 201412 . As noted above, avocado is the leading fresh produce export from the DR to the U.S., with the American market taking 60 percent of the DR's global avocado exports in 2020. These production data as reported by the International Trade Centre (ITC) and included in IESC’s October 2016 avocado value chain analysis and February 2016 baseline report do not square with FAOStat data and are particularly divergent with respect to area under cultivation. FAOStat reports about half the number of hectares under cultivation in the DR, and therefore, avocado yields are reported to be much higher than on-the-ground checks. For example, IESC reports 38,000 Ha of avocado under cultivation in 2014 while FAOStat indicates only 12,927 Ha. While total production is about the same in 2014 (FAOStat reports 513,961 MT and ITC 428,300 MT), the calculated yields (FAOStat is 39.8 MT/Ha vs. ITC is 11.27 MT/Ha) are vastly different. We provide this caveat to distinguish between the data presented in program reports which formed the basis of working assumptions at the outset of activity implementation which conflicts with data from equally well-known and reliable sources. Ministry of Agriculture sources indicate overall avocado yields are currently 12-15 MT/ha and considered quite high and comparable to those in Mexico, the world’s leading avocado exporter. FAOStat reports Mexico’s average avocado yield in 2019 as 10.7 MT/Ha13 . 11 Source: USDA/Global Agricultural Trade System (GATS) 12 Dominican Republic Exporting Quality Program – Semiannual Report No. 12, May 2021 quoting Oficina Nacional de Estadìstica (ONE), 2014 data 13 https://www.fao.org/faostat/en/#data/QCL 4 Figure 1 shows avocado planted area and yield from 2009-2019 in the DR and, given the caveat explained above, is included to indicate production trends, only. Figure 1: Avocado yield and planted area, 2009-2019, Dominican Republic (FAOStat) As noted above, avocado is the leading fresh produce export from the DR to the U.S., with the American market taking 60 percent of the DR's global avocado exports in 2020. It is important to distinguish between the two main cultivars of avocado grown in the DR, the green-skinned (tropical) avocado which accounts for about 60 percent of production, and the dark green, rough-skinned Hass avocado (15 percent)14 . IESC’s 2016 avocado value chain analysis15 indicated that production at the farm level is characterized by inconsistent adoption of best management practices (pruning, irrigation, fertilization, pest control) and that a limited number of famers (perhaps only five percent) are organized into producer groups. The analysis adds that farmers generally lack market information that hampers their ability to sell their product at favorable prices. The value chain study recommended that Dominican producers focus on branding and marketing of their green-skinned avocados but also expand production of the more well-known Hass avocado. The authors recommend training on good agricultural practices at the producer level as well as certification in Global G.A.P. to access additional export markets. Pineapple value chain. Pineapple production in the DR is recovering since two major international producers left the country in the early 1990's. At that time, the country was the leading pineapple producer and exporter in the Caribbean and Central American region. Since then, the Government of the Dominican Republic (GoDR) with assistance from donors such as the Inter-American Development Bank, USAID, and the EU have supported the introduction of new pineapple varieties, especially the MD￾2 from Costa Rica, in an effort to recover production and export markets. Due to high local market prices 14 IESC: “Exporting Quality and Safety Program – Baseline Study” (February 11, 2016) 15 IESC: “Exporting Quality and Safety Project Dominican Republic – Global Value Chain Analysis: Avocados” (October 2016) 25,000 30,000 35,000 40,000 45,000 50,000 6,000 7,000 8,000 9,000 10,000 11,000 12,000 13,000 14,000 15,000 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 Kilogram (kg)/hectare (ha) Hectares (ha) Year DR Avocado Yield and Planted Area (Source: FAOStat) Area (ha) Yield (kg/ha) 5 relative to exports, pineapple16 is still primarily sold in the domestic market in the DR, with exports of merely $6 million in 2020 and slow growth (16 percent) relative to other agricultural exports from 2016- 2020. Domestic pineapple production is still oriented to primarily supplying the domestic market, which is much more lucrative and less demanding with respect to product quality than the international one. Indeed, Dominican pineapple output is mainly consumed domestically, with domestic supply unable to meet potential export demand and domestic sale prices equal to those fetched on export markets once transportation costs are considered. However, a small group of producers seems to be targeting foreign trade once the output reaches adequate volume to permit sea transportation to the EU. A small percent share of pineapple output is processed in juice and (frozen) pulp for sales to domestic retailers or further processing17 . According to the IESC value chain analysis of pineapple in the DR, expansion of production in the country is possible because of the advantages of land availability, favorable agro-climatic conditions, and access to new, quality genetic material adapted to world markets18 . Challenges to growth of the sector include weak Ministry of Agriculture support in GAP, IPM and extension outreach; poor producer production practices; lack of exporters and marketing efforts to supply consistent quantities of fruits to world markets; and limited processing capacity19 . As noted with avocado (above), there is a serious discrepancy among Ministry of Agriculture, Oficina Nacional de Estadística (ONE), and FAOSTat with respect to DR’s pineapple production, area under cultivation, and yields. For example, for 2014, MinAg/ONE report 350,000 MT of pineapple produced on 3,300 Ha for an average yield of 106 MT/Ha20. FAOStat for the same period reports 436,304 MT produced on 8544 Ha for an average yield of 51 MT/Ha21. The IESC pineapple value chain analysis report postulates that because pineapple produces twice per year, harvested area has been double counted by FAOStat. If that is the case, then the calculated yields between the two sources agree quite well. In any event, FAOStat data shown in Figure 2 shows planted area and yield trends for pineapple in the DR; these data show that from 2009-2016 planted area to pineapple stayed relatively flat while yield increased significantly. 16 https://www.fruitrop.com/en/Articles-by-subject/Full-country-profile/2014/Pineapple-in-the-Dominican-Republic 17 IESC: “Exporting Quality and Safety Program – Baseline Study” (February 11, 2016) 18 IESC: “Exporting Quality and Safety Project Dominican Republic – Global Value Chain Analysis: Pineapple” (July 14, 2016) 19 ibid 20 IESC: “Exporting Quality and Safety Program – Baseline Study” (February 11, 2016) 21 https://www.fao.org/faostat/en/#data/QCL 6 Figure 2: Pineapple yield and planted area, 2009-2019, Dominican Republic (FAOStat) Cocoa value chain. Cocoa and cocoa products are one of the DR’s most important agricultural export commodities, second in value only to cigars. About 36,000 small-scale cocoa growers in the country produce between 54,000 and 70,000 metric tons (MT) of high-quality cocoa each year. Production in the DR is largely for export (98 percent of production, mostly as beans) with a value of nearly $200 million per year. About 38 percent of cocoa bean exports are organic – the DR is the world's leading exporter of organic cocoa22 . In addition to the country's exports of cocoa beans and limited cocoa products, the tourism industry represents a growing opportunity for the country’s artisanal chocolate producers. Numerous small and medium-scale chocolate processors and related cocoa farms market “bean-to-bar" chocolate products and offer cocoa tours and chocolate-making classes to tourists visiting the DR's many beach resorts; the tourist industry is a growing niche market for cocoa and chocolate in the DR. According to IESC’s value chain analysis of the DR’s cocoa sector, the future is bright for the country’s production of organic, fine-flavored cocoa23: “…it is widely forecasted that a shortage in global cocoa supply will develop further and continue to grow. Even under current market conditions, cocoa, and its primary final product, chocolate, hold strong international demand and a thriving commodity futures market which supports a relatively stable pricing regime. The global market for fine or flavor cocoa is markedly unsaturated. In the world of cocoa, there is a market for every bean produced. The Dominican Republic is positioned like no other cocoa-exporting nation to take advantage of these dynamics.” To take advantage of the DR’s unique position in the cocoa world, the value chain analysis report makes these two critical recommendations: 1) Increase the quantity and quality of farm-level cocoa production, which denotes improvements in farming (i.e. improve operations at the farm level: farm materials and farm systems); and 2) Increase processing of more finalized products through 22 Fine Chocolate Industry Association and https://portalexport.wordpress.com/2014/03/24/the-production-of-organic-cocoa￾in-the-dominican-republic/ 23 IESC: “Exporting Quality and Safety Program Dominican Republic – Global Value Chain Analysis and Strategic Advisory for the Cocoa Sector” (August 2016) - 10,000 20,000 30,000 40,000 50,000 60,000 7,000 8,000 9,000 10,000 11,000 12,000 13,000 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 Kilogram (kg)/hectare (ha) Hectares (ha) Year DR Pineapple Yield and Planted Area (Source: FAOStat) Area (ha) Yield (kg/ha) 7 infrastructure investments, which denotes assisting post-harvest processors in systems and infrastructure investments. FAOStat data indicate a modest decrease in land area planted to cocoa from 2009 (153,219 Ha) to 2019 (150,943 Ha) but increasing productivity in the same time period from 359 kg/Ha to 589 kg/ha. The recent Gaia Cacao supply case study done for IESC under the EQ program indicates increased on-farm productivity of cocoa beans in the DR from 226 kg/Ha in 2005 to 620 kg/Ha in 2018, as well as increased area under production from 125,600 Ha in 2002 to 172,700 Ha in 202024. Gaia Cacao reports that the average cocoa yield of producers surveyed in 2021 for their study was 566 kg/Ha. Figure 3 (below) uses FAOStat data. Figure 3: Cocoa yield and planted area, 2009-2019, Dominican Republic (FAOStat) Cocoa yields are especially variable due to differing levels of management and farm inputs, shade management, weather (especially rainfall), levels of disease, planting density, and cultivars. Farms manage an estimated range of approximately 200 to 1,500 kilograms of cocoa production per hectare; those with higher productivity are using clonal material and have already switched to improved genetics and farming systems25 . For example, cocoa yields in the DR dropped from 415 kg/Ha in 2004 to 226 kg/Ha in 2005 due to the effects of Hurricane Jeanne, and yield dropped from 2010 to 2011 due to drought. Greenhouse vegetables. Greenhouse vegetable production in the DR is primarily an export-oriented business directed at the United States market and targeting higher prices in the American off-season (winter). The major crops grown in greenhouses are tomatoes, bell peppers, cucumbers, zucchini as well as some Asian vegetables, especially chili (hot) peppers. This sector has grown significantly in the past decade due to the adoption of innovative technologies such as hydroponic production, use of seeds of improved cultivars, and improved greenhouse management. 24 Gaia Cacao B.V.; “Dominican Republic Cocoa Supply Case Study” (November 15, 2021) 25 IESC: “Exporting Quality and Safety Program Dominican Republic – Global Value Chain Analysis and Strategic Advisory for the Cocoa Sector” (August 2016) 300 350 400 450 500 550 600 650 146,000 148,000 150,000 152,000 154,000 156,000 158,000 160,000 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 Kilogram (kg)/hectare (ha) Hectares (ha) Year DR Cocoa Beans Yield and Planted Area (Source: FAOStat) Area (ha) Yield (kg/ha) 8 FAOStat data for vegetable production and productivity are not comparable to the data collected by IESC for the baseline study; IESC used Oficina Nacional de Estadística (ONE) data for greenhouse production and International Trade Centre data for exports. FAO does not separate out greenhouse vs. field production of tomatoes, for example. IESC reported 4,800 greenhouse producers with 870 Ha under cultivation (Table 1), whereas FAOStat indicates fairly flat production of around 7,000 Ha of tomatoes from 2009-2019 and yields fluctuating from 30 to 66 MT/Ha (30,000 to 66,000 Kg/Ha), averaging 40 MT/Ha (40,000 Kg/Ha) (see Figure 4). Figure 4: Tomato yield and planted area, 2009-2019, Dominican Republic (FAOStat) Asian vegetables. Asian vegetables have also witnessed increased production and exports to supply ethnic markets in the U.S., Canada, and the European Union in part because of the adoption of enhanced cultivation methods and imported seeds. The production of Asian vegetables consists mainly of chili (hot) peppers, various types of eggplant, okra, bitter melons, long squash, long beans, and others. There is no recent (specific) statistical data on area cultivated with oriental vegetables, the last available data being about 2,500 hectares and a production of 16,700 MT (2006). Current annual production has been estimated at about 20,000 MT. Initially, yields of Oriental varieties were well below international average, but after decades of experiences, the producers of these vegetables have managed to improve their productivity via the use of imported seeds and enhancement in the methods of cultivation26 . With respect to chili peppers, FAOStat data indicates an increase in area planted from 7,500 Ha to 8,700 Ha over the period 2009-2019, with yields flat over that time frame, averaging 5,500 kg/Ha (see Figure 5). 26 IESC: “Exporting Quality and Safety Program – Baseline Study” (February 11, 2016) 25,000 30,000 35,000 40,000 45,000 50,000 55,000 60,000 65,000 70,000 3,000 4,000 5,000 6,000 7,000 8,000 9,000 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 Kilogram (kg)/hectare (ha) Hectares (ha) Year DR Tomato Yield and Planted Area (Source: FAOStat) Area (ha) Yield (kg/ha) 9 Figure 5: Chili pepper yield and planted area, 2009-2019, Dominican Republic (FAOStat) Eggplant production is similar to chili peppers, FAOStat indicating flat production area (around 4,000 Ha) and yields (average 5,200 kg/Ha) over the period 2009-2019 (see Figure 6). Figure 6: Eggplant yield and planted area, 2009-2019, Dominican Republic (FAOStat) The leading vegetables exported from the DR are all peppers ($25 million in 2020) and tomatoes ($22 million in 2020), with the U.S. taking approximately 67 percent of pepper exports and 80 percent of tomato exports- see Table 3 on page 3 above. Challenges to the DR’s agricultural export sector. The humid climate and year-around high temperatures in the DR are beneficial to crop growth and development but also represent major challenges to the five value chains of interest in the form of losses from crop pests and diseases. As such, according to the Ministry of Agriculture, one of the biggest on-going challenges for exports of fresh produce from the country is the presence of residues due to overuse and/or misuse of chemical 4,600 4,800 5,000 5,200 5,400 5,600 4,000 5,000 6,000 7,000 8,000 9,000 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 Kilogram (kg)/hectare (ha) Hectares (ha) Year DR Chili Pepper Yield and Planted Area (Source: FAOStat) Area (ha) Yield (kg/ha) 3,000 3,500 4,000 4,500 5,000 5,500 6,000 6,500 7,000 3,000 3,500 4,000 4,500 5,000 5,500 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 Kilogram (kg)/hectare (ha) Hectares (ha) Year DR Eggplant Yield & Planted Area (Source: FAOStat) Area (ha) Yield (kg/ha) 10 pesticides as well as insects such as thrips that can result in the closure of markets or the rejection of export shipments by the receiving country. The Mediterranean fruitfly, or Medfly (Ceratitis capitata) is a particularly serious and damaging insect pest; its detection in the DR in March 2015 resulted in an immediate ban on avocado, tomato and pepper exports to the U.S. and resulted in extraordinary efforts (and significant cost) for surveillance and eventual eradication of the pest. In general, a lack of knowledge of sanitary and phytosanitary (SPS) regulations and the processes and documentation needed for addressing SPS issues, particularly the requirements for export markets, is a factor holding back exports of numerous commodities from the DR. Improved value chain coordination in the five targeted value chains is needed to add more value to raw commodities and to address inefficiencies and reduced competitiveness, including improper or insufficient post-harvest handling and storage of commodities, lack of processing, and a shortage of cold chain infrastructure. Specifically, the EQ baseline study, through interviews with producers, processors, packinghouses, exporters, and government officials, identified twelve main challenges in the five targeted value chains, including: certifications for Global GAP (Good Agricultural Practices) and Organic produce; water conservation; delay in the transfer of technical know-how; failure of obtaining quick results from analysis of pesticide residues; shortage of reliable export outlets; delay in the introduction of processing technologies; insufficient orchard renovation; delay in fabricating alternative raw or semi-processed cocoa based produce for the local market. Women participation in the five targeted value chains is small in percent but not insignificant. Women have significant roles either as owners or as association executives in all the target value chains and are particularly strong in the processing of cocoa beans and their commercialization. In addition, women account for a significant percent share of packinghouse employees. A few women-only associations are active in the processing of cocoa beans27 . USDA and Donor Programming in the DR. The design of the USDA FFPr Exporting Quality Program was not informed by previous donor programming in the DR, however, subsequent programs like the USDA FFPr Trade Safe (TraSa) Project, which is also implemented by IESC has been largely informed by the EQ Program The EQ Program had coordinated efforts with two USDA funded programs in the DR. 1. FAST Program implemented by USDA and the U.S Food and Drug Administration (FDA) and funded the US Agency for International Development (USAID) in the DR, EQ had collaborated with FAST on a few Food Safety Modernization Act (FSMA) readiness activities. 2. USDA FFPr SAFE/PROGRANA Program (September 2015- November 2021.) Implemented by NCBA/CLUSA, this program focuses on increasing agricultural productivity and trade in the beef and dairy sectors with the aim to facilitate beef exports from the DR to the U.S. While EQ’s value chains are different from SAFE/PROGANA, and PROGANA’s relevance to EQ largely indirect, cross￾27 USDA FFPr Exporting Quality Program Baseline Study, February 2016 11 learning about public and private sector capacity building, and the model of farmer field schools has been mutually beneficial for both programs. Exporting Quality (EQ) is a Food for Progress program in the Dominican Republic funded by the U.S. Department of Agriculture (USDA). Initiated in September 2015, this six-year, $16,861,02228 million program is implemented by International Executive Service Corps (IESC). This program aims to increase the productivity and sales in domestic and export markets of the following high-value fruit and vegetable global value chains: avocados, cocoa, pineapple, and greenhouse and Asian vegetables, and is designed to improve product quality, increase production efficiency, increase the value of post-harvest products, and enhance marketing and market linkages. EQ Program Activities. The main components of EQ’s assistance to the five value chains are through 1) Capacity Building: Producer Groups/Cooperatives; 2) Capacity Building: Trade Associations; 3) Cold Chain Improvement; 4) In-Kind Grants: Equipment; and 5) Public Information Campaign: Disperse Improved Market Information. EQ has over 60 interventions, across these five key activities which have been undertaken by the program to increase agricultural productivity and exports. Table 1 below outlines the key activities and associated interventions undertaken to date by the EQ Program. Table 4: USDA Exporting Quality Program Activities and Interventions (Source: EQ Semi-Annual Reports) Key Activity Sub-Activities/Interventions Undertaken from 2015 Number of beneficiaries Activity 1: Capacity Building: Producer Groups/Cooperatives Market-driven certification programs 14 Introduction of & Upgrades to traceability systems 101 Actions to increase compliance with sanitary and phytosanitary standards 2,323 Farmer Field Schools 854 Pedernales Avocado Initiative 197 Cocoa Tree Revitalization (Renovation) 119 Greenhouse Sector Diagnostic 117 Improve Producer Group Governance and Services 46 Climate-Smart Agriculture: Pilot Precision Agriculture Initiative 78 Establish And Engage Postharvest Handling Partnerships 252 Prevention of Microbiological Contamination Trainings 22 Pruning and Fertilization in Avocado 547 Training In the Cocoa Value Chain 1,526 28 The total budget including commodity and freight costs is $21,108,644. 12 Key Activity Sub-Activities/Interventions Undertaken from 2015 Number of beneficiaries Integrated Pest Management (IPM) Demonstration Farms 34 Farm management workshops 696 Rehabilitation of Orius production laboratory at the agricultural campus of the National Autonomous University (UASD) 1 Development of the Global G.A.P National Interpretation Guide and Strengthening of the Global G.A.P National Technical Working Group (NTWG) 38 TORO irrigation kit 9 Chlorinators 38 Provide Networking for Women in Trade Leadership 86 Activity 2: Capacity Building: Trade Associations Facilitate Participation in External Trade Shows: Chocoa, Fruit Logistica, PMA, Salon du Chocolat, among others. 52 Trainings: INCOTERMS, Entry Requirements for Fresh Produce to the EU Market; E￾Commerce Mentoring; Business Budget Workshop; Toward Quality Inspection Control in The Dominican Republic; Market Trends and Requirements for Dried Fruits; cocoa and chocolate quality evaluation 350 Improve Marketing and Branding 11 Young Exporter Network 122 Dominican National Chocolate Awards 12 Inward Trade Missions 78 Dominican Chocolate Festival 11 Manage and Build Buyer-Seller Relationship 41 Facilitate Participation in Internal Trade Shows: Agroalimentaria, HUB, etc. 62 Collaboration with ADOEXPO to develop the Tropical Avocado collective trademark Collaboration with 4 institutions Conduct End Market Analysis 401 Facilitate Cold Chain Knowledge-Sharing and Action Planning 338 Activity 3: Cold Chain Improvement Participate In Cold Chain Knowledge Sharing: Technical Assistance, GCCA Construction Board Technical Assistance; Study Tour to Mexico and Miami; Chicago 338 13 Key Activity Sub-Activities/Interventions Undertaken from 2015 Number of beneficiaries Global Cold Chain Expo 2018; GCCA Institute Mexico 2018 Participation in WFLO Latin America Congress 14 GCCA DR Cold Chain Assessment (2016) 18 Technical Assistance and Training Mission to exporters and to Dominican government inspections institutions 6 Organize Cold Chain Technology Demonstration 40 MOU to Strengthen the Dominican Cold Chain 1,071 Update WFLO Dominican Republic 2009 Cold Chain Analysis 4 Implementation of a Product Inspection Pilot that does not Break the cold chain at an airport in the Dominican Republic 124 Preparation and implementation of perishables manual communication plan TBD Processing equipment: Pineapple peeling, Pulping machine, Production line for avocado oil, One-shot depositor filler, Cauliflower press, Classification/packing line, Roaster and bottle filler for cocoa and cocoa-based products, Cocoa roaster, huller, and press; Cold press for pineapple, Cocoa fermentation facility, Tempering machine, Cocoa press and molds, Cocoa toaster, 22 Activity 4: In-Kind Grants: Equipment Cold Chain Equipment: Jet cooler, cool room equipment 1 Inspection cabins 14 Activity 5: Public Information Campaign: Disperse Improved Market Information Food Safety Campaign Aimed at Producers Food Safety Campaign Aimed at Consumers 5,728,481 EQ Program Activity Locations. The geographic focus for each product has been determined by production centers, aggregation sites (including processing and packing), and the important exporters 14 that exist in each province. EQ Program has worked in 25 provinces (as outlined in Figure 1 below) and has reached over 9,000 unique direct beneficiaries29 . Figure 7: Geographic Distribution of EQ Program Value Chains and Beneficiaries Target Population. Value chain actors through which production and exporting of these commodities was expected to improve include agricultural producers; agricultural producer organizations and cooperatives; processors, exporters, and packinghouses; trade associations; private sectors entities engaged in trade and related business; and government agencies. These are the groups, along with IESC, USDA, and Government of the Dominican Republic (GoDR) line agencies, from which the evaluation team will gather data for the evaluation. Key Year 6 Activities and Milestones. On September 25, 2020, IESC received approval from USDA to extend the program through November 30, 202130 and as such EQ is in its final year of implementation. The program was further extended until December 31, 2021. Key technical activity areas that EQ has been working to conclude prior to program closeout included 31: Activity 1: Capacity Building: Producer Groups/Cooperatives 1. Electronic Traceability Systems and Quick Response Module Technical Assistance 29 Per the final EQ beneficiary list provided by the EQ Program team. This number is different from the life of project direct beneficiary count found in Table 10 on page 27. the total of beneficiaries stated in Table 10 is the sum of beneficiaries registered in all the activities developed by the program (including people counted more than once, as allowed and according to the 2014 Food for Progress and McGovern-Dole Indicators and Definitions 30 USDA FFPr Exporting Quality Semi-Annual Report No. 11 31 USDA FFPr Exporting Quality Semi-Annual Report No. 12 15 2. Integrated Pest Management Support to Ministry of Agriculture (MinAg) technicians, academic institutions, and participating farm owners. 3. Moniliasis Pest Detection and Surveillance Training to MinAg technicians 4. Farmer Field School Sessions 5. Farm Management and Moniliasis Workshops with FUPAROCA 6. Pedernales Avocado Initiative: Crop Management and Producer Safety Training 7. Cocoa Tree Revitalization: Cocoa Plant Donations to Producers 8. Greenhouse Capacity Building workshops for DEPROBAP technicians 9. Produce Safety Training for Growers and MinAg technicians Activity 2: Capacity Building: Trade Associations 1. Dominican National Chocolate Competition and Festival 2. Chocolate Tasting Training by International Institute of Chocolate and Cacao Tasting 3. Chocolate Communications Campaign 4. Global Cocoa Market Study and Dominican Supply Case Study 5. Tropical Avocados Branding Logo Redesign Activity 3: Cold Chain Improvement 1. Design and development of a promotional campaign for MinAg’s perishable products handling manual (by EQ consultant) 2. Pilot Program of MinAg’s perishable products handling manual at Las Américas Airport (by another EQ consultant) 1.3 Results Framework Building on the successes and best practices IESC has developed in previous and current programs, the EQ program was designed to contribute to Food for Progress Strategic Objective (SO) 1 – Increase Agricultural Productivity and Strategic Objective (SO) 2 – Expanded Trade of Agricultural Products (Domestic, Regional, and International). Through activities designed to: 1) increase food safety and quality control, primarily focusing on organizational capacity within the value chains of avocados, cocoa, pineapple, greenhouse vegetables, and Asian vegetables; 2) improve the market driven incentive structure to encourage full participation; and, 3) provide technical training to ensure that beneficiaries have up to date knowledge and skills to meet market standards, EQ program helps to increase exports for the above mentioned high value horticulture commodities and contribute to the DR’s long-term competitiveness. The goal and other broad objectives of the EQ program, along with intermediate results that contribute to these higher-level results in the DR Food for Progress (FFPr) Results Frameworks #1 and #2, shown in Annex C of this evaluation. While the EQ program did not have standalone theory of change statement or diagram, five underlying assumptions that were seen as essential for the program’s success include: 1. The Dominican government will remain constant in or increase its support for private sector-led development, including continuous improvement of the government’s fiscal ability to adequately execute its regulatory and inspection functions. 2. Global markets will continue to be relatively stable and the global economy will steadily improve, particularly in the U.S. and the EU, the DR’s primary export markets. 16 3. The DR’s macro-economic climate will remain stable (inflation is currently in check, and electricity and gas costs will remain constant or improve). 4. There will be no major natural disasters. 5. Elections continue to be transparent, and transfers of power are clean and effective, despite some public dissatisfaction with the government32 . 1.4 Purpose of the Evaluation The purpose of the Exporting Quality Program Final Evaluation is to impartially assess whether the project achieved the expected results as outlined in the Results Framework in the Section 1.2 Program Background section below. This evaluation is conducted by the evaluation team comprising of the evaluation team lead, I4DI HQ team including the evaluation methods expert, I4DI evaluation manager, and the data collection partner firm. The scope of the evaluation comprises: • Exporting Quality (EQ) program design, implementation, management, and replicability; • Lessons learned/conclusions and recommendations for USDA, EQ participants, and other key stakeholders for future projects; • Follow-up on midterm evaluation questions33, including assessing direct and indirect, intended and unintended, positive or negative impacts; and, • The extent to which EQ addressed midterm evaluation recommendations and incorporated them into the program’s subsequent activities and operations The evaluation examined both administrative and programmatic aspects of EQ related to data capture, measurement, and intervention impact. The analysis of progress against indicator targets will define areas of shortfalls and areas of success that may highlight opportunities for IESC and USDA to design future successful interventions leading to greater impact. This analysis has been included in Section 3.1 of this evaluation report. The evaluation will also capture stories of “significant change” from the field as described by program beneficiaries to communicate success due to program interventions and to develop case studies for additional, future learning. This final evaluation was also informed by the EQ baseline and mid-term evaluations conducted in 2016 and 2018 respectively. The key audiences for the EQ Final Evaluation include the following: • IESC Exporting Quality program staff, including the Exporting Quality Steering Committee comprised of key U.S. government and Dominican government stakeholders, USDA staff, and other USDA implementing partners • EQ’s partners Global Cold Chain Alliance-World Food Logistics Organization (GCCA-WFLO) and Florida Agricultural and Mechanical University (FAMU), local partner Centro para el Desarrollo Agropecuario y Forestal (CEDAF) • Exporting Quality participants and beneficiaries, and trade associations and agencies within the government of the Dominican Republic. • USDA implementing partners, the development community, and the general public 32 Exporting Quality proposal document 33 EQ Program Baseline study was conducted in 2016, and midterm evaluation in 2018- both of which were led by Antonio Cordella 17 The findings of this final evaluation will aid USDA in the establishment of new programming that are reflective of the agricultural context in the DR and lay the framework for future USDA performance monitoring for similar active projects such as USDA FFPr Trade Safe Project, USDA FFPr PROGANA Project, and future projects in the DR and the region. Findings will also contribute towards FFPr Learning Agenda priority questions in the following focus areas: 1) Market Systems, 2) Quality of Standards, and 3) Risk and Uncertainty, as well as cross-cutting areas namely culture, demographics, and sustainability. 1.5 Evaluation Questions This mixed methods evaluation focused on three set of questions raised in the evaluation Statement of Work namely: • How well the EQ program performed against targets established for the program’s official performance indicators detailed in Section 3.1: Performance against EQ Program Performance Indicators of this report • How well EQ activities (and their interventions) performed, i.e., what portion of the target population participated in those activities and what the effects of each activity were, as detailed in Sections 3.2-3.4 of this report • EQ program’s performance around an internationally recognized set of evaluation scope topics identified for attention in the evaluation’s Statement of Work34, detailed in Section 3.5: Evaluation Questions linked to OECD/DAC Criteria of this report The full list of evaluation questions is included in Annex B of this evaluation report. The three sets of questions described above reflect different perspective from which readers may wishto understand the EQ program. Thus, the status of performance indicators called for by the first question set may be of particular interest from an accountability for results perspective. The second question set, in contract is more focused on the interventions or activities the EQ program used to produce results supportive of FFPr Strategic Objectives 1 and 2 and the industry groups – producers, processors, exporters and other trade facilitators – it sought to engage in these activities. Questions in the final are less specific to the EQ program but help capture information of interest to program managers regardless of the sectoral focus of the program examined. While useful, there are natural overlaps between these perspectives and thus some degree of repetition across sections of this report that describe evaluation findings from each of these perspectives. I4DI has minimized redundancy across the study’s findings, conclusions, and recommendations by consolidating complementary or similar questions to the degree possible. 1.6 Data Sources I4DI started by conducting extensive document review of all relevant EQ Program documents which helped inform the design of the qualitative and quantitative instruments. The documents reviewed included 12 EQ semiannual performance reports, EQ Performance Monitoring Plan (PMP), EQ Program baseline study, and EQ Program mid-term evaluation. Then, I4DI collected quantitative data through a survey of farmers, and employees of firms including exporters, processors, packinghouses, private enterprises, and government. All quantitative surveys 34 See Annex G: Evaluation TOR for more details. 18 were complemented by qualitative key informant interviews with representatives from the above stakeholder groups as well as mini-surveys, with senior leadership in firms, that were semi-structured with both open-ended and close-ended questions. Qualitative data helped contextualize findings from the surveys and provide an overall insight into the agricultural sector at various levels of the value chain. Quantitative and qualitative data collection instruments were developed in English and then translated into Spanish after approval from the EQ Program. Section 2: Evaluation Design and Methodology I4DI led a mixed method evaluation of the EQ Program by systematically integrating data collected though qualitative and quantitative methods and triangulating findings with document review of EQ Program materials. Section 2.4 provides more details. This final evaluation was conducted in compliance with USDA’s Monitoring and Evaluation (M&E) Policy, Food for Progress and McGovern Dole Indicators and Definitions35, and the approved Exporting Quality Program’s Monitoring and Evaluation Plan36 . I4DI took a participatory approach to the evaluation. The evaluation team, comprised of the evaluation team lead, I4DI HQ team, and DR-based data collection partner actively engaged key program stakeholders and principal evaluation users throughout the evaluation, including the evaluation design process. This participatory approach processes contributed to a more robust understanding of program achievements and lessons learned. Impartiality and lack of bias have been safeguarded by the evaluation methodology which relied on several complementary information sources. In selecting methods and designing evaluation instruments and samples, the evaluation team used a social inclusion lens and paid attention to various forms of diversity, ensuring representation of diverse beneficiary groups where possible. 2.1 Sampling Methods Quantitative Data Collection (Surveys) The strategy for sampling the different beneficiaries of the EQ Program is described below. Samples were drawn from final beneficiary lists provided by IESC on August 13, 2021, which contained required data for beneficiary groups, namely agricultural producers and people in firms, i.e. exporters, packinghouses, processors, and people in government. The sampling approach used also takes differences in the size of value chain, i.e. number of participants in each EQ value chain and producer and processor sub-groups into account. 1. Agricultural Producers. I4DI used the following calculation and approach for sampling agricultural producer beneficiaries enumerated by value chain per the final beneficiary list provided by IESC. 35 Dated 7/14/2015 36 Approved 3/29/2016; revised and submitted on 1/22/2021 19 Given that the number of beneficiaries in each stratum is quite different, a proportional stratified random sampling approach was adopted for this beneficiary group. This helped ensure that strata with small numbers of beneficiaries are as well represented as possible across a common sampling approach for the whole population. Table 5: Sampling Distribution for EQ Producer Beneficiary Group, Disaggregated by Value Chain Province Avocado Producer Total Avocado Producer Sample Cocoa Producer Total Cocoa Producer Sample Pineapple Producer Total Pineapple Producer Sample Greenhouse Vegetable Producer Total Greenhouse Vegetable Producer Sample Asian Vegetable Producer Total Asian Vegetable Producer Sample Total Sample Azua 129 10 0 0 0 0 0 0 0 0 10 Duarte 0 0 602 62 0 0 0 0 0 0 62 Elias Pina 117 7 0 1 0 0 2 0 0 0 8 Hato Mayor 0 0 117 0 0 0 35 11 0 0 11 La Vega 1 0 21 3 0 0 98 12 295 31 46 Monsenor Nouel 0 0 250 29 0 0 0 1 0 0 30 Monte Plata 0 1 254 32 245 18 13 1 28 0 52 San Cristobal 199 22 156 18 0 1 1 0 0 0 41 San Jose de Ocoa 166 19 0 0 0 0 14 0 0 0 19 San Juan 286 32 0 0 0 0 35 0 0 0 32 Sanchez Ramirez 0 0 0 10 153 11 0 0 0 0 21 Total37 898 91 1,564 155 398 30 198 25 323 31 332 37 While the total number of usable producer records in the EQ beneficiary list is 4,645 (encompassing all EQ provinces), this table shows total producers in the 11 provinces. 20 As shown in Table 5 a total of 332 producers were surveyed from the 11 provinces selected for the sample of whom 91 were avocado producers, 155 were cocoa producers, 30 were pineapple producers, 25 were greenhouse vegetables producers, and 31 are Asian vegetables producers. These sample size varies slightly at the value chain level, from those proposed in the work plan, but total sample size for producer survey proposed in the work plan (n=330) has been exceeded and the distribution by value chain still provides a proportionate sample. See Annex E-1: Approved EQ Final Evaluation Work Plan for details. Annex E-2 provides more details regarding sample selection and response rates for the farmer survey. 2. People in Firms. Individuals working in firms is the second largest beneficiary group, with an estimated 68438 unique beneficiaries, as shown in the beneficiary list provided by IESC. The distribution of people in firms in the EQ beneficiary list, and the sample size for this evaluation (at a 95 percent confidence level and 8 percent error) is shown in Table 6 below. Table 6: Sampling Distribution for People in Firms (based on EQ Beneficiary List) People in Firms EQ Beneficiary Population (By firm type) # of People in Firms Sampled (By firm type) Confidence Level/Error Exporters 253 18 - Packinghouses 249 42 - Processors 71 32 - Government 111 32 - Total 684 124 95%/8% The work plan/design stage sampling plan we had proposed set out to survey 171 people in firms, which corresponds to 57 exporters, 54 packinghouses, 41 processors. This was however not possible due to the large number of hard-to-reach respondents despite multiple attempts made at different times of the day. This has been further detailed in the Evaluation Limitations section. In the interest of not delaying data analysis and reporting for the evaluation, the evaluation team deemed that a 72 percent return rate (i.e. 124 phone surveys out of the originally planned 171) is a respectable rate of return, and captures relevant insight with representation from all firm types. Annex E-2 provides more details regarding the sample and response rates for the firm survey. 3. People in Firms (Leadership). In order to get a different perspective on EQ, its activities and impact from the people in firms stakeholder group, the evaluation team additionally administered online mini-surveys to senior and executive management of firms and private enterprise namely exporters, packinghouses, processors; government; as well as lead representatives of trade associations based on the organizational contact list provided by EQ Program. The online modality for data collection for this particular subset of people in firms seemed appropriate and valid for the following reasons: • Over 85 percent of the contacts on the executives/senior leadership of firms list had email addresses. • Provides a cost-effective and fast way to reach a wider audience and receive critical information from a sub-group of stakeholders that have easy access to technology 38 684 is the number of unique people in firms beneficiaries after senior leaders/executives were removed from the beneficiary list. Executives/senior leadership in people in firms were administered the online mini-survey and key informant interviews. 21 • Responses from mini-surveys provides meaningful insight that can inform key informant interviews (KIIs) which were conducted with a subset of these mini-survey respondents This online mini-survey was administered to a census of 100 contacts with email addresses to maximize opportunities to higher response rates. Additionally, I4DI and Rapid Asia enumerators conducted follow￾up nudge emails and calls to non-respondents to ensure higher response rates. Draft mini-survey instruments for senior management and leadership from people in firms and representatives of trade associations can be found in Annex E: EQ final evaluation work plan and Collection Instruments. Table 7 below provides a summary of the quantitative data sources for this evaluation. Table 7: Summary of Quantitative Data Sources Stakeholder Type Data Collection Method Target Sample Size Actual Sample Size Return Rate Agricultural Producers In-Person Survey 330 332 100% People in Firms Phone Survey 171 124 72% People in Firms (Leadership Positions only) Online Mini- Survey Census39 (100) 19 2040% Qualitative Data Collection. EQ Program provided a separate list of contact information for executives and senior leadership at firms, with representatives from government, exporters, processors, packinghouses, and private enterprises, and NGOs who were recipients of EQ training and interventions. Table 8 below breaks down the KII distribution by firm type and provides a comparison of KII respondents’ target vs. actual size, based on their overall firm proportions. Specific names and contact information of representatives of stakeholder groups 1 through 4 in this table, were provided by the EQ program team, while the names of key informant interview (KII) participants from groups 5 through 8 were randomly drawn using a randomizer tool based on their respective proportions. A small proportion of senior leaders in firms declined to be interviewed, while others did not respond at all, despite multiple attempts to contact them. This caused for a slight discrepancy between the target and actual sample sizes. This has been further detailed in the Evaluation Limitations section. The final sample of KII respondents do however represent each type of firm and as such, their responses have been analyzed to supplement the people in firms survey. 39 The online mini-survey was developed to be administered online amongst leadership in firms and was sent to leaders/executives on the beneficiary list who had valid email addresses, i.e. 100. 40 For online surveys that only had cold-call type of notifications and reminders, and no incentives offered to respondents for completing the survey, a 20 percent response rate is considered respectable, especially given these were senior leaders in firms with busy schedules. On other evaluations and assessments, where online surveys (with no incentives provided for completion) typical response rates have ranged from 5 to 15 percent. Due to the need to wrap up all data collection on time, and because finite resources available in terms of enumerator level of effort, the decision was made to focus remaining efforts towards increasing people in firms survey responses. 22 Table 8: Summary of Qualitative Data Collection Sources and Sampling Distribution for KIIs Group # KII Stakeholder Groups # of Senior Leadership in Firms Beneficiaries for EQ41 Proportion of Total Beneficiaries Target Sample Size Actual Sample Size Data Collection Modality 1 USDA Washington N/A N/A 2 2 Virtual 2 USDA Santo Domingo N/A N/A 2 2 In-Person 3 IESC Washington N/A N/A 3 3 Virtual 4 IESC Santo Domingo N/A N/A 6 6 In-Person 5 Senior Leadership: Government of DR 26 17% 13 10 In￾Person/Virtual 6 Senior Leadership: Trade Associations 18 12% 4 7 Phone 7 Senior Leadership: Exporters 50 32% 12 6 Phone 8 Senior Leadership: Packinghouses, Processors, Private Enterprise 57 37% 14 10 In￾Person/Virtual 9 Senior Leadership: Non-Governmental Organizations42 4 3% 0 0 N/A Total 155 - 56 46 - 2.2 Data Collection Methods The main data collection methods and design constructs the Evaluation Team used and lists the questions and indicators to which each have been applied is shown in Table 9 below. Data collection instruments developed for this final evaluation have been included as Annex E: Approved EQ Final Evaluation Work Plan and Instruments. Table 9: Main Methods for Data Collection Across the Three Main Groups of Questions Data Collection Method Evaluation Question Group A: Results & Activity Indicators (Data Source: Result and Activity Indicators per Midterm Evaluation & EQ PMP) Evaluation Question Group B: Activity Specific Questions (Data Source: Activity Questions included in RFP Table 2 on an illustrative basis) Evaluation Question Group C: OECD/DAC (OECD/DAC criteria questions including “other” questions from RFP) Desk Review All result and activity indicators, including All questions, including questions asked in the All questions, using word searches of documents for 41 The list of senior leadership/executives from people in firms who were EQ beneficiaries, was provided by the EQ Program. 42 Given the relatively small number of NGOs within the senior leadership/executives list (i.e. representing 3% of the total list), the evaluation team decided not to conduct any interviews with NGO leadership. 23 Data Collection Method Evaluation Question Group A: Results & Activity Indicators (Data Source: Result and Activity Indicators per Midterm Evaluation & EQ PMP) Evaluation Question Group B: Activity Specific Questions (Data Source: Activity Questions included in RFP Table 2 on an illustrative basis) Evaluation Question Group C: OECD/DAC (OECD/DAC criteria questions including “other” questions from RFP) midterm and PMP values reported to which final values were compared midterm about activities underway before the EQ baseline or which were asked about on an EQ activity basis in the midterm evaluation key terms: relevance, effectiveness, efficiency, sustainability, impact Quantitative Data Collection Instrument 1: Agricultural Producers Survey All indicators in this cluster, to the degree the midterm evaluation used its producer’s survey to gather that data. The exceptions here were a small number of indicators that focused on specific value chains, i.e., avocados, cocoa. This method produced the largest volume of data, but at the analysis stage each of these methods were treated as a distinct “line of evidence” and the findings from various “lines of evidence” were compared to determine the degree to which answers from various methods converge or diverge. This helped ensure that volume of data alone does not dictate the evaluation’s findings for any group of evaluation questions.43 All agricultural survey user respondents were asked which EQ activities they participated in and then asked those Group B questions in the survey about each activity in which they participated, e.g., those that attended farmer field school were asked survey questions related to Group B Evaluation Question A.1.4. The survey instrument used skip techniques to direct each survey respondent to answer only those Group B questions about which they have first-hand knowledge Selected questions based on the OECD criteria list are being framed for inclusion in this survey. The most prominent among these are Group C sub￾questions from the RFP about effectiveness and efficiency, and at least one question each about relevance and sustainability were included in the instrument and pre-tested to determine how accurately they are understood and addressed by farmer respondents. Data Collection Instrument 2: People in Firms Survey Same as above with the inclusion of questions related to EQ Activity 2: Capacity Building to Trade Associations; and Same as above with the inclusion of questions related to EQ Activity 2: Capacity Building to Trade Same as above 43 For these approaches see the description provided from Rudd, et al., in USAID’s volume on mixed methods evaluation. 24 Data Collection Method Evaluation Question Group A: Results & Activity Indicators (Data Source: Result and Activity Indicators per Midterm Evaluation & EQ PMP) Evaluation Question Group B: Activity Specific Questions (Data Source: Activity Questions included in RFP Table 2 on an illustrative basis) Evaluation Question Group C: OECD/DAC (OECD/DAC criteria questions including “other” questions from RFP) Activity 3: Improve Cold Chain Associations; and Activity 3: Improve Cold Chain Data Collection Instrument 3: Mini Surveys Filtered from the people in firms survey Instrument or otherwise adapted based on the relevance and topical expertise of the groups surveyed. These mini surveys were disseminated (via email) to senior and executive leadership of private enterprise, exporters, processors, packinghouses, and trade associations. I4DI captured as much of the same information from people in firms survey as will be relevant for findings about these groups or comparisons between them and producer survey respondents. As with the people in firms survey the mini survey’s had respondents identify the Group B activities in which they were involved and then answer questions about those experiences written from the perspective of their type of organization. Same as above Qualitative Data Collection Instrument(s): Key Informant Interviews: 1) USDA; 2) IESC; 3) Government; 4) Exporters, Packinghouses, Processors, Private Enterprise, Filtered for inclusion on a respondent-by￾respondent basis from the midterm evaluation list of result and activity indicators based on key informant’s likely expertise with indicators. Key informants were asked as many Group B questions as the evaluation team determined they were likely to have expert or first-hand knowledge on which to base their answers. Questions in Group C were included in the KII instruments framed from a manager or observer perspective and with an eye to evidence utilization for future programming rather than from a first￾hand perspective. All the questions in Group C were covered with this tool which may produce most of the data on these questions. 25 Data Collection Method Evaluation Question Group A: Results & Activity Indicators (Data Source: Result and Activity Indicators per Midterm Evaluation & EQ PMP) Evaluation Question Group B: Activity Specific Questions (Data Source: Activity Questions included in RFP Table 2 on an illustrative basis) Evaluation Question Group C: OECD/DAC (OECD/DAC criteria questions including “other” questions from RFP) Trade Associations44 Within this data collection structure, the evaluation team embedded techniques for eliciting specific types of data needed to address evaluation questions: - Most Significant Change (or intervention): This is an approach for eliciting beneficiary perceptions about the most important effects or change in their lives brought about by a program, or a single intervention. Alternatively, this technique can be used to obtain beneficiary perceptions about the relative importance of different interventions in bringing about a specific result or change. Questions embedded in the study instruments aimed to elicit both of these types of information. - Contribution Analysis is used in evaluations to try to understand what interventions or factors in an environment played an important role in bringing about (causing) a specific result. It is frequently used in evaluations of multi-donor programs to determine whether and to what degree several different interventions (e.g., a government program, a donor program, NGO assistance or citizen action) seem to affect a situation or problem on which they were all focused. In this evaluation, Contribution Analysis constructs were used to embed questions in study instruments to elicit from program beneficiary information about any assistance they may have received from sources other than the EQ program. Such questions were included in both the farm and firm surveys. 2.3 Field Work Quantitative Data Collection: Farm and Firm Surveys Rapid Asia mobilized 20 local enumerators and 5 supervisors to conduct the in-person farmer survey, and a team of 5 interviewers, and 1 supervisor to conduct the phone-based firm survey in the DR from September 22 to 30, 2021. All surveys were administered using computer-assisted personal interviewing (CAPI) called Survey Solutions. All interviews with farmers were administered at the farm level in order to facilitate direct observation and obtain farm coordinates (latitude/longitude) for future reference and analysis. Surveys with people in firms were conducted via phone using the contact information provided in EQ’s beneficiary list. Electronic tablets were used to administer and record survey results, allowing for automated tabulation of raw data using CAPI (Survey Solutions), reduction in data entry error and information processing time. This has proven to be effective in terms of quality assurance for tabulation and analysis of data. 44 KIIs were used to interview specific individuals from USDA, IESC, DR Government organizations and other expert on topics where this is the best method. 26 Rapid Asia trained the enumerators on the data collection instruments, with ten enumerators reporting to one supervisor. The training was done online, and the interviewers were at their office in Santo Domingo. Training modules focused on topics including methodological guidelines, data entry into the tablet-based survey, capturing farm coordinates. In addition, a quick instrument pre-testing and piloting was conducted in advance of actual data collection to help familiarize enumerators with the data collection process and to minimize potential errors. Trained enumerators were additionally be guided through their first few interviews by the field supervisor, to ensure they understood the entire process and could collect accurate data from beneficiaries. Enumerators were also advised to take high quality photographs during data collection, given this was an IESC requirement for the draft and final evaluation report. Consent was explicitly sought from producers prior to taking photographs of their farm operations. The Rapid Asia team coordinated and supervised field-based interviews and an independent audit team in Santo Domingo, Dominican Republic carried out a quality control inspection of at least 10% of the total of the surveys as is standard practice at Rapid Asia. This spot check, carried out in person via observation or over the phone, ensured reliability of the data collection process. Via the CAPI system, Rapid Asia also had a direct communication channel with the enumerators to clarify doubts, provide instructions and verify the progress of the survey in targeted provinces. All throughout this field work, the data collection team was made to comply with the Safety protocol under COVID-19 pandemic condition, presented in below in Figure 9. Farm Survey Respondent Population Characteristics Distribution of Farms, by Province and Value Chain. As seen in Figure 8, farmers in the sample represent all five EQ value chains and are spread across 11 provinces in the country. The color gradation on the map depicts the relative number of farmers surveyed in a specific province. The darker shaded green depicts provinces with more sampled farmers, when compared to the lighter shaded green provinces. Of the 332 farmers surveyed, around 48 percent were cocoa producers; 27 percent were avocado producers, 12 percent were greenhouse vegetable farmers; 9 percent pineapple farmers, and around 3 percent were Asian vegetable farmers. Of these 332 farmers, only 11 percent (37) were female. The largest proportion of these farmers were in the 41-60 age group, (i.e. 46 percent), followed by farmers aged 61-80 (32 percent). Most farmers in the sample began working with EQ in 2018 (29 percent; 96 farmers) and 2019, all other years engaged many fewer farmers with the exception of Year 1 of the program, followed by 75 farmers (23 percent) in 2019. A significant portion (21 percent; 69 farmers) of the sampled farmers have been program beneficiaries since 2015. Farmer dataset, that detail farmer demographic data is presented as Annex D: Datasets of this evaluation. 27 Figure 8: Farm Sample Distribution in the DR, by Value Chain Farm Size. Of the 332 farms visited and farmers surveyed, around 84 percent (278) of farms that grew their main EQ value chain crop were larger than 10 hectares (HA), followed by 16 percent of farms sizes ranging from 2-10 HA. Only one farm in the sample was one HA in size. Table 10 below provides the average farm size (in HA) by value chain of sampled farms. Table 10: Average Farm Size of Sampled Farms Value Chain Average Farm Size (in hectares) Avocado 2.7 HA Cocoa 8.9 HA Pineapple 31.5 HA Greenhouse Veg 2.6 HA Asian Veg 3.7HA45 See Figure 9 below which details the geographic distribution of farmers surveyed as part of this final evaluation, and their farm size (in HA). 45 One farm with size > 100,000 HA has been disregarded as that skews the overall overage farm size. 28 Figure 9: Farm Sample Farm Size (in Hectares) Farmer Household Size. As Figure 10 indicates, more than half of the farmers surveyed (177; 53 percent) had a household of 3-6 members. This information is useful since farmer household members represent indirect beneficiaries of the EQ program per result indicator 2. Figure 10: Farmer Sample, Household Size Firm Survey Respondent Population Characteristics Distribution of People in Firms, by Firm Type and Value Chain. Of the 124 people in firms surveyed, around 34 percent were employees of packinghouses; 26 percent were processors, 26 percent were government representatives; and 14 percent were exporters. 39 (31 percent) out of the 124 people surveyed were female. The largest proportion of these people in firms were in the 25-50 age group, (i.e. 68 percent), followed by individuals older than 50 (27 percent). A quarter of the people in firms in the sample began their work with EQ in 2019, and around 18 percent in 2018 and 2020. All other years 38% 53% 8% 0% 20% 40% 60% 1--3 3--6 7--10 Household Size Farmer Sample (n=332): Household Size 29 engaged many fewer people in firms. Most firm beneficiaries became associated with the program between Years 2 and 4 of EQ. Most of the first year for a USDA project is spent on planning for monetization, project staff recruit and onboarding, the initial required deliverables, and stakeholder outreach and relationship building. During Years 2 to 4, the program developed its expertise in FSMA trainings in coordination with USDA Foreign Agricultural Service (FAS) in the DR, with training outreach and delivery during this time being broad and wide ranging. Later, in Year 5, the program focused on providing more targeted support to firms that made the required cost-share commitments to focus on deepening the sustainability of this intensive intervention. The more focused training meant that the EQ Program was not adding new firm beneficiaries at the same rate as in previous years. During Years 5 to 6, the COVID-19 pandemic presented an additional obstacle to delivering larger group outreach and orientation trainings; therefore, the project focused on maintaining the existing targeted group of beneficiaries in one-on-one, virtual and onsite trainings. Finally, the program’s Year 6 represented a limited, costed extension to the program wherein the interventions delivered by IESC specific to a smaller pool of expanded and targeted beneficiaries, mostly replicating one-on-one, onsite intensive coaching at firms. See Figure 11 below for more details. Figure 11: Year in which Firms First Started Getting Support from the EQ Program (Represented as % of firms) Around 90 percent (112) of the people in firms sample worked in firms that had between 0-25 employees. Only one firm had over 100 employees. People in firms dataset, that details demographic data is presented as Annex D: Datasets of this evaluation report. Figure 12 below shows a distribution of these people in firms by their respective value chains. 15% 0.80% 5% 18% 25% 18% 15% 5% 0% 5% 10% 15% 20% 25% 30% 2015 2016 2017 2018 2019 2020 2021 Don't know % of Firms Receiving EQ Support Year that Firms Started Getting EQ Program Support (n=124) 30 Figure 12: People in Firms Survey Sample, by Value Chain (in %) Online Mini- Survey Respondent Population Characteristics The evaluation team achieved a 20 percent response rate on this online survey targeted at leaders in firms. Around 26 percent of firm leader respondents were female, and the remaining 74 percent were male. A breakdown of the respondent profile is outlined in Figures 13, and 14 below. Figure 13: Online Mini Survey Respondents (Firm Leaders), by Firm Type 15% 16% 2% 32% 35% People in Firms Sample, by Value Chain Avocado Cocoa Pineapple Greenhouse Vegetables Asian Vegetables 37% 16% 11% 16% 5% 11% 5% 0% 5% 10% 15% 20% 25% 30% 35% 40% Exporter Processor Packinghouse Trade Association Private Enterprise Non-governmental organization (NGO) Other Online Mini Survey Respondents, by Firm Type 31 Figure 14: Online Mini Survey Respondents (Firm Leaders), by Value Chain (in %) Qualitative Data Collection: Key Informant Interviews (KIIs) with Stakeholders The Evaluation Lead was in the DR in September 2021 to participate directly in the interviews of the key government functionaries, firms, EQ program Team in the DR, as well as USDA/Santo Domingo. The Evaluation Lead also interfaced extensively with EQ team in the DR, and members of the various supply chains, to gain a complete overview of the program, its opportunities, and challenges. Rapid Asia appointed local moderators conducted around 13 key informant interviews to support qualitative data collection work of the Evaluation Lead. General COVID-19 Protocol. Rapid Asia and its local partners and sub-contractors adhered to World Health Organization (WHO) guidelines for protecting themselves and others from the spread of COVID￾19. The following precautions were implemented as standard during fieldwork: • When possible, conducting interviews over the phone or online. In this case, producer surveys were done face to face and all other interviews were conducted via phone. • Follow any government-imposed restrictions • Wear masks at all times • Wash hands or using hand sanitizer before each interview • Adhere to social distancing • Avoid interviewing anyone at high risk of having contracted the COVID- 19 virus and • Confirm that the respondent is not immune compromised before conducting an interview 11% 16% 47% 16% 11% Online Mini Survey Respondents, By Value Chain Avocados Pineapples Cocoa Greenhouse vegetables Asian Vegetables 32 Fieldwork Health Kit. Field personnel were provided with the following: Masks, sanitizer, wet tissues, and face shield. Documentation (if required). Health certificate, letter of endorsement or permit (if necessary). This applies to all Field Interviewers/Group Leaders/Team Leaders Survey Screening Questions. Rapid Asia field team included screening questions to ensure the safety of the interviewers and respondents. Respondents were screened out if: • They travelled outside of the country or to areas listed under the Department of Health (DOH) in the past 3-4 weeks. • Any family members travelled outside of the country or to areas listed under the Department of Health (DOH) in the past 3-4 weeks. • Any household member is experiencing fever, dry cough, difficulty breathing, shortness of breath, or fatigue. • Any household member has been in contact with a confirmed or suspected COVID-19 case. 2.4 Data Analysis Methods Both quantitative and qualitative methods of data collection were employed in collecting data for this Exporting Quality program final evaluation. Quantitative Data Analysis. For two of the quantitative datasets, namely farmer survey, people in firms survey, the evaluation team used STATA, while the online mini survey dataset was analyzed using SPSS. All the quantitative data collected (either SPSS and STATA) were cleaned and generated into frequency tables and analyzed. Using frequency tables allowed the evaluation to record the number of observations falling in each variable via percentages, analyze categorical data, and screen that data for data entry errors. For all the quantitative data (either in SPSS or STATA) cross tabulations were generated to analyze the relationship between two different variables, establish trends, if any, and make associated recommendations. This approach also enabled the evaluator to group variables together and to understand the correlation between the different variables, thus showing how correlations change from one group of variables to another. It also allowed for the identification of patterns, trends, and probabilities within data sets. Figure 15: COVID-19 Pamphlet Developed by Rapid Asia for Enumerators 33 Qualitative Data Analysis. Content analysis was used in analyzing the qualitative key informant interview data, and open-ended questions for surveys together with the transcriptions. The transcripts received were read several times and cross-checked with the questions to identify, code, and categorize primary patterns to establish the key emergent themes across interview guides. The key emergent themes helped establish most significant change (MSC) and most significant effect pathways which underlie the evaluation teams conclusions and recommendations. Triangulation and Integration of Qualitative and Quantitative Data. The evaluation team’s s mixed method design created multiple lines of evidence about the EQ program which the team then used to triangulate the quantitative and qualitative analyses alongside EQ Program documents such as semi￾annual reports, to address the evaluation questions. Data was first analyzed separately, as described above, and then analyzed against each other to identify whether and how the three sets of findings (qualitative, quantitative, and EQ program documents) coincide in their answers to the evaluation questions. Data were first analyzed separately according to the way they were collected, as described above, and then data from different lines of evidence analyzed in comparison to each other to identify whether and how the three sets of findings (qualitative, quantitative, and EQ program documents) converged or diverged with respect to their answers to the evaluation questions. See Figure 16. Through a workshop process during the analysis stage, evaluation team members brough to bear evidence from these lines of evidence on each question, in each of the three question sets. When answers from various lines of evidence converged, those answers were treated as being more strongly supported than answers that came from only one line of evidence or answers from various lines of evidence that differed. By and large, in this evaluation, convergence among answers from the lines of evidence was seen. As indicated in the data collection discussion above, I4DI also drew on contribution analysis techniques to obtain and analyze data on the relative importance of individual EQ interventions in bringing about specific results, and to obtain and use data from EQ beneficiaries about other sources of assistance they may have received. The inter-EQ data gathered was used in answering evaluation questions about which activities were most effective or had the greatest impact. The data on other sources of assistance, on the other hand were used in reporting on the impact question in the third question set the evaluation addressed. However, as mentioned in the Evaluation Limitations below, one of the limitations of this study is the concurrent collection of quantitative and qualitative data which may have limited the extent to which the quantitative findings can be contextualized. Figure 16: Data Triangulation and Integration Approach Followed by the Evaluation Team 34 2.5 Evaluation Limitations Study limitations are identified in evaluation reports to transparently alert readers to issues that may have arisen that would compromise reader’s ability to put confidence in the findings, conclusions, and recommendations from an evaluation. Several issues arose during this evaluation which required attention. None, however, were deemed to be major limitations to the validity and reliability of the study’s findings. The few limitations the evaluation team did encounter detailed below: • Self-Reporting Problem. In evaluations that include questions about beneficiary improvements, evaluators obtain data to answer such questions from independent, objective sources, or by using stable measurement devices (such as scales, indices etc.). They may also obtain self￾reported data from beneficiaries but try not to have that be the only source. For some types of information, however, beneficiaries may be the only available data source. In this evaluation, as in many other evaluations, answers to some questions posed to beneficiaries about changes in their practices, adoption of a technology, or improved business outcomes may not be supported by a second source. Given the frequency with which this situation arises, evaluators are generally vigilant in their efforts to spot outlier responses and check their plausibility where possible. In this evaluation there were no reports from data collectors of outlier responses that brought the validity of answers they received into question. • Lack of Disaggregated Program Data. The EQ program is governed by the 2014 Food for Progress and McGovern-Dole Indicators and Definitions, some of which did not require value chain-wise disaggregation at the producer or processor level as stated in the EQ PMP. As such value-chain wise disaggregation for activity and results indicators was not collected at an activity-level during program implementation and across prior evaluations. This posed a limitation for this evaluation in limiting the amount of comparison of progress (at a value chain level) across different points in time, for key measures of productivity, yield, sales value, and sales volume. The evaluation team looked to secondary data from sources such as FAOstat to be able to draw some of the requisite comparisons for value-chain disaggregated data collected for this final evaluation. • Respondents Not Willing to Disclose Sales Data. Given the sensitivities around revealing sales data, both farm survey respondents and employees of firms frequently declined to answer this type of question, resulting in fewer responses for those items, which is evident in survey tallies included in this report. Among surveyed farmers, non-response for sales value data was noted for some avocados producers (29 percent of the total 91 surveyed avocado producers), greenhouse vegetables (20 percent), and Asian vegetables producers (13 percent) while sales volume data was not disclosed among some producers of avocados (26 percent), cocoa (2 percent), greenhouse vegetables (52 percent), and Asian vegetables (13 percent.) • Hard-to-Reach Respondents. At the start of the survey of firm workers the evaluation team encountered difficulties connecting by phone with members of the survey sample, many of whom may have worked on jobs that kept them on the floor of their businesses and away from phones. The evaluation team was able to work through this issue and reach enough sample group members to sustain its target confidence level for this survey. • Duplicate Interviews. As noted above, as a function of parallel yet somewhat distinct key informant interviewing and survey administration under this evaluation, a few individuals in the leadership of firms in the value chains on which the EQ focused may have provided data twice. 35 However, the questions differed in these two collection methods with one seeking answers in forms that can be quantified while the other sought descriptive information. This limited overlapping of sources and methods is not deemed a threat to study validity or reliability by the evaluation team. • Data Collection Sequencing. It can be desirable to develop and deploy evaluation research data collection instruments in a sequence so that findings from one type of data collection process informs the next. Options in this regard were discussed, but in the end the logistics of the study precluded their structured use. Nevertheless, the evaluation team did collaboratively develop the three survey instruments. It also collaboratively analyzed data from the study’s multiple data sources to arrive at the study’s findings, conclusions, and recommendations for each of the three sets of questions the evaluation addressed. • Recall Bias. Since several questions asked on surveys and in key informant interviews deal with issues that took place in the past, recall bias cannot be excluded. The specific concern for this evaluation stems from the fact that record keeping is not a common practice among farmers, especially those that do not apply improved practices. Generally speaking, this problem tips more towards fine grain information, e.g., metric tons of a specific crop harvested a few of years ago, than for major facts, e.g., did you use a biofertilizer last year. No specific instances of recall bias were brought to the attention of the evaluation team for this study by its survey research partner in the Dominican Republic. A related but distinct issue emerges when respondents blur the tense in which they are speaking and say “Yes, I intend to do that,” when what they mean is that they intend to do something in the future that they are not yet doing. This type of blurring may occur with questions about the adoption of new practices. To minimize this problem the evaluation team tried to word questions to minimize the opportunities for tense blurring, particularly in relation to the practices recommended in EQ activities. • Halo Bias. There is also a known tendency among respondents to under-report socially undesirable answers and alter their responses to approximate what they perceive as the social norm (halo bias). The extent to which respondents will be prepared to reveal their true opinions may also vary for some questions that call upon the respondents to assess the performance of their colleagues or people on whom they depend for the provision of services. While this and other bias possibilities exist, no instances of this bias were identified to the evaluation team for this study. Section 3: Evaluation Findings (F), Conclusions (C), & Recommendations (R) Evaluation findings in this section are presented in relation to FFPr Strategic Objective (SO) 1: Increase Agricultural Productivity and Strategic Objective (SO) 2 – Expanded Trade of Agricultural Products (Domestic, Regional, and International). Subsequent sections will also present findings pertaining to each of the three groups of evaluation questions described above: (1) performance in relation to targets on official program indicators; (2) responses to questions about specific EQ activity results and (3) answers to questions posed in relation to internationally accepted evaluation scope topics identified in the evaluation Statement of Work. 36 3.1 Performance against EQ Program Performance Indicators Evaluation Findings The EQ program underwent a series of revisions to indicator targets, measurements, and definitions, which is not uncommon for large and complex programs. Some EQ revisions to results and activity indicators responded to midterm evaluation recommendations, and some were reportedly a product of the changing agricultural landscape in the DR in the past six years, and the impact that had on agricultural value chain actors. Table 11 presents the EQ program’s performance indicators, and its documented life of program (LOP) results46 The evaluation team did not conduct any indicator data verification in the form of spot checks on indicator calculation methodology as this was outside the scope of this evaluation TOR. 46 EQ Program Results Summary Table provided by EQ program team 37 Table 11: EQ Performance Indicator List, LOP Target, and Actuals Achieved Indicator # Indicator Name LOP Target Baseline Value Achievement as at Midterm % of LOP Target Achieved at Midterm Achievement as at Endline % of LOP Target Achieved at Endline Result Indicator #1 (FFPr 17) Total number of individuals benefiting directly as a result of USDA assistance 13,544 0 2,650 20% 15,791 117% Result Indicator #2 (FFPr 18) Total number of individuals benefiting indirectly as a result of USDA assistance. 67,720 0 13,250 Multiplier of 5 on Result Indicator #1 20% 78,955 117% Result Indicator #3 (FFPr 16) Number of individuals who have received short-term agricultural sector productivity or food security training as a result of USDA assistance. 12,300 0 2,236 18% 14,734 120% Result Indicator #4 (FFPr 1) Number of hectares under improved techniques or technologies as a result of USDA assistance. ( in hectares) 12,882 0 597 5% 15,690 121% Result Indicator #5 (FFPr 2) Number of individuals who have applied new techniques or technologies as a result of USDA assistance. 5,420 0 947 17% 5,580 103% Result Indicator #6 (FFPr 3) Number of individuals who have applied improved farm management practices (i.e., governance, administration, or financial management) as a result of USDA assistance. 1,050 0 425 40% 2,381 226% Result indicator #7 (FFPr 13) Value of sales by project beneficiaries. (in USD) $124,440,478 0 $11,718,778 9% $138,830,145 112% Result indicator #8 (FFPr 14) Volume of commodities (metric tons) sold by project beneficiaries. 117,806 0 20,491 17% 123,615 105% 38 Indicator # Indicator Name LOP Target Baseline Value Achievement as at Midterm % of LOP Target Achieved at Midterm Achievement as at Endline % of LOP Target Achieved at Endline Result indicator #9 Percentage of tests of target post￾production ag products that come back clean. 80% 0 100% 125% 95% 119% Result indicator #10 Number of direct beneficiaries achieving internationally recognized certifications. 15 0 0 0% 16 106% Result indicator #11 Average percent reduction in time a product takes to get through problem points in the value chain. 45% 0 0% 0% 54% 120% Result indicator #12 Number of partner packinghouses using a traceability system as a result of USDA assistance. 60 0 0 0% 64 106% Result indicator #13 Number of post-harvest equipment and facilities improved after USDA assistance. 238 0 4 2% 336 141% Result indicator #14 Number of food safety/standards-ready target commodities with improved marking and branding. 27 0 1 4% 30 111% Result indicator #15 (FFPr 9) Value (in USD) of new public and private sector investment leveraged as a result of USDA assistance. $1,222,500 0 $278,236 23% 4,151,317 339% Result indicator #16 Value (in USD) of buyer/seller contracts facilitated. $11,500,000 0 $374,584 3% 14,491,081 126% Result indicator #17 Average Organization Capacity Index (OCI) score of target producer groups 85% - No intermediate targets or results - 85% 100% Result indicator #18 Percentage of target value chain groups that have a systematic way of disseminating and updating market information. 90% 0 0% 0% 90% 100% 39 Indicator # Indicator Name LOP Target Baseline Value Achievement as at Midterm % of LOP Target Achieved at Midterm Achievement as at Endline % of LOP Target Achieved at Endline Result indicator #19 (FFPr 8) Number of public-private partnerships formed as a result of USDA assistance. 150 0 15 10% 196 130% Result indicator #20 (FFPr 7) Number of private enterprises, producers organizations, water users associations, women's groups, trade and business associations, and community-based organizations (CBOs) that applied improved techniques and technologies as a result of USDA assistance. 270 30 11% 417 154% Activity indicator #1.1 Number of trainings delivered to increase capacity of fruit and vegetable producers, processors, and packhouses. 780 0 77 10% 749 96% Activity indicator #1.2 Number of producers benefiting from training and technical assistance. 8,879 0 1,100 12% 10,069 113% Activity indicator #1.3 Number of processors benefiting from training and technical assistance. 150 0 85 57% 371 248% Activity indicator #1.4 Number of packhouse staff benefiting from training and technical assistance. 2,000 0 367 18% 2,387 119% Activity indicator #1.5 Value (in USD) of grants provided to producers. 507,171 0 $5,678 1% 603,449 119% Activity indicator #2.1 Number of processors/exporters trained in marketing and branding techniques. 1,087 0 294 27% 1,154 106% Activity indicator #2.2 Number of direct beneficiaries participating in trade shows, buyer visits to DR, and exporter visits to the US and Europe. 325 0 98 4% 329 101% 40 Indicator # Indicator Name LOP Target Baseline Value Achievement as at Midterm % of LOP Target Achieved at Midterm Achievement as at Endline % of LOP Target Achieved at Endline Activity indicator #2.3 Number of producer association and trade association members benefiting from end-market assessments 850 0 86 10% 777 91% Activity indicator #2.4 Number of individuals from organizations trading target commodities benefitting from capacity building activities 564 0 11 2% 567 100% Activity indicator #2.6 Number of women in trade and producer organizations management and private businesses who participate in program sponsored leadership training events 120 0 3 2% 226 188% Activity indicator #2.7 Value (in USD) of grants provided to trade associations or producer associations that promote exports. $34,000 0 $816 2% 91,337 269% Activity indicator #3.1 Number of Individuals attending cold chain knowledge sharing events 750 0 477 63% 1,443 192% Activity indicator #4.1 Number of fruit and vegetable processors benefiting from in-kind equipment to create value added products 39 0 2 5% 40 103% Activity indicator #4.2 Value (in USD) of in-kind grants provided to processors. 350,000 0 $58,150 17% 640,638 183% Activity indicator #5.1 Number of individuals reached by public information campaign. 1,500,000 0 298,629 20% 5,728,481 381% 41 Conclusions: EQ program met or exceeded life of program (LOP) targets for all but two indicators, with critical ramp up in activities seen between midterm and endline. At the time of the midterm evaluation, more than three quarters of the results and activity indicators had achieved less than 50 percent of the LOP target, but by the subsequent years past the midterm evaluation, EQ indicator targets had been exceeded on all but two indicators. A few of EQ indicators and their targets had been modified over the course of the program based on programmatic changes, and feedback from evaluators at baseline and midterm. Some of these changes have been referenced below. - Result Indicator 7: Value of sales (in USD) by project beneficiaries. Original cumulative target at the end of Year 4 was $73,000,000. This was brought down to $29,085,550 to better reflect the sales generated with the help of USDA assistance. This indicator target was subsequently increased to $124,440,478 for its Year 6 cumulative target. - Result Indicator 8: Volume of sales (in MT) by project beneficiaries. Original cumulative target at the end of Year 4 was 52,000 MT. This was brought down to 27,628 MT to better reflect the sales generated with the help of USDA assistance. This indicator target was subsequently increased to 117,806 MT for its Year 6 cumulative target. - Activity Indicator 3.1: The indicator name was changed from ‘Number of individuals attending demonstration of cold chain activities’ to ‘Number of individuals attending cold chain knowledge sharing events’ to reflect the number of individuals participating in various types of cold chain related events: Consequently, the original cumulative Year 4 target was increased from 358 to 750. This remained as the life of project target, i.e. Year 6 cumulative target for the program. - Result Indicator 3: Number of individuals who have received short-term agricultural sector productivity or food security training as a result of USDA assistance. The original LOP target for this indicator was 6,333. This was subsequently increased to 12,300. - Activity Indicator 1.1: Number of trainings delivered to increase capacity of fruit and vegetable producers, processors, and packhouses.The original LOP target for this indicator was 17. This was subsequently increased to 780. - Activity Indicator 2.3: Number of producer association and trade association members benefiting from end-market assessments The original LOP target for this indicator was 60. This was subsequently increased to 850. Result Indicator 4: Number of hectares under improved techniques or technologies as a result of USDA assistance. The original LOP target for this indicator was 5,000 hectares. This was subsequently increased to 12,882. - Result Indicator 5: Number of individuals who have applied new techniques or technologies as a result of USDA assistance.The original LOP target for this indicator was 6,000. This was subsequently decreased to 5,420. - Result Indicator 6: Number of individuals who have applied improved farm management practices (i.e., governance, administration, or financial management) as a result of USDA assistance. The original LOP target for this indicator was 1,250. This was subsequently decreased to 1,050. - Activity Indicator 2.4: Number of individuals from organizations trading target commodities benefitting from capacity building activities. The original LOP target for this indicator was 140. This was subsequently increased to 564. - Activity Indicator 2.6: Number of women in trade and producer organizations management and private businesses who participate in program sponsored leadership training events. The original LOP target for this indicator was 107. This was subsequently increased to 120. These data with several increases that had been made to indicator targets over the life of program, together with interview and survey responses led the evaluation team to conclude that EQ’s technical and operational management by program staff was effective. Sections 3.2, 3.3, and 3.4 below provide 42 additional context for activity and results indicators by assessing the various activity-level findings from the quantitative and qualitative data that were collected as part of this evaluation. Recommendations: No recommendations 3.2 EQ Program Performance in relation to Activity Questions This section starts with the evaluation team’s findings about the degree to which target group farms and firms actively participated in or received support from EQ program activities. Participation of Target Group Farms EQ Training and/or Support Received. Of the 332 farm survey respondents, the highest participation was seen for who participated in EQ integrated pest management (IPM) training (57 percent), followed by 49 percent (161 farmers) exposed to farm management training. Nineteen percent of the farm survey respondents (64 farmers) received in-kind grants: equipment assistance from EQ. See Figure 17 below. Figure 17: EQ Support Received by Farmer Beneficiaries in Sample (by %) Participation of Individuals Working in Target Group Firms EQ Training and/or Support Received by People in Firms. Of the 124 respondents to the people in firms survey, the highest participation was seen for the food safety modernization act (FSMA) training (87 percent), followed by 84 percent for certification programs. Six percent of the sample received cold chain support from EQ. See Figure 18 below for more details47 . 45% 57% 36% 49% 28% 19% 0% 10% 20% 30% 40% 50% 60% Sanitary and food safety standard training IPM Farmer Field School Farm Management Training Plants Donation In-Kind Grants: Equipment Farmer Sample (n=332) Participation in EQ (in %), by Support/Training Received 43 Figure 18: EQ Support Received by Firm Beneficiaries in Sample (by %) Training or Support reported by Firm Leadership. As Figure 19 below indicates, of the 19 firm leaders who responded to the online mini survey, 68 percent participated in the EQ traceability systems activity and for the FSMA compliance activity, 68 percent participated, followed by 58 percent for the in-kind equipment grants activity. Figure 19: EQ Support Received by Firm Leaders in Sample (by %) 87% 84% 6% 11% 23% 15% 9% 0% 20% 40% 60% 80% 100% FSMA Training Certification Program Cold Chain Support In-Kind Grants: Equipment Branding & Marketing Buyer-Seller Negotiation Support Trade Show & Trade Mission People in Firms Sample (n=124) Participation in EQ (in %), by Support/Training Received 42% 68% 63% 68% 32% 32% 42% 53% 37% 32% 37% 21% 58% 26% 16% 0% 10% 20% 30% 40% 50% 60% 70% 80% Post-Harvest Partnerships FSMA Compliance SPS Compliance Traceability System Market-Driven Certification Producer Group Governance & Services End Market Analysis Trade Show & Trade Missions Trade Leadership for Women Marketing & Branding Young Exporters Network Cold Chain Improvement In-kind Equipment & Grants Public Information Consumer Food Safety Campaign Organizational Capacity Reviews Mini Survey Respondents' Participation Rate (in %) in EQ Interventions (n=19) 44 3.3 Broad Results of EQ Activity Participation- EQPerformance Related to Achieving Strategic Objective 1 and Strategic Objective 2 Before turning to answers to questions about the performance and effects of specific EQ program activities or interventions, the evaluation team presents its finding from the three surveys it conducted and key informant interviews on the broad effects of participation in the EQ program. These broad answers focus on the extent to which activities and sub-activities meet FFPr Strategic Objective (SO) 1: Increase Agricultural Productivity, with productivity being measured by yield, production costs, product quality, and application of improved techniques and technologies, and Strategic Objective 2: Expanded Trade of Agricultural Products (Domestic, Regional, and International), measured by market access, sales volume, and sales value. The answers here are for the program as a whole, not specific activities. SO1: Increase Agricultural Productivity 1. Application of Improved Techniques and Technologies Application of Improved Techniques and Technologies on Farms. Of the 332 farmers sampled, 270 (81 percent) stated that they had applied improved techniques and technologies that they were trained on by the EQ Program, on their farm. See Figure 13 below for a breakdown of the different techniques and technologies applied by farmers on their farms. Figure 20: Farm Beneficiaries' Rate of Application of Improved Techniques (by %), by Value Chain Application of improved techniques and technologies varied amongst farmers in the different value chain. Among cocoa farmers surveyed, ~55 percent of them stated they applied improved pruning techniques, improved fertilization techniques and improved pest and disease management. Around 95 percent of the greenhouse vegetable farmers, and 93 percent of Asian vegetable farmers surveyed 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Improved Fertilization Techniques Pest & Disease Management Pruning Techniques Harvest/Post-harvest Handling Techniques Techniques/Technologies Adopted by Farmers in Sample (in %) by Value Chain (n=270) Asian Vegetables Greenhouse Vegetables Pineapples Cocoa Avocados 45 responded they apply improved fertilization techniques and pest and disease management techniques. Around 77 percent of avocado farmers surveyed stated they improved their fertilization techniques with EQ support. In general, one notes that a high portion of farmers (>50%) in all value chains adopted improved pruning techniques, pest and disease management strategies, and improved fertilization methods. The one exception is pruning in pineapples (pruning is not generally used). Fertilization and pest and disease management showed the highest rates of application and adoption which is to be expected given the relatively rapid response of both annual crops such as vegetables, and tree crops such as cocoa and avocado to those improved practices. Figure 21: Most Beneficial EQ Activities as Stated by Sampled Farmers (in %) As figure 14 indicates, the vast majority of farmers surveyed across each different value chain found EQ trainings to be the most beneficial- between 85 percent of surveyed cocoa farmers to up to 96 percent of all surveyed avocado farmers. These trainings ranged from sanitary and phytosanitary standards, FSMA readiness, certification training etc. In-kind equipment and tools activity was stated as being beneficial to around 47 percent of the surveyed pineapple farmers, 35 percent of cocoa farmers, and between 20 and 30 percent of greenhouse and Asian vegetable farmers respectively. Public information campaigns that targeted the general public in raising food safety awareness was considered beneficial among 8-10 percent of surveyed farmers across the different value chains. A very small percentage (between 2 and 4 percent) of surveyed EQ beneficiaries claimed that none of the activities were beneficial to them. Application of Improved Techniques and Technologies in Firms. Of the 92 people in firms survey respondents48 employed by exporters, processors, packinghouses, 77 (84 percent) received training from EQ on adopting improved techniques and technologies in their firms, 82 percent of whom (63) have applied what they learned from EQ in their work. The most widely applied techniques by people in 48 Note that of the 124 people in firms surveyed, 32 respondents were government representatives to whom the application of improved practices/technology question on the firm survey is inapplicable. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Training In-Kind Equipment/Tools Public Information Campaign None EQ Activities Most Beneficial to Sampled Farmers (%) by Value Chain Asian Vegetables Greenhouse Vegetables Pineapples Cocoa Avocados 46 firms has been the good manufacturing practices and adopting chlorination systems in their factories (90 percent). See Figure 22 and Table 12 below for a breakdown of the different techniques and technologies applied by people in firms. Figure 22: Firm Beneficiaries' Rate of Application of Improved Techniques (by %) The table below breaks down figure 22 by value chain to provide context as to adoption practices acrossprocessors, packinghouses, exporters representing different value chains. Table 12: Number of Surveyed Firms Adopting Improved Techniques/Technologies with EQ-Support Improved techniques/technologies adopted by firms Number of Surveyed Firms Adopting Improved Techniques/Technologies Avocados Cocoa Pineapple Greenhouse Veg. Asian Veg Total Good Manufacturing Practices 6 8 2 21 20 57 Preventative Controls for Human Food (FSMA) 2 1 0 4 6 13 Traceability System 8 3 2 16 14 43 Chlorination System 10 7 2 21 17 57 Inspection Table/Cabin 3 2 0 6 2 13 Conclusion. As stated in the findings, around 81 percent of surveyed farmers and 82 percent of surveyed people in firms have adopted improved practices and techniques as part of their operations, as a result of EQ support. The rate of adoption varies by practice, but the overall adoption is indicative of their 90% 21% 68% 90% 21% 0% 20% 40% 60% 80% 100% Good Manufacturing Practices Preventive Controls for Human Food (FSMA) Traceability System Chlorination System Inspection Table/Cabin Application of Improved Techniques & Technologies by Firms (n=63) 47 overall benefits towards farm and firm operations. Further investigation by EQ or other programs on why certain practices are adopted more than others could provide useful insight into country-specific preferences. Recommendation. None 2. Production Volumes and Production Costs Effect of EQ Support on Farmer Input/Output(s). Majority of farm survey respondents (i.e. 57 avocado farmers [63 percent], 103 cocoa farmers [66 percent], 13 pineapple farmers [43 percent], 80 percent of 20 greenhouse vegetable farmers [80 percent] and 29 Asian vegetable farmers [94 percent]) reported having increased production volumes with increased production costs, as a result of receiving EQ support. This increased cost could be attributed to their increased production volumes (see Table 13 below), and stem from the increased use/application of biofertilizer, or the implementation of EQ training components on their farms, which often have monetary costs associated with them. Interestingly some farmers reported having increased production volume but with decreased production costs. These include around eight avocado farmers (nine percent of the surveyed avocado farmers), five cocoa farmers (three percent), eight pineapple farmers (27 percent), three greenhouse vegetable farmers (12 percent), and 1 Asian vegetable farmer (3 percent). Specific practices adopted by this small group of farmers who stated having increased their production volumes with decreased production costs, were not explored during the farmer survey. Table 13: Sampled Producers’ (by value chain) (in %) Perception on EQ Support’s Effect on Cost and Volume of Production Producers (in %) Perception on EQ Support's Effect on Production Volume & Costs Production Volume Production Costs Increased No change Decreased Don't know Total Avocado Increased 63% 11% 9% 2% 77 No change 3% 8% 0% 0% 10 Don't know 0% 0% 0% 4% 4 Total 66% 19% 9% 7% 91 Cocoa Increased 66% 6% 3% 1% 118 No change 1% 12% 1% 0% 22 Decreased 1% 1% 8% 1% 15 Total 68% 19% 12% 1% 155 Pineapple Increased 43% 0% 27% 0% 21 No change 0% 13% 0% 0% 4 Decreased 3% 3% 10% 0% 5 Total 47% 17% 37% 0% 30 Greenhouse Vegetables Increased 80% 0% 12% 4% 24 Decreased 0% 0% 4% 0% 1 Total 80% 0% 16% 4% 25 Asian Vegetables Increased 94% 0% 3% 0% 30 48 Producers (in %) Perception on EQ Support's Effect on Production Volume & Costs No change 0% 3% 0% 0% 1 Total 94% 3% 3% 0% 31 Figure 23 below shows that of the 42 farmers who stated their production costs had reduced with EQ support, the majority (across the different value chains) experienced cost reduction by 30 percent. These 42 farmers with reduced production costs include 8 avocado farmers, 18 cocoa farmers, 11 pineapple farmers, 4 greenhouse vegetable producers, and 1 Asian vegetable producer. Figure 23: Sampled Farmers (in %) Stating the Extent of Production Costs Reduction Rate They've Seen with EQ Support Effect of EQ Support on Firm Input/Output(s). The majority of firm survey respondents (over 80 percent) have increased their outputs (production volume, and production quality) as a result of participation in EQ activities. Eighty-four percent stated that their market access has increased due to support received from EQ. Notably close to 90 percent of firm survey respondents (i.e. 82 respondents) also stated that their firms’ production costs have increased upon receiving EQ support. This could be attributable to the increased production volume for firms, or that they require more raw material inputs, since out of these 82 firm representatives, 80 also reported an increase in production volumes, while the remaining two firms recording a decrease. Five firm respondents who reported a decrease in production costs, also reported a decrease in productions volume. See Table 14 below for more details. Table 14: Number of Firms Staff Interviewed According to the Cost and Volume of Production Perceived Changes in Firm Production Volumes Perceived 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0-<10% 20% 30% 40% 50% >50% Don't know % of Farmers % Reduction in Production Costs % Reduction of Production for Sampled Farmers (in %) Having Reduced Costs (n=42) Avocado Cocoa Pineapple Greenhouse Veg Asian Veg 49 Changes in Firm Production Costs Perceived Increased No Change Decreased Don’t Know Total Increased 80 0 2 0 82 No change 0 2 0 0 2 Decreased 0 0 5 0 5 Don’t know 0 1 0 2 3 Total 80 3 7 2 92 All firm survey respondents who stated that their firms’ production volume had increased as a result of EQ support, also reported an associated increase in inputs as well. See Figure 16 for more details. 3. Yield Yield is output per unit of production area, and is difficult to attribute to specific interventions due to its multi-dimensional nature. Season-on-season, yield is impacted by weather (temperature, rainfall), solar radiation, soils and soil fertility, pests and diseases, and cultivar or variety being produced. Management plays a big role in yield, including such practices as planting time, variety used, fertilization, use of pesticides and/or Integrated Pest Management (IPM), irrigation (if used), and in the case of tree crops, pruning of trees and shade management. The EQ program supported increased on-farm productivity through training in improved practices of fertilization, pest management (IPM), pruning, use of improved seeds and/or seedlings, and timing and techniques of harvest. In cocoa and avocado, especially, the EQ program included support for the renovation of farms, focused mostly on the planting of new, improved seedling varieties. Yield however was not an indicator that the EQ Program was reporting or tracking, nor was it measured for program beneficiaries at baseline and midterm evaluations. As such, the evaluation team did not conduct a comparison of perceived changes in yield across different time periods. 50 Figure 24: Sampled Farmers’ Changes in Yield as a Result of EQ Support As figure 25 above indicates, most surveyed farmers (ranging from between 73 percent of pineapple farmers and up to 100 percent of greenhouse vegetable farmers) stated that their yields have improved as a result of EQ support. A small fraction of sampled farmers, i.e. around 13 percent of pineapple farmers, and 10 percent of cocoa farmers stated decreased yields. Of those that stated increased yields, the majority of the farmers, across the different value chains saw increases between 10 and 30 percent- see table 15 below. Table 15: Sampled Farmers’ Percentage Increase in Yield as a Result of EQ Support Percentage Increase in Yield by Sampled Farmers (in %) % Increase in Yield Avocado Cocoa Pineapple Greenhouse Vegetables Asian Vegetables <10% 1% 10% 0% 4% 10% 10% 29% 35% 23% 12% 7% 20% 28% 34% 28% 28% 28% 30% 21% 10% 32% 28% 23% 40% 4% 4% 5% 8% 10% 50% 11% 4% 9% 12% 13% >50% 5% 2% 9% 8% 10% Don't know 1% 1% 0% 0% 0% 4a) Product Quality-Farmers As shown in figure 25 below, between 80-100 percent of sampled farmers across the different value chains stated that their product quality had improved as a result of receiving EQ support. Key informant interviews also indicated that producers of annual crops (vegetables, pineapple) reported improved 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Avocados Cocoa Pineapples Greenhouse Vegetables Asian Vegetables Farmers (in %) Stating Change in Yield as a Result of EQ Support Increased No change Decreased Don't know 51 quality of their produce due to the use of improved varieties, improvements in harvesting techniques and post-harvest handling, and the increased knowledge of pest and disease control gained via the Farmer Field Schools (FFS). Avocado producers indicated improved fruit quality from pruning and shade management, proper use of fertilizers, along with proper harvest timing and techniques. For example, representatives of a farmers’ association reported that their EQ/FFS training led them to use gloves when harvesting avocados to avoid scratching the fruits. This practice led to a significant increase in the percentage of fruit eligible for export markets. This converges with farmer survey findings as shown in figure 25 below, where over 90 percent of the sampled avocado farmers stated that their product quality had increased with EQ support. Cocoa producers, similar to avocado growers, cited improved tree fertilization, better shade management, increased knowledge of pests and diseases, and more accurate pruning techniques contributed to both increased cocoa bean yields and cocoa quality. Figure 25: Sampled Farmers (in %) Perception of Product Quality Change as a Result of EQ 4b) Product Quality-Firms EQ supports firms with training and technical assistance to improve product quality. While the program does not have set definitions for product quality for agricultural products, the program does have two measures of quality improvement that it routinely monitored namely Results indicator #9: Percentage of tests of target post-production agricultural products that come back clean and Results indicator #14: Number of food safety/standards-ready target commodities with improved marking and branding. Post￾production agricultural products that “pass” or come back with no significant findings indicate improved quality. Types of produce sample tests include those that detect the presence of salmonella in chili peppers, and other kinds of infestations, such as thrips (an insect whose presence has been detected in both Asian and greenhouse vegetables) or any sort of fungi, bacteria, or virus. The aim of the indicator is 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Avocado Cocoa Pineapple Greenhouse Veg Asian Veg % of sampled farmers Change in Product Quality Percieved by Sampled Farmers (in %) as a result of EQ Support Don't know Decreased No change Increased 52 to show progressive improvement in the quality of post-production produce measured by the reduction of the infestation level49 . For the endline evaluation, of the surveyed firms, 41 had submitted tests to local testing labs. Table 16 below shows the breakout of the tests that can back clean, thus meeting requirements for post￾production processing & handling, and indicating product quality improvements. Of the 41 firms that submitted their products for testing, 19 firms (45 percent) stated that between a quarter and a half (25%-50%) of the products submitted for testing came back clean. Around 15 firms stated that between half and three-quarters of tests they submitted to labs came back clean. Table 16: Number of Firms that Submitted Ag Products for Testing & the Percentage that Came Back Clean Percentage of tests that came back clean among sampled firms Value Chain: # of Firms that submitted ag products to testing labs Avocado Cocoa Pineapple Greenhouse Veg. Asian Veg Total <25% 0 1 0 1 1 3 25-50% 2 1 1 9 6 19 51-75% 3 1 0 3 8 15 76-99% 2 0 1 0 0 3 100% 0 0 0 0 1 1 Total 7 3 2 13 16 41 Results indicator #14: Number of food safety/standards-ready target commodities with improved marking and branding was another measure of product quality. Specific branding related support was offered to packinghouses and processors in the form of improved design of packages, logos, nutrition information, to make those products export ready. While EQ Program did not collect value chain disaggregated data, a large portion of branding efforts were focused on the pineapple, avocado, and cocoa value chains. See table 17 below for more details on EQ’s achievement on RI#14. Table 17: EQ Result Indicator 14 LOP Target and Actuals Result Indicator #14 LOP Target Baseline Value Achieved at midline Midline Achievement (As % of LOP Target) Achieved at endline Endline Achievement (As % of LOP Target) Number of food safety/standards￾ready target commodities with improved marking and branding 27 0 1 4% 30 111% 49 FFPR USDA EQ Program Midterm Evaluation, 2018 53 SO2: Expanded Trade of Agricultural Products (Domestic, Regional, and International) 1. Market Access As shown below in Figure 26, between 70 and 100 percent of sampled firms that work in the different value chains expressed that their market access had increased after receiving support from EQ. A small fraction of firms (17 percent of greenhouse vegetable firms, 6 percent of Asian vegetable firms, 7 percent that work with cocoa, and 14 percent of firms that work with avocado) stated that their market access had decreased. Figure 26: Perceived Change in Firms’ Market Access by Sampled Firms (in %) as a Result of EQ 2. Sales Value-Farms and Firms EQ Program collected sales value data (in USD) for farms and firms, under Result Indicator #7 (FFPr 13): Value of commodities (in USD) sold by project beneficiaries without disaggregating by type (i.e. farm/firm) or by value chain. The endline farmer and firm survey collected sales value data (disaggregated by value chain) from farmers and firms at two discrete time periods to assess whether COVID-19 played a particular role in beneficiary sales. For the purpose of analysis, considering the lack of programmatic data at the necessary disaggregated level (by firm/farm type; by value chain), the evaluation team collected secondary data from sources including FAOstat, and EQ documentation including but not limited to the baseline, and midterm evaluations. Sales value data collected at endline across 2019 and 2020, and EQ data collected over the past six years (cumulative as shown in midterm and endline points of the program) through RI#7/FFPr 13 will be used to assess whether the datapoints collected across different lines of inquiry follow the national trend as shown by FAOstat. Using FAOStat’s data on gross production value as a reasonable proxy for sales, the figure below provides trends for cocoa, pineapple, and avocado value chains between 2015-201850 . 50 2018 is the latest year for which gross production value information for the DR is presented on FAOStat. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Avocado Cocoa Pineapple Greenhouse Veg. Asian Veg. Value Chains in which the Firms Operate % of firms Change in Firm's Market Access Percieved by Sampled Firms (in %) as a result of EQ Support Don't know Decreased No change Increased 54 Figure 27: Gross Production Value (a proxy for sales) for Select Value Chains in the DR, 2009-2018 Figure 27 shows pineapple production value was steadily increasing from 2009 and almost quadrupling from $120 million to 2016 to $451 million in 2017. All three value chains see a drop in value from 2017 to 2018 with avocado’s production value dropping by over 50 percent. Cocoa’s largest production value dropped slightly from $202 million to $188 million between 2017 and 2018. Figure 28 shows the sales value data for farmers and firms surveyed for the final evaluation. The downward trend for cocoa and pineapple value chains between 2019 and 2020 is following the national trend (in Figure 27) between 2017 and 2018), knowing that the time periods covered by FAOStat data and the final evaluation data are different. Avocado beneficiaries in the sample saw an upward trend between 2019 and 2020, which is incongruous with the 2017-2018 national trend for the crop. Figure 28: Consolidated Value of Sales (in USD) for Farm and Firm Beneficiary in Sample $- $50,000 $100,000 $150,000 $200,000 $250,000 $300,000 $350,000 $400,000 $450,000 $500,000 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 $ (in '000s) DR Gross Production Value (in '000s) for Select Value Chains, 2009- 2018 (Source: FAOStat) Avocado Cocoa Pineapple $- $200,000 $400,000 $600,000 $800,000 $1,000,000 $1,200,000 $1,400,000 $1,600,000 Avocado Cocoa Pineapple Greenhouse Veg Asian Veg Value of Sales (in USD) of Sampled Beneficiaries 2019 2020 55 The majority of the farms surveyed (i.e. 80 percent in 2019 and 78 percent in 2020) recorded annual sales up to $10,000, while the majority of people in firms surveyed (i.e. 61 percent in 2019 and 58 percent in 2020) reported annual sales between $10,000 and $100,000. Table 18 below shows EQ’s achievement towards FFPr 13: Value of commodities (in USD) sold by project beneficiaries that count both producer groups and processors’ value of sales that combines all value chains, since the program was not required to collect disaggregated data. Table 18: EQ LOP Target and Achievement of FFPr 13 Value of Commodities Sold by Project Beneficiaries FFPr 13 LOP Target Baseline Value Achievement as at Midterm Midterm achievement (As a % of LOP target) Achievement as at Endline Endline achievement (As a % of LOP target) Value of commodities (in USD) sold by project beneficiaries $124,440,478 0 $11,718,778 9% $138,830,145 112% 3. Sales Volume- Farmers and Firms EQ Program collects sales volume data (in MT) for producer groups and processors under Result Indicator #8 (FFPr 14): Volume of commodities (metric tons) sold by project beneficiaries without disaggregating by type (i.e. farm/firm) or by value chain. The endline farmer and firm survey collected sales volume data from farmers and firms at two discrete time periods to assess whether COVID-19 played a particular role in beneficiary sales. For the purpose of analysis, considering the lack of programmatic data at the necessary disaggregated level (by firm/farm type; by value chain), the evaluation team collected secondary data from sources including FAOstat, and EQ documentation including but not limited to the baseline, and midterm evaluation. Sales volume data collected at endline across 2019 and 2020, and EQ data collected over the past six years through RI#8/FFPr 14 will be used to assess whether the datapoints collected across different lines of inquiry follow the national trend as shown by FAOStat. See figures 29 and 30 below for sampled beneficiary sales volume data as it compares to national level production volume data in the DR. 56 Figure 29: DR Production Volume (in MT) The two-year trends of volume of sales for project beneficiaries in the sample for avocado and cocoa track similarly to the national production volume trends. Pineapple is an outlier, showing a large decrease for project beneficiaries while the national trend is slowly up. Project beneficiaries in greenhouse and Asian vegetables greatly increased their volume of sales from 2019 to 2020, whereas national production trends are fairly flat- see figure 30 below. Figure 30: Volume of Sales (in MT) for Sampled Beneficiaries - 100,000 200,000 300,000 400,000 500,000 600,000 700,000 Avocados Chili Peppers Cocoa beans Pineapple Eggplant Tomato Metric Tons of Commodity Produced DR Production Volume (in MT) (Source FAOStat) 2015 2016 2017 2018 2019 0 1000 2000 3000 4000 5000 6000 7000 8000 Avocado Cocoa Pineapple Greenhouse Veg Asian Veg Metric tons of commodity produced Volume of Sales (in MT) for Sampled Beneficiaries 2019 2020 57 Figure 31: Firm Sales Volume (in %), by Year, for EQ Beneficiaries in Sample As Figure 31 indicates, the majority of firms (i.e. 52 percent in 2019 and 48 percent in 2020) sold up to 100 metric tons (MT) of goods. Only a very small proportion of firms (i.e. 3 percent in 2019 and 2020) sold over 5,000 MT of goods. The sales volume data are quite different when it comes to farms. In 2019, around 97 percent of all surveyed farms and 98 percent in 2020 sold up to 100 MT of goods. Figure 32: Volume of Sales for Sampled Farmers (in %) As shown in figure 32, close to 90 percent of sampled farmers (i.e. 256 in 2019 and 269 in 2020) sold up to 20 MT of their main crop in 2019 and 2020. Around 3 percent of sampled farmers sold more than 100 MT of their main crop in 2019 (i.e. 8 farmers) and 2 percent (i.e. 6 farmers) in 2020. This data however does not present the history of changes in sales value and volume throughout the life of the EQ program. EQ program exceeding its life of project (LOP) targets for Results Indicators #7: Value of 52% 34% 10% 3% 48% 31% 17% 3% 0% 10% 20% 30% 40% 50% 60% 0-100 MT 101-1,000 MT 1,001-5,000 MT >5,000 MT 0-100 MT 101-1,000 MT 1,001-5,000 MT >5,000 MT 2019 2020 Firms (represented in %) Sales Volume (in MT), by Year 0% 20% 40% 60% 80% 100% < 5 MT 5-15MT 15.1-30 MT 30.1-100 MT >100 MT Sampled Farmers (represented in %) Volume of Sales (in MT), 2019 Avocado Cocoa Pineapple Greenhouse Veg Asian Veg 58 beneficiary sales and Results Indicator #8: Volume of beneficiary sales however do provide some insight into EQ program’s positive effect on its beneficiary sales, although it may not be the only effect on beneficiary sales values and volumes. Conclusion: The degree to which the EQ program, versus other interventions contributed to these export sector effects cannot be discerned since sales value especially is determined by extraneous factors outside the program control-see Table 2 under Section 1.1 Program Context for more details. But the findings here indicate that sales value and volume trends have remained largely the same over the last two years for farms and firms, possibly suggesting a fairly minimal effect of COVID-19 on farm and firm operations when it comes to sales. 3.4 Activity Specific Evaluation Questions In this section the evaluation team presents study findings about specific activities under the program’s two capacity building clusters: (a) Producer groups/cooperatives and (b) Trade Associations Activity 1: Capacity Building: Producer Groups/Cooperatives For each of the interventions led under this activity, this section: • Answers the specific question asked about that activity using findings from all the methods used by the evaluation to collect data • States the conclusions of the evaluation team based on those findings • Presents any specific recommendations that follow from those activity specific findings and conclusions. A1.1 What impact has market-driven certification programs had on beneficiaries' access to markets and exports Evaluation Findings. Evaluation evidence from multiple data sources the team used (surveys, documents, KIIs) indicates that the EQ program’s market-driven certification activity improved firms’ agricultural certification status and ability to meet US or European production standards. Table 19 below shows that the around 31 percent of people in firms, and 50 percent of firm leaders who participated in EQ’s certification program worked in the greenhouse vegetable value chain. Table 19: Certification Program Beneficiaries by Data Source, and Value Chain Data Source Certification Program Beneficiaries Avocado Cocoa Pineapple Greenhouse Vegetable Asian Vegetable Total Firm Survey Firm Beneficiaries by Value Chain (n=78) 10 14 1 24 29 78 % 13% 18% 1% 31% 37% - 59 Data Source Certification Program Beneficiaries Avocado Cocoa Pineapple Greenhouse Vegetable Asian Vegetable Total Mini survey Firm Leader Beneficiaries by Value Chain (n=6) 1 1 1 3 0 6 % 17% 17% 17% 50% 0% - In the cocoa value chain, organic certifications were reported to be more important than the EU￾certification- the Global GAP, while key informants in fruit and vegetable value chains cited Global GAP certification as being more important than organic certifications. Participants from firms had positive experiences as a result of working with EQ on the certification program. Around 90 percent of people in firms surveyed, and 83 percent of mini survey respondents either strongly agreed or agreed that the certification program was relevant and increased their firms’ access to markets. In addition, 94 percent of the firms surveyed stated that their product quality had improved as a result of EQ support. As stated in Section 3.3 above, result indicator 9: Percentage of tests of target post-production ag products that come back clean is a quality measure for firms. In the firm survey, 41 firms that responded they submitted agricultural products for testing at labs, and around 37 percent of those firms stated that between half and three-quarters (i.e. 50-75 percent) of those tests come back clean. Additionally, around 71 percent (74) of certificate program support recipients also attended trade shows and trade missions, 96 percent of whom stated that their knowledge of trade and market access had improved from participating in trade shows. The market-driven certification program has been cited as one of the most beneficial interventions offered by the EQ program. Notably, a large number of respondents stated that they intended to maintain the certifications they achieved with EQ program support. Close to 99 percent of the survey respondents who benefited from the certification program also reported that they were very likely or likely to continue to implement the practices and training that led to certification. Conclusion. Certification is critical for market access and exports for fresh produce and organic cocoa. EQ provided valuable support to producers, processors, and exporters, who wanted to get certified. Recommendation. Certification of production and/or processing is so critical to access to many export markets that USDA should engage directly in a transition effort at the end of this program to ensure that support to farms and firms in this area continued with Government and/or other donor resources, or with the help of private sector actors mobilized to carry on this vital work. "Global G.A.P certification with the help of EQ gave me access to new markets, especially in Europe." -Certification Program Activity 60 A1.2 What effect has the increased presence of traceability systems had on beneficiaries' market access and exports? Evaluation Findings: Of the 92 firms51 that responded to the firm survey, 72 percent (66 firms) had a traceability system in their firm, of which 68 percent (45) were implemented with EQ support. Close to 93 percent of these firm respondents reported that EQ’s support on traceability systems helped increase their firm’s access to markets, and around 62 percent of leaders in firms stated that their firm exports had increased as a result of receiving support from EQ on their traceability systems. Table 20 below provides a breakdown of EQ-supported traceability system beneficiaries by value chain. Table 20: Traceability System Program Beneficiaries by Data Source, and Value Chain Data Source Traceability System Program Beneficiaries Avocado Cocoa Pineapple Greenhouse Vegetable Asian Vegetable Total Firm Survey Firm Beneficiaries by Value Chain (n=45) 9 4 2 14 16 45 % 20% 9% 4% 31% 36% Mini Survey Firm Leader Beneficiaries by Value Chain (n=13) 2 3 3 3 2 13 % 15% 23% 23% 23% 15% All of the firm survey respondents, and 92 percent of leaders in firms who received traceability systems support either strongly agreed or agreed that they will continue to implement the improved practices and trainings offered by EQ. Conclusion: Traceability systems address the requirement of food safety and quality standards for entry into international markets. EQ was effective in helping its program beneficiaries adopt these systems and realize their benefits. Recommendation: The support for traceability systems provided by EQ to CEDAF is playing a significant role for firms trying to access new export markets or maintaining market access especially in the U.S. via FSMA. It is recommended that support continue under the USDA FFPr TraSa Project and that a plan be developed to institutionalize these efforts inside the GoDR and/or private associations. Particularly for FSMA-readiness and in support of other international market standards, support for 51 Excludes government representatives since traceability system is not applicable to them. "The traceability system helped our firm meet market demands for our food products and led to the discovery of new markets." -Exporter, EQ beneficiary 61 continued traceability systems development and CEDAF software maintenance should be a priority for the GoDR. A1.3 What impact has improved compliance with sanitary and phytosanitary standards (SPS) had on market access and exports? and A1.8 What impact has the FSMA readiness food safety strategy for packinghouses and related interventions had on market access and exports? Evaluation Findings: EQ led a series of support services to help firms increase compliance with SPS. Compliance with SPS standards helps reduce rejection rates of products at market entry due to pests, and pesticide residues. A total of 104 out of 10852 firm survey respondents (96 percent) indicated that they received SPS support from EQ. The mini-survey answered similarly. Around 92 percent of firm leaders surveyed reported that the SPS support received from EQ was effective and relevant to their firms work. Key informant interviews revealed that training in SPS assisted participants in developing process manuals and Dominican food safety registrations and helped keep the businesses in compliance with national regulations. Almost 100 percent of the firm survey respondents affirmed that they increased their compliance with SPS as a result of receiving this support from EQ. Around 95 percent of respondents indicated that they saw their production quality improve, with EQ support. Consequently, these firms access to markets also increased, with around 90 percent citing their access to both international and domestic markets, or exports increased. Similar trends in responses were provided by surveyed senior leaders who responded to these questions on the mini survey. All firm survey respondents who received SPS compliance support affirmed that they are either very likely or likely to continue to implement what they have learned from EQ. 100 out of the 108 (93 percent) of people in firms who received FSMA related support had also participated in FSMA readiness food safety strategy training activities. More than 85 percent of these 100 firms asserted that both their access to domestic and international markets and exports had increased as a function of the training they received. This could be attributed to the fact that the majority of the firm survey respondents who received FSMA training stated that the quality of their agricultural products had improved as a result of EQ support-See table 21 below. FSMA training was seen to be beneficial in terms of improved product quality to all firms in the cocoa value chain that received the training and to around 84 percent of firms that work with Asian vegetables. FSMA training was found to have affected no change in 43 percent of avocado firms that received the training, and all the pineapple firms that received the training. 52 Only 108 out of 124 people in firms surveyed responded to the question asking if they received SPS training from EQ " EQ support improved our product quality moving into the market and increased our knowledge of SPS requirements in new markets." -Exporter, EQ beneficiary “EQ held our hands as we navigated FSMA￾readiness, we could not have done it alone.” -FSMA training beneficiary 62 Table 21: Sampled Firms’ Perception of FSMA Training’s Impact on Agricultural Products How did the FSMA training impact the quality of your agricultural products? Value Chain for Firms Avocado Cocoa Pineapple Greenhouse Veg Asian Veg No change 43% 0% 100% 31% 13% Improved 57% 100% 0% 66% 84% Deteriorated 0% 0% 0% 3% 0% Not Applicable 0% 0% 0% 0% 3% This converges with responses from firm leaders on the mini survey who indicated receiving FSMA support helped increase access to markets and increased exports through EQ support. Around 99 percent of the surveyed individuals working in firms indicated that they were very likely or likely to continue to implement the training and practices provided by EQ, even beyond the end of the program. Conclusion. Compliance with SPS standards helps reduce rejection rates of products at market entry due to pests and pesticide residues. EQ’s success in fostering beneficiary improvements in SPS compliance for nearly 100 percent of beneficiaries assisted vastly improved their market access and reduced the likelihood of product rejection at market entry. Along a similar vein, respondents across different lines of inquiry indicated that FSMA readiness training and strategy development had a significant positive impact on access to markets and exports. Study data show that this training played a critical role in firms meeting SPS requirements and satisfying exports regulations. Recommendation. Inspections of produce and monitoring of SPS standards should be primarily a government function in an effort to continue to grow agricultural exports; the Ministry of Agriculture should mobilize resources to carry on this critical activity. USDA/APHIS should stay engaged in this effort. EQ demonstrated that training and the development of food safety strategies at packinghouses are investments in resources that firms should continue to carry on, to maintain access to important export (especially the U.S.) markets. 63 A1.4 What effect have the use of farmer field schools had on organizational effectiveness, production costs and yields? and A1.7 What impact has integrated pest management (IPM) had on beneficiary costs, yields, access to markets? Evaluation Findings Of the 120 farmer field school farmer beneficiaries, 43 percent were cocoa farmers, followed by 20 percent Asian vegetable farmers, 13 percent greenhouse vegetable growers, 13 percent pineapple farmers, and 12 percent avocado growers. Only three percent of these beneficiaries were Asian vegetable growers. Effect on Organizational Effectiveness. The evaluation team used a proxy indicator to assess organizational effectiveness in relation to this question- - since EQ does not have a measure for organizational effectiveness other than the Organizational Capacity Index (OCI) which is discussed later in the report and does not apply to FFS-- namely farmers’ application of improved techniques and technologies on their farms, which, in turn resulted in improvements in yield, sales and or exports. Of the 120 farmer field school beneficiaries, 94 percent, according to the farm survey, responded that they apply improved techniques and/or technologies on their farm as Figure 33 illustrates. The EQ program did not monitor the FFS beneficiaries’ adoption of improved techniques at different periods in time, and hence a comparison of adoption rates with baseline and midterm of the program was not conducted. However, through result indicator #5/FFPr 2: Number of individuals who have applied new techniques or technologies as a result of USDA assistance--the performance indicator aimed The Farmer Field School (FFS) is a training and knowledge transfer methodology that features hands-on, in-field and peer-to-peer learning activities. In the Dominican Republic, the Exporting Quality program utilized FFS and in-kind grants of tools to introduce new production practices, techniques, and technologies in all the targeted value chains. 83% 86% 63% 27% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Improved fertilization techniques Improved Techniques for Pest & Disease Management Improved Pruning Techniques Improved Harvesting & Post-Harvesting Techniques Farmer Field School Beneficiaries' Application (in %) of Improved Practices/Techniques on their Farms (n=112) Figure 33: Farmer Field School Beneficiaries' Application of EQ-supported Improved Techniques 64 at measuring improved farm management53, the number of beneficiaries who have applied improved techniques at baseline and midterm can be ascertained and is captured in Table 22 below. Table 22: EQ achievement of Results Indicator #5/FFPr#2 Result indicator #5/FFPr 2: Number of individuals who have applied new techniques or technologies as a result of USDA assistance LOP Target Actuals as of Baseline Actuals as of Midterm Actuals as of Endline Actual as of Endline as % of LOP Target 5,420 0 1,450 5,580 103% From the firm’s perspective, some firm leaders stated that their participation in farmer field schools allowed them to connect with producers and reduce the need for negotiations with intermediaries. FFS Effect on Production Yields. Around 95 percent of farmer field school beneficiaries reported that their production yields had increased, while 4 percent stated no change to their yields as a result of EQ support. A value chain wise breakdown of FFS beneficiaries and their percentage production yield increase is shown in figure 34 below. Over 45 percent of pineapple farmers in the sample saw 30 percent yield increases while for avocado farmers, under 30 percent (i.e. 4 farmers) saw 30 percent increases in yield. According to the president of a producers’ association engaged in the production of mango and avocado for export, Farmer Field School training through EQ for their 44 members resulting in a doubling of fruit yields in just two years. This key informant attributes the yield increase to improved tree pruning and fertilization methods.54 Figure 34: Percentage Yield Increase Among FFS Beneficiary Farmers in the Sample 53 USDA EQ Midterm Evaluation, 2017 54Key Informant Interview respondent, Personally Identifiable Information (PII) removed 0% 10% 20% 30% 40% 50% <10 % 10% 20% 30% 40% 50% >50% Don't know % of FFS beneficiary farmers % increase in yield Sampled FFS Beneficiaries (in %) by Value Chain & Percentage Yield Increase Avocado Cocoa Pineapple Greenhouse Veg Asian Veg 65 FFS Effect on Production Costs. The majority of farmer field school beneficiaries (81 percent) report increases in their total production costs, while only 8 percent saw decreased production costs. Of those who decreased their production costs, around half of them reported at least a ten percent decrease in costs. Around 97 percent of farmers who participated in EQ-led farmer field schools affirmed that they would continue to implement what they learned from their sessions even after EQ Program ends. Some key informants intimated that the DR’s Ministry of Agriculture does not have the resources to continue farmer field schools. However, some cocoa trade association affirmed they are willing to sustain farmer field schools for the benefit of their farmers. In addition to the positive responses by producers to Farmer Field School training, nearly all Ministry of Agriculture personnel engaged in Key Informant Interviews indicated that the EQ program training in all value chains was invaluable, resulting in increased commodity yields and reduced pesticide use, and that one of the most serious concerns they have about the sustainability of those activities is lack of Government resources to continue post-EQ. IPM was a key topic offered through the FFS. As figure 35 below indicates, of the 332 farm survey respondents, 57 percent had received IPM support from EQ. Around 70 of these farmers (37 percent) were cocoa growers, and 60 (32 percent) avocado farmers. Figure 35: IPM Beneficiary Famers in Sample, by Value Chain IPM Impact on beneficiary costs. Key informants representing cocoa trade associations reported that cocoa producers have adopted improved techniques, and have reduced their production costs as a result of IPM support from EQ. However, these interview statements diverge from data from the farmer survey answers, where around 60 percent of cocoa farmer IPM beneficiaries stated seeing increased production costs, and only six percent stated their costs decreased. Overall 74 percent of recipients of 32% 37% 6% 12% 14% IPM Beneficiaries, in % (n=190), by Value Chain Avocado Cocoa Pineapple Greenhouse vegetable Asian Vegetable “The Farmer Field School training was excellent; our farmer association members who participated in learning better pruning practices, pest control and soil management were able to double their tree yields in as little as a year.” -Representative from a fruit tree grower association in the DR. 66 IPM support indicated that their production costs had increased, and only 9 percent saw their costs decrease as a result of this type of EQ support. This trend of increasing production costs associated with increasing production volume is a common theme amongst the farmer and firm survey respondents and is representative of farms and firms that are in the growth stage. The same is the case for IPM beneficiaries, and as such, production costs should not be considered independent of production volume. See Figure 36 below for a value chain wise breakdown of IPM beneficiary farmer perception on their overall production costs. Figure 36: IPM Farmer Beneficiaries' Perception of Impact on Production Costs as a Result of EQ Support IPM Impact on beneficiary yields. The majority of IPM beneficiaries saw increased yields (88 percent). Around 8 percent reported no change to their yields while only 3 percent reported decreased sales. Table 23 below shows the range of yield increase for IPM beneficiaries as a result of EQ support. Table 23: Percentage Increase in Yields for IPM Beneficiaries By what percent has your yield increased with EQ support? n=167 Less than 10% 10% 20% 30% 40% 50% Don't know # of IPM beneficiaries with Increased Yield 8 32 56 35 10 19 7 % of IPM beneficiaries with Increased Yield 5% 19% 34% 21% 6% 11% 4% IPM Impact on beneficiary access to markets. The evaluation team did not find a relationship between IPM and access to markets since access to markets was not one of the outcomes measured by the farmer survey. Around 98 percent of IPM beneficiaries reported that they are either very likely or likely to continue to implement the improved practices and IPM trainings received from EQ. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Avocado Cocoa Pineapple Greenhouse Veg Asian Veg Sampled IPM Farmer Beneficiaries (in %) Perception of Impact of EQ Support on Production Costs Don't know Decreased No change Increased 67 Conclusion. Farmers who participated in farmer field schools reported that they increased their technical knowledge and capacity for improved product quality and increased yields. Increases in the volume of product generated were, however, associated with increased production costs, as noted earlier. Unfortunately, net farm income, which is a central measure of organizational effectiveness, was not an EQ performance indicator nor was it reported on by other data collection methods this study used. As indicated by farmers who received IPM related training from EQ, IPM training resulted in the adoption of production techniques that led not only to reduced application of pesticides, increased yields, but also to increased production costs. Impact on net farm income was not captured by EQ indicators. Study data did not provide an answer on the impact of IPM on access to markets. Recommendation. FFS was an effective knowledge and technology transfer methodology; the Ministry of Agriculture should make every effort to carry on and expand this on-farm work, with its own or donor resources. In addition, future projects should include a measure of net farm income change as an indicator of the effectiveness of activities/interventions being supported. Given the importance of reducing on-farm pesticide use and the resulting reduction of pesticide residues on Dominican agricultural exports, it is essential that the GoDR/Ministry of Agriculture make every effort to carry on the well-received training and demonstrations in IPM techniques that EQ introduced. As noted above (FFS), future projects should include a measure of net farm income change as an indicator of the effectiveness of activities/interventions being supported. A1.5 Has the revitalization of cocoa and avocado farms led to significant increases in overall production and yields? Findings: Around 43 percent of farmer survey respondents (144) had implemented a revitalization/renovation program on their farm of whom 70 percent were cocoa farmers, and 30 percent avocado farmers. Close to 90 percent of avocado growers and 60 percent of cocoa farmers reported a significant increase in overall productivity on their farm, while a smaller proportion (12 percent of avocado growers, and 36 percent of cocoa farmers) stated that it was too soon to know. Four percent of cocoa farmers reported no increase in farm productivity as a result of cocoa renovation/revitalization on their farm. Figure 37 below provides a value chain-wise disaggregation of EQ-supported renovated farms. Figure 37: Renovation and Its Effect on Overall Farm Productivity 88% 12% 0% 60% 36% 4% 0% 20% 40% 60% 80% 100% Yes Too Soon to Know No Has Renovation Led to Significant Increase in Overall Productivity on your Farm? (n=144) Avocado Cocoa 68 Conclusion: The cocoa producers who participated early have already realized increase in productivity and yield. It is unclear from the data as to whether revitalization/renovation (planting new seedlings) had a definitive additive effect on production and yields, when compared to other farmers who participated in EQ training activities but did not replant new seedlings. Recommendation: The Ministry of Agriculture should carry on the importation and testing of improved cocoa germplasm while the cocoa and chocolate industry (including producer associations) need to develop support for the expansion of cocoa seedling production for producers. It should be noted that attention must be taken to ensure that introduced cocoa varieties that may carry resistance to pests and diseases do not negatively impact the cocoa/chocolate flavor profiles that are unique to local Dominican (“criollo”) varieties. A1.6 Has the implementation of recommendations by the organizational capacity index led to significant performance improvements in the organizations? Findings. Between 2016 and September 2019, EQ conducted organizational capacity assessments with Dominican producer organizations and provided them with a range of organizational capacity support activities. Those activities included organizational strengthening plan development assistance; business plan assistance; sustainable business practices training; a savings and loan activity; and access to farmer field schools association workshops. To measure organizational capacity, IESC/EQ utilized a measurement tool called the "Organizational Competencies Index” (OCI)55, an objective measurement expressed in numbers on a preset scale, of the levels of development of a producer organization, including democracy in the management of the organization, the provision of services to partners, and the economic, financial and human development of the organization. This organizational capacity assistance activity was a pilot program that involved eight producer associations, cooperatives, and cluster organizations through which it reached larger numbers of individual producers. During this period, the EQ facilitated three rounds of OCI ratings. The rating instrument gathered information about the status of various organizational features and uses those ratings to form a composite score between 0 and 100. This process was overseen by EQ partner, the Center for Agriculture and Forestry. Development (CEDAF)56 . Initial OCI ratings, mostly in 2017, for the eight pilot producer associations ranged from 65 to 85; final ratings in 2019 ranged from 69 to 100. The average number net gain on these OCA ratings was 17.3 points, while the range from which this average emerged ran from a gain of 9 points to a gain of 28. The evaluation team also noted that the average number of months between an organization’s completion of its Organizational Strengthening Plan and its final rating was 21.7 months, though the range, depending on when each of the eight pilot organizations created their organizational strengthening plans ran from 17 to 30 months. The organizational capacity index (OCI) was administered to eight organizations by an EQ program partner skilled at assessing organizational capacity. Of the eight firms approached to participate in this activity, the evaluation team received responses to this question from three of them. 55 https://www.scribd.com/document/275531643/Indice-ICO 56 http://www.cedaf.org.do/documents/OCI-Internacional.pdf 69 Two out of the three that answered this question reported that they have implemented the recommendations provided by the review. All three either strongly agreed or agreed that their firms now have technical strengths and improved services as a result of participating in the OCI review. The evaluation team’s review of these data also shows that all eight organizations in the pilot organizational strengthening program completed their plans, including a business plan program component, and a sustainable business practices training. In addition, six organizations participated in Farmer Field School activities and three engaged in a savings and loan activity. Overall, five of the pilot associations engaged in at least three organizational strengthening activities and three organizations engaged in four program sub-activities. Given the small scale of this pilot initiative, in organizational terms, these findings cannot be generalized to future or larger scale replications. One finding from evaluation team’s reanalysis of these data which may warrant attention as local EQ partners, including government, consider how to sustain aspects of the EQ program, is the absence of a relationship between the number of months during which organizations with completed organizational capacity strengthening plans in place engaged with this pilot program and their net gain in OCI scores. As Figure 38 shows, these pilot (small N) data do not show an association between the access to assistance period and organizational improvement gains. This could however change, with a larger sample, or a longer assistance period, with more complete engagement in all program activities. Conclusion. OCI’s recommendations appear to have triggered actions by organizations for which the OCI was administered. Actions taken, based on recommendations are a likely cause of performance improvements reported but other interventions in which the same organizations participated could also have contributed to those improvements. 30 22 22 22 22 22 17 17 9 16 19 28 16 21 20 9 0 5 10 15 20 25 30 0 5 10 15 20 25 30 35 Org 1 Org 2 Org 3 Org 4 Org 5 Org 6 Org 7 Org 8 Gain in OCI score Number of Months Organizations Figure 38: OCI Activity Metrics Months between Organization's OCI Plan & Net OCI Score Gain Months After OCI Plan Net OCI Score Gain 70 Recommendation. Activities with the objective of association strengthening would be ones that might be “handed off” to another donor (such as USAID, EU, IDB) for follow-on; it would be informative to continue to track organizational capacity and performance for the 8 organizations for several more years post-EQ. A1.9 Did improved post-harvest infrastructure contribute to an increase in the value added to postproduction agricultural products? Findings Farms. Around 86 percent of surveyed farmers either strongly agreed or agreed that EQ had helped improve the farm’s post-harvest equipment/facilities. As Figure 40 indicates, between 65 and 100 percent of farmers across the different value chains either agreed or strongly agreed that EQ support helped improve their farm’s post-harvest infrastructure. Around 25 percent of avocado farmers disagreed that EQ helped improve their farm’s post-harvest infrastructure. Figure 38: Sampled farmers' (in %) extent of agreement regarding EQ’s impact on their farm’s post￾harvest equipment/facility improvement Those farmers who either agreed or strongly agreed that EQ helped improve their farm’s post-harvest equipment/facilities were subsequently asked whether their agricultural products produced had value-add before leaving the farm, to which 62 percent responded in the affirmative. Figure 41 below shows that the vast majority of farmers (between 65 and 94 percent) across the different value chains (with the exception of cocoa and pineapple indicated that their agricultural products had value add before leaving their farm. Only 51 percent of cocoa farmers and 53 percent of pineapple agreed that their products had value add before leaving their farm. This could be due to cocoa and 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Strongly Agree Agree Neutral Disagree Strongly Disagree % of farmers Extent to which sampled farmers (in %) agree/disagree that EQ helped improve farm’s post-harvest equipment or facilities Asian Veg Greenhouse Veg Pineapple Cocoa Avocado “We are now able to identify phytosanitary problems before exports & avoid warnings notifications to the country... this has added value in the management of the production farm, until the delivery of our cocoa product” -EQ beneficiary 71 pineapple being commodities at farm level, i.e. cocoa value add starts with fermentation which is rare in DR at farm level. This is more so done by aggregators, or exporters. Pineapple is still mostly for the domestic market, so little or no processing or packaging yet occurs at farm level. Finally, it is not always clear what a farmer understands as "value added". For instance, with vegetables, producers may consider an export-quality tomato/eggplant, chili pepper, etc. is value added. The same could be the case for avocado producers. Figure 39: Percentage of farmers that have value added agricultural products prior to leaving their farm Firms. Of the 92 firms that responded to the questions related to post-harvest facilities, virtually all (91) either strongly agreed or agreed that EQ helped improve their firm’s post-harvest equipment facilities. Around 96 percent of these firms either strongly agreed or agreed that improved post-harvest infrastructure increased the value-add of their products. Conclusion. Farmers and firms surveyed, indicated that EQ support to improve post-harvest infrastructure in the form of equipment facilities, was effective in increasing value addition to post￾production agricultural products for firms and farms. It is important to note the value chain differences among respondents; for avocado and vegetable producers growing produce for the export market, EQ interventions that addressed product quality and marketability result in the highest positive responses to improvements in value addition before leaving the farm. With cocoa, most value add begins with bean fermentation which in the DR is rarely undertaken on-farm. Pineapple is still mostly produced for the domestic market, which has little or no premium for on-farm value add for product quality, packaging or processing. Recommendation. The positive return (value add) from the post-harvest equipment investment would indicate that private sector farms and firms will continue to invest in appropriate technologies that support their economic activities. As such, the GoDR should support these investments by disseminating knowledge of emerging technologies in the sector and also ensure that government policy supports the ability of farms and firms to access such technologies. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Avocado Cocoa Pineapple Greenhouse Veg Asian Veg % of farmers Do the agricultural products produced on your farm have any value-added before leaving the farm? Yes No 72 A1.10 Has the support for the Ministry of Agriculture in monitoring and trapping Medfly and Tuta Absoluta led to notable commercial impacts for the private sector (e.g., sales/exports) and/or the ministry's disposition to continue to apply such mitigation/preventive measures after program close￾out (sustainability)? Findings. The EQ program provided significant monetary and planning and operational resources to the Ministry of Agriculture (MinAg) in DR to eradicate the Mediterranean fruit fly (Medfly) infestation in the DR. The evidence of relevance of the work done by EQ is in how IESC, in concert with the DR’s MinAg and USDA￾Animal and Plant Health Inspection Service (APHIS) worked to surveil and eradicate the Medfly infestation in a timely manner. This tripartite collaboration helped pool resources and set processes in place to curtail the problem, which was affecting exports of DR agricultural products, particularly avocados, tomatoes, chili peppers and bell peppers, for ten months (March 2015 to January 2016). As a result of this joint effort, medfly was declared eradicated from the DR in July 201757, consequently reopening DR’s export markets. The Ministry of Agriculture indicated to the evaluation team its intention to sustain the surveillance steps and actions for Medfly and Tuta absoluta, if national resources permit. Respondents also reported that EQ strengthened the capacity of the Department of Plant Health and the National Phytosanitary Surveillance System, thereby improving plant health, increasing production, and enhancing export opportunities in the country. EQ's (and by extension USDA’s) flexibility in supporting the Ministry of Agriculture’s identified needs for personnel training, logistical support and technical expertise was noted by numerous Ministry officials in key informant interviews and credited with resolving a potentially huge crisis for produce exports to the U.S. in a relatively short period of time. Conclusion. EQ support for the Ministry of Agriculture was critical to their surveillance and eradication efforts for Medfly. The significance of the declaration of Medfly eradication’s commercial impact was the reopening of the US market to Dominican fresh produce. The Ministry of Agriculture indicated to the evaluation team its intention to continue the surveillance efforts for Medfly and Tuta absoluta, however subject to national resources, i.e. budget. Recommendation. Surveillance of Medfly, Tuta absoluta, and other pests that can potentially shut down Dominican exports must continue to avoid the catastrophic economic impacts that can result from their detection; the Ministry of Agriculture must ensure that sufficient resources are mobilized for field monitoring and reporting. USDA/APHIS can and should continue to support the GoDR in these efforts. Programming flexibility enabled EQ and USDA to quickly respond to the urgent needs of the Ministry of 57 USDA FFPr EQ Program, Semi Annual Progress Report #3. “EQ technical & logistical support in trapping, monitoring and eventually eradicating Medfly directly benefited Dominican producers and exporters by re￾opening the U.S. market” Government of DR, Ministry of Agriculture Specialist FFPr Learning Agenda Priority Question #5: What is the minimum level of national or local government support for a program that is necessary for that program to be effective in pursuing trade and market goals? Government buy-in and continued support is critical in sustaining the wins from project implementation beyond LOP. The eradication of medfly was a successful endeavor because of the concerted efforts of EQ, GoDR and other stakeholders, Which helped reopen trade from the DR. 73 Agriculture; such flexibility should be part of future project design and U.S. Government (USG) programming. Activity 2: Capacity Building: Trade Associations This subsection provides the evaluation team’s answers to questions about the effects of EQ program activities designed to benefit trade associations. A2.1 What impact has establishing market linkages through participation in trade shows, trade missions, market promotion, and market studies had on expanding market access and increasing sales? And A2.2 Due to the pandemic some trade fairs were virtual. What was the impact of participation on virtual trade fairs? Findings. EQ’s trade show activities have helped open DR firms to a previously untapped foreign market. Respondents cited that EQ’s introduction of trade shows opened their firms to new clients and opportunities they didn’t have previously, giving them exposure to international marketing, and their products new visibility. Around 71 percent of the firm survey respondents had attended EQ led trade shows, with most of the participants representing the Asian vegetables (35 percent) and greenhouse vegetable value chain (30 percent), while 40 percent of the firm leaders from the mini survey worked in the cocoa value chains. Figure 42 below shows the value chain-wise distribution of firm respondents (represented in percentage), who participated in trade shows, and trade missions with EQ support and their perception on whether participation in these trade events helped improve their market knowledge and market access. As is evident from the figure, the vast majority (between 93-100 percent) of firms that participated agreed that the had improved the knowledge of trade and market access for their main crop. Respondents indicated increased ability to identify new markets and market requirements, from participating in EQ’s trade show and trade mission activities. USDA FFPr Learning Agenda: Trade Expansion and Agricultural Market Development Priority Question #23-What models of communication from buyers on requirements in terms of quality, standards, and aesthetics will support better partnerships with producers, suppliers, traders, and processors? As indicated across the different lines of inquiry, trade shows can play a critical role in facilitating dialogue regarding product quality standards, between buyers and sellers and may help support new partnerships through increased knowledge of market requirements and access. 74 Figure 40: Sampled Firms’ Perception of Trade Show Participation’s Effect on Market Access & Trade Knowledge Majority of the respondents on mini survey for firm leaders (70 percent) also reported this activity to be relevant and that it helped increase their firms market knowledge, market access, and exports. Virtual trade fairs drew fewer participants than in-person events with the impact to market knowledge, market access, and exports were reduced. Respondents across multiple data sources reported that virtual trade show experiences are not a great substitute for face-to-face encounters with potential clients. However, some key informants indicated that virtual trade fairs led to the development of assistance to firms in the form of virtual trade booths and website development support. Conclusion: This activity saw high participation among firms, with vast majority reporting an increase in market knowledge, increase in market access, and an increase in exports. This activity had its intended impact on expanding market access and exports for firms. The EQ program was able to adapt to changing circumstances and deliver value to Dominican exporters. Recommendation: To sustain the positive impacts that were reported from participants in these export promotion activities, national entities, notably ProDominicana, must step up to carry on and expand this work. Although there is no definitive evidence that virtual trade fairs were an effective substitute for in￾person events, the fact that there were EQ beneficiaries who indicated they were able to make or maintain contacts with customers warrants further research into how to incorporate virtual activities and increased on-line presence (website development, use of virtual trade booths) into market development activities. A2.3 Did program interventions have an impact in increasing the value and volume of exports? Has increasing the value and volume of exports altered the scale, composition or structure of value chains? Findings Impact on value and volume of exports: The EQ Program doesn’t explicitly monitor beneficiary exports but rather beneficiary sales (Result indicator #7/FFPr 13) overall. Findings from multiple data sources in 100% 93% 100% 96% 97% 0% 7% 0% 4% 3% 0% 20% 40% 60% 80% 100% Avocado Cocoa Pineapple Greenhouse Veg Asian Veg. % of firms Did participation in trade shows increase your knowledge of trade & market access of your main crop? Yes No 75 this evaluation converged to indicate EQ interventions have increased their exports58 , with key activities cited being EQ’s certification program, and trade shows and trade mission activity. Figure 43 below shows a value chain breakdown of trade show participants’ perception of changes to their export as a result of participating trade shows supported by EQ. Between 93 percent and 100 percent of firms that their exports had increased as a result of participating in trade shows, and trade and export missions. Key Informant Interviews confirmed these results as a large majority of producers and firms indicated that participation in trade shows had introduced them to new markets and also increased the knowledge among exporters of specific product quality standards that different markets required. Similarly, many exporting firms and some integrated producers (farm to packing house to export) indicated that EQ training in and adoption of certification mechanisms (organic, GlobalG.A.P., etc.) resulted in access to new markets that require adherence to strict production and export standards in order to access those markets. Figure 41: Trade Event Participation's Effect on Firm Exports A majority of key informants also indicated that as increased exports drove up demand, this also altered the scale and composition of the value chain with the entry of new and smaller producers into the value chain. Conclusion. EQ interventions had a positive effect on exports and value chains through its various interventions such as the market-driven certification program, and trade shows and trade mission activity. 58 Note that sales data (value and volume) for respondents captures only discrete data for 2019 and 2020 annual sales to better understand whether COVID had significant impact on sales. This sales data however may not be linked with exports for survey questions asked in relation to A.2.3, since it only captures two years’ data points for sales. Note also that the only measure of sales by the EQ program relate to FFPr 13 and the indicator is only capturing export sales. 100% 93% 100% 96% 97% 0% 7% 0% 4% 3% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Avocado Cocoa Pineapple Greenhouse Veg Asian Veg % of sampled firms Did participation in trade events increase your firm’s exports of main crop? Yes No 76 Recommendations. Although increased exports cannot be directly attributable to EQ program interventions, survey results suggest that a multi-dimensional approach has had a positive impact and should be considered in future programming. A2.4 and A2.5 Has the introduction of new export products and the development of new markets altered the scale, composition or structure of value chains? Findings. Multiple key informants reported that the introduction of new export products have increased the scale and composition of value chains, primarily with the introduction of newer actors, especially smaller entrants into the value chain who have benefited from the opportunities afforded by contract production. Avocado value chain actors specifically identified tropical avocado when discussing new export products. This new product positively impacted the scale of the value chain in the form of expanded production. Over 80 percent of firm leadership who responded to the survey directed specifically to them indicated that new markets developed with EQ program support positively affected the scale of the commodity value chain, while 75 percent stated new markets developed through EQ positively altered the structure. Key informants in the avocado value chain reported new markets created for tropical avocado positively altering the scale of that market, through market expansion. Key informants in cocoa value chain stated that with the increased awareness for Dominican fine chocolate, the cocoa sector with exports of around $200 million per year can be further enhanced. As such, this evaluation has found convergence between data sources regarding the development of new markets positively altering the scale of value chains. Conclusion. Based on interviews with key informants, and EQ activity results, it is likely that introduction of new export products in certain value chains, like avocado, had a positive effect on the scale of that value chain. As evidenced by EQ’s support for a new product such as tropical avocado, such support, although complex, should continue to be supported - the groundwork done by EQ on the avocado value chain has been handed off to ADOEXPO for continuity. Key informants also stated that the development of new markets can positively affect the scale and structure of value chains. Future programming by the GoDR, donors like USDA and the private sector should note the link that EQ demonstrated between market development and growth as well as changes in value chains, and continue to provide support for new market development for Dominican products. A2.7 How has training beneficiaries to identify market trends and requirements impacted their participation in export markets and their exports sales? and A2.9 To what extent did improved marketing and branding of agricultural products contribute to an increase in access to markets to sell agricultural products? Findings. Both firm leaders and firm workers who responded to evaluation surveys indicated that EQ program trainings, and certification program, and their participation in trade shows led to increased knowledge of markets, market trends, and market requirements (See A.1.1, A.1.3, A.1.3 above.) This enhanced knowledge helped increase their participation in export markets and their exports. Participation in the branding activity was moderate at 23 percent of the total firms surveyed. Participation varied considerably by value chain ranging from 0% in pineapples to around 43 percent in Asian vegetables. All of the firms that received branding and marketing support from the EQ program reported that the technical assistance they were provided benefited their firm. Of these participants, around 91 percent realized an increase in their access to markets, and exports. Some key informants in the pineapple sector reported that the improvements they have seen in their access to markets through 77 better branding and marketing of their products is motivating them to continue their work, while other key informants in the cocoa sector remarked that initiatives like the National Chocolate Competition and Festival have helped showcase Dominican chocolate and build the brand for fine Dominican chocolate59 . Recommendation. Training in identifying export market trends and increasing knowledge and understanding export market requirements should be carried on by local institutions, most notably ADOEXPO. Resources to support branding and marketing activities of agricultural products (whether for domestic or export markets) must be taken over by the relevant commodity associations/organizations in order to ensure sustainability of these activities. A2.8 Has training beneficiaries in financial and accounting management and ecommerce led to organizational improvements and improved commercial performance (sales, exports)? Findings. EQ program coordinated virtual technical assistance during covid to lead training for exporters and growers on market trends and requirements for dried fruits, finance and accounting management, and e-commerce and digital marketing60 . Financial management and accounting management training are also covered under EQ’s farm management training. While the participation61 in the farm management training was moderate at 48 percent (161 out of the farmer survey sample), the evaluation team did not find an association between farm management training and improved commercial performance in the form of sales or exports, based on respondent feedback and EQ program documents. Conclusion. None Recommendations. None A2.10 Has the implementation of initiatives to ensure inclusive growth and diversity within the export sector, for e.g. through the Young Exporters Network and Networking for Women in Trade Leadership, led to an enhanced participation of youth and women in private and/or public organizations focused on trade, and/or greater recognition of their contributions at such enterprises?62 59 EQ Program Semi Annual Progress Reports. 60 Dominican Republic Exporting Quality Program, Semiannual Repot No 11 61 Surveys and KIIs failed to gather specific input for A.2.9 from trade associations. 62 Revised as agreed with IESC prior to data collection 78 Findings. Participation of women in this sector has historically been low. The Young Exporters Network and the Networking for Women in Trade Leadership Initiative activities appear to have increased women and youth engagement in EQ’s value chains. Multiple data sources reported that the Young Exporters Network and Networking for Women in Trade Leadership programs helped foster a culture of women entrepreneurship and engagement with women and youth in the export sector. Around 70 percent of firm leaders who participated or worked with these initiatives indicated their effectiveness in increasing market access. Around 42 percent of these respondents attributed their increased market access and increase in exports to the Networking for Women in Trade Leadership initiative. Evaluation team found no evidence with respect to the recognition to their contribution by others. Conclusion. Interventions to support networking in women and youth groups had positive social and commercial effects. Recommendation. Support of these two important organizations should continue post-EQ. ADOEXPO has indicated it will continue to support the Women in Trade Leadership group which is a positive indication of the institutionalization of this effort that will contribute to its sustained impact. The Young Exporters Network rightfully deserves similar support through the TraSa Project and its associated interventions. A2.11 Has the promotion of cocoa and chocolate through activities like the National Chocolate Competition and Festival lead to a discernible impact on the chocolate industry and exports? Findings. Data sources, especially through key informant interviews with sector actors indicated there is growing recognition of Dominican ‘fine’ organic chocolate in the domestic market, but no discernible impact to exports was identified. Around one-third of key informants in the cocoa value chain participated in trade shows but no clear impact on ‘chocolate’ exports. Respondents reported that efforts by EQ to bring international experts to evaluate Dominican chocolate and train Dominican chocolate experts resulted in a group of local experts with a common language and skills to assess Dominican chocolate. One respondent replied to this question in this way: “The jury is still out on impacts; there are a large number of small bean-to-bar chocolate producers trying to figure out how to access the large tourist industry in the DR and also reach niche markets, especially in Europe.” According to an industry expert, “The promotion of cocoa and chocolate through activities like the National Chocolate Competition and Festival are solid support for new producers of chocolate, but a follow-up strategy is needed and there is a need to institutionalize the programs to guarantee sustainability." Conclusion. Attention and recognition, especially domestically, of Dominican fine chocolate is evident, and evidence exists of an expanding artisanal chocolate industry that benefits from tourism. EQ interventions in the cocoa/chocolate sector, particularly the National Chocolate Competition and EQ provided ASOPROPIMOPLA, a pineapple producer organization formed by young Dominican entrepreneurs, the opportunity to develop new customer contacts and marketing channels by participating in several international trade fairs that resulted in the DR’s first pineapple exports to Israel. After four years of EQ support and collaboration, the firm is one of the DR’s largest pineapple exporters with customers in Russia, Spain, Italy, France and Sweden. “The EQ program has been our commercialization ally since day one,” according to the firm’s General Manager. 79 Festival, seem to have contributed to a raised awareness of Dominican chocolate quality in the domestic market, but there is no definitive evidence yet of overall improved chocolate quality nor of any impact on export sales of cocoa or chocolate products. The impact of the competition and festival will take years to come to fruition and will likely be successful only if sustained. Small-scale (artisanal) chocolate producers indicated in evaluation surveys and interviews that EQ support with process improvement and equipment was key to their growth. Recommendation. In terms of sustainability planning of the National Chocolate Competition and Festival, IESC should target and partner with local chocolate companies to sustain this festival beyond EQ. In addition, given the large and growing tourism sector in the DR as well as growing artisanal (“bean￾to-bar") chocolate sector, there is potential synergy in linking these two industries to make the DR a "chocolate destination”; such efforts should be supported by the GoDR and the private sector. Finally, the chocolate industry itself must come together and take the leading role to promote Dominican fine chocolates in the world market. Activity 3: Cold Chain Improvement A3.1 To what extent did key players – logistics and export operations actors who transit cargo, like Cold Chain Working Group members - recognize and comply with the program’s cold chain pilot program guidelines and use the related manual? Findings. Participation of the firm sample in EQ’s cold chain activities was fairly low, with only six percent of people surveyed in firms having received any cold chain support. Participation in cold chain activities was evenly distributed among packinghouses, processors, exporters, and government. A small proportion of the mini survey respondents (n=4) were familiar with and worked on EQ’s cold chain pilot program. All 100 percent of those respondents indicated compliance with cold chain pilot program’s guidelines. Some key informants reported that EQ’s cold chain pilot program may be followed up by the USDA FFPr TraSa Project, currently being implemented by IESC in the DR. Conclusion. EQ program’s development and dissemination of the cold chain pilot program were effective and led to compliance Recommendation. Ultimately, the private sector should be responsible for carrying on these activities. Working with private sector actors, TraSa interventions should be aimed at the institutionalization of these efforts. A3.2 What was the impact of the World Food Logistics Cold Chain Institute training? And A3.3 Did cold chain interventions have an effect on other sectors? If so, to what extent? Findings. Only a small number of firms in the sample (26 percent, n=36) had participated in EQ’s cold chain activity concentrated in greenhouse vegetables and avocados, and Asian vegetables. See figure 44 below for the participant value chain breakdown. 80 Figure 42: Cold Chain Event/Training Participants in Firm Sample, by Value Chain Figure 45 below shows firm survey respondents (as a percentage) who attended cold chain events and their perception of those event’s impact on their knowledge of cold chain as it relates to the main crop they process/export. Most participants who attended cold chain training and events stated that their knowledge of cold chain as it relates to the main crop they process/export improved. Only eight percent of greenhouse and Asian vegetable processors stated no effect was seen from attending these events. Figure 43: Cold Chain Event/Training Participants Perception of its Effect on their Knowledge of Cold Chain While still largely positive, the percentage of firms that stated improvements in post-harvest techniques/practices from having attended cold chain events, ranged between 62 percent of firms that work with Asian vegetables (n=8), 67 percent of cocoa firms (n=2), 83 percent of avocado firms (n=5). Both pineapple firms that attended cold chain events and training stated applying improved post￾harvest techniques for their crop. 17% 8% 6% 33% 36% Cold Chain Events/Training Participanting Firms (in %), by Value Chain Avocado Cocoa Pineapple Greenhouse Veg Asian Veg 0% 20% 40% 60% 80% 100% Avocado Cocoa Pineapple Greenhouse Veg Asian Veg % of firms Did attendance in cold chain evens/training increase your knowledge of cold chain for your firm's main crop? No Effect Improved 81 Figure 44: Cold Chain Event/Training Participants Perception of its Effect on Post-Harvest Techniques in Sampled Firms (in %) In terms of this training’s sustainability, some key informants in cold chain indicated the continuation of chain training, and investment in cold chain. EQ’s support in drafting the national cold chain strategy may have effects on other sectors, going forward. The evaluation team however did not discover any effects on other sectors. The two most important areas of impact of EQ’'s cold chain interventions, according to key informant interviews, were 1) improved knowledge of and investments in cooling/refrigeration of perishables at packing houses as soon as possible post-harvest, and 2) maintenance of a cold chain through transportation and inspection at ports and on to export markets. Conclusion. Survey and key informants felt that World Food Logistics cold chain institute’s training was effective in contributing to improved product quality. As noted above, program beneficiaries in cold chain activities gave EQ high marks for the cold chain interventions and indicated positive impacts on their businesses, but the small scale of the activity likely did not result in significant impacts on other sectors.Recommendation. The potential high positive impact of these types of training and knowledge transfer activities on the country's fresh produce exports warrants continued support by the GoDR working in conjunction with the private sector. These types of interventions in cold chain strengthening should be supported in the future by Government and the private sector. In addition, the work done under EQ with the GoDR to develop a national cold chain strategy should be continued under USDA FFPr TraSa project or similar programming. Activity 4: In-kind Grants: Equipment A4.1 What impact did in-kind equipment grants have on productivity, e.g. enhanced production volumes? and A4.3 Did increased productivity from improved equipment increase domestic demand for raw materials? If so, what was that estimated increase? 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Avocado Cocoa Pineapple Greehouse Veg Asian Veg % of firms Did attendance in cold chain evens/training improve your post-harvest techniques/practices for your main crop? No Effect Improved 82 Findings. Around 81 percent of the farmer in-kind equipment recipients (n=64) in the sample were cocoa farmers (i.e. n=52), being the largest proportion of farm in-kind equipment recipients. Key informants attribute this to the small and artisan chocolatiers of the country who were very motivated to apply for in-kind equipment grants in order to compete with the international market. One chocolatier grant recipient went to the extent of moving from his own premises to a rented space to be able to install the requested equipment. Eleven percent of the pineapple farmers sampled had also received in-kind equipment from EQ. The largest proportion of firms that received in-kind equipment from EQ worked with greenhouse vegetables (42 percent.) Findings indicate that while only moderate participation by farmers (19 percent, n=64) and firms (11 percent, n=12) was seen for this activity in the quantitative sample, a high proportion of participants (i.e. 83 percent of firms and 73 percent of farmers) reported increased productivity. Around 71 percent of key informants from trade associations had received in-kind equipment from EQ, a majority of whom were in the cocoa value chain. Around two-thirds of these cocoa trade association members reported increased volume and immense value in receiving pruning equipment from EQ. All firm in-kind equipment recipients that reported increased productivity also reported an increase in raw material purchases, while 91 percent of farm in-kind equipment recipients that reported increased productivity also indicated increased raw material purchases. According to key informant interviews, farmers mostly increased their purchases of fertilizers, bio￾pesticides, and tools. For firms, especially small-scale processors, their increased raw materials purchases consisted mostly of farm commodities that they then processed or packaged. Out of the 59 in-kind farmer recipients who provided sales volume data, 72 percent of them stated that their production volume had increased with EQ-supported in-kind equipment grants, and 25 percent stated no change in their production volumes. I4DI defines smallholder farmers in the sample as those who produced 0 – 5 MT of product in 2019. Smallholders who responded that in-kind equipment donation increased their production volumes reported a 52 MT increase to their production performance. Sampled farmers who sold between 5 – 20 MT of product in 2019 and reported that in￾kind equipment helped increase production volume reported a 180 MT increase to production. Sampled farmers who sold upwards of 20 MT in 2019 and who stated in-kind equipment helped increase production volumes reported an increase of 247 MT in production. The 25% of farmers who reported no change to production after receiving an in-kind equipment donation reported a total 610 MT of production between them. The EQ Program did not collect baseline or midline productivity data from in-kind grant recipients, and hence a comparison cannot be made between three distinct time periods of the program. Conclusion. In-kind equipment grants have been positively received by over 70 percent of farmer recipients who state increased production volumes. Program’s investment in in-kind tools and equipment not only leveraged firm resources, it also yielded benefits to the economy through increased purchases of raw materials and inputs. FFPr Learning Agenda Priority Question #18: What types of market linkages help reduce the obstacles in value chains that hinder agricultural actors from benefitting from existing infrastructural facilities? In-kind equipment grants have been found to be quite helpful in matching agricultural actors who have limited infrastructure and resources, with the resources they need to be productive players in the domestic and international markets. This is evidenced by the various cocoa value chain beneficiaries referenced in the narrative. 83 Recommendations. To sustain activities such as these that have an equipment grants component, it is recommended that farmer associations, commodity groups, firms, and Government will have to engage to a greater extent, utilizing EQ’s success. The GoDR should consider policies (tax, investment, import) that support the identification, demonstration and access to equipment, tools and technologies that improve production, post-harvest handling and processing of agricultural commodities, the benefit of which was demonstrated by EQ. A4.2 What impact did in-kind equipment grants have on product quality as experienced by the program beneficiaries who used equipment donated by the program? Findings. The in-kind equipment grants activity provided farmers and firms with a wide array of equipment intended to help increase productivity, enhance efficiencies, and improve product quality. Across all data sources, including mini-survey, KIIs, and quantitative surveys, over two-thirds of respondents reported increased product quality. See Figure 47 below that details in-kind equipment recipients indicating product quality improvements and productivity increases due to EQ support. Figure 45: In-Kind Recipients (represented as a %) Improvements in Product Quality and Productivity Conclusions. A large proportion (>70 percent) of sampled in-kind recipients stated seeing improved product quality and increased productivity. Key informant interviews indicated that farmers viewed improved product quality as more produce of sellable quality, while firms indicated lowered rejection of product by buyers. However, since EQ does not collect product quality or productivity data at the activity level, comparisons of changes over time cannot be established by this evaluation. Activity 5: Public Information Campaign: Disperse Improved Market Information A5.1 What degree of coverage and outreach was achieved through the Consumer Food Safety Campaign? and A5.2 What were the discernable behavioral changes that may have occurred in particular population segments as a result? Findings. EQ utilized multiple media sources including national TV and radio outlets, as well as social media to reach consumers with food safety campaign messaging. EQ program documents report that 92% 79% 83% 73% 0% 20% 40% 60% 80% 100% Farmers (n=64) Firms (n=11) In-Kind Equipment Recipients in Sample: Improved Product Quality & Increased Productivity improved quality increased productivity 84 5.7 million consumers reached through the Consumer Food Safety Campaign. This shows that the program significantly exceeded EQ target of 1.5 million for this is activity (Activity indicator 5.1.) Key informants in trade associations were familiar with this campaign and all 100 percent of them asserted that it reached consumers with very targeted messaging. However, around 83 percent of the key informants in the cocoa value chain interviewed did not know about this campaign. While evidence suggests EQ managed to secure widespread coverage outreach of the Consumer Food Safety Campaign, no discernible behavior changes were identified by this evaluation team. Conclusions. The methods utilized under EQ were effective in delivering the consumer food safety information to the public. Without undertaking a follow-up consumer survey, it is impossible to determine the impact on consumer behavior. Recommendation. Consumer awareness campaigns should include a follow up element (including a consumer survey) to determine their effectiveness. A follow-up survey should be undertaken to determine the effectiveness of the public awareness campaign. Evaluation Questions linked to OECD/DAC Criteria Relevance R.1. To what extent did the Exporting Quality project design address the core issues of target beneficiaries? and R.3 Was the project designed taking into account economic, cultural, and political contexts? Findings. Some of the core issues faced by beneficiaries working with avocados and pineapples included their low processing capacity, the long time period to get GlobalGAP certification without adequate technical assistance. For smallholder cocoa farmers, in order to increase on-farm value add, they needed to expand the wet fermentation process, and overall improvement of cocoa manufacturing, especially those that target the high-value sector. For greenhouse and Asian vegetables, their core issues revolved around many pest treatments that reduced the quality of the produce, increased production costs, and lowered competitiveness of the produce63 . More than 80 percent of respondents representing the entire spectrum of EQ beneficiaries, namely producers/farmers, people in firms and firm leadership representing processors, packinghouses, exporters, trade associations, and government, reported that EQ addressed the core issues of target beneficiaries. Some key informants indicated the EQ may have been too broad and not deep enough, i.e. trying to do too much for too many beneficiaries, with one interviewee calling it a “first come-first served” approach that was not strategic enough in proportioning out project resources. As mentioned previously, as the vast majority of respondents, directly responded that EQ was able to address their core issues, this indicates a high convergence across data sources and activities. Regarding project design, the EQ program did consider the underlying scenarios in the DR, as indicated by the large number of studies commissioned in the start-up phase of the program focused on gathering information related to aspects of the agricultural value chains. On the farmer field schools, one key informant reported that training was designed so that the economic status of producers was not limiting to participation, and culturally, there was a recognition that the technical methodologies had to be relevant to the prevailing conditions in Dominican agriculture, characterized by adults and limitations in reading and writing. The program design also seems to have adaptive management 63 USDA FFPr Exporting Quality Program, Midterm Evaluation, 2018 85 institutionalized into its programming. This is evident from the program’s rapid response to unexpected and unforeseen events such as Medfly infestation in the DR, and more recently with COVID-19. The program seems to have worked well with changing administrations in government. See I.2 below. Conclusions. Convergence in responses from the majority of respondents representing various beneficiary groups, responded that their core issues that target firm and farm beneficiaries perceived had been addressed by EQ. Program design well-reflects US government best practice for the design and implementation of development assistance projects. Recommendations. No recommendations R.2 How were existing relevant USDA and U.S. government activities leveraged? To what extent did the program align with the Dominican Republic’s agriculture and/or development investment strategy? With USDA and USG’s development goals? Findings. This program aligned in a limited capacity with USDA PROGANA program in adopting the farmer field school model and collaborated briefly with USAID FAST program implemented by USDA and Food and Drug Administration (FDA) on programming some FSMA readiness activities. Strong collaboration between EQ and USDA/APHIS was evident as part of follow-up activities after the Medfly outbreak. The work undertaken by EQ through its five activities were well aligned and contributed to Food for Progress Strategic Objective (SO) 1 – Increase Agricultural Productivity and Strategic Objective (SO) 2 – Expanded Trade of Agricultural Products. At a higher level, the EQ activities also contribute to overall FFPr DR’s Learning Agenda focus areas namely 1) Market Systems, 2) Quality of Standards, and 3) Risk and Uncertainty, as well as cross-cutting areas namely culture, demographics, and sustainability. Table 24: EQ Stakeholder Groups across the Different Sectors Sector Institution Government/Public Ministry of Agriculture and its departments: - Department of Production, Environmental Protection (DEPROBAP) - Food Safety Department (DIA) - Division of the National Phytosanitary Surveillance System - Plant Health Department - Cocoa Laboratory Pro Dominicana (formerly CEI-RD) CEDAF Ministry of Industry, Commerce and Mipymes (MICM) ProConsumidor Dominican Agribusiness Board (JAD) National Competitiveness Council (CNC) Punta Cana International Airport Dominican Airports Century XXI (AERODOM) “The EQ program is strongly aligned with the strategic objectives of the DR’s National Development Strategy that promotes value-added agricultural exports and envisions the country to be the logistics hub of the region.” -Government of DR, Ministry of Agriculture Representative 86 Sector Institution Private American Chamber of Commerce of the Dominican Republic (AmChamRD) Dominican Association of Exporters (ADOEXPO) Foundation for Social Assistance, Recovery, and Organic Management of Cocoa Trees (FUPAROCA) Dominican Association of Exporters of Asian Vegetables (ADEXVO) DP World Caucedo Haina International Terminals International Trade Center (ITC) Dominican Environmental Consortium (CAD) More than 110 packinghouses, processors, exporters benefited by the program University Biotechnology Laboratory of the Faculty of Agronomic and Veterinary Sciences (Universidad Autónoma de Santo Domingo) ISA University Conclusion. EQ program reflects the partnership aspirations of the US Government’s 2016 commitment to global development through its collaboration, with a) other US government programs, and b) the DR’s development policies and strategies that aim to promote a profitable and competitive agricultural sector in national and international markets. Recommendation. EQ’s model of collaboration with different stakeholders, including key functionaries in the government is worth replicating for future USDA programming. The commitment of GoDR resources necessary to sustain some EQ activities into the future should be a condition of continued USG support. Effectiveness EE.1 What were the major results of the project in achieving the goal and intended objectives? and EE.2 To what extent did Exporting Quality achieve the specific targets and results established? Findings. Virtually all interventions led by EQ yielded demonstrable effects on program and FFPr strategic objectives: increased productivity and increased exports. Some interventions, such as the renovation programs, networks for youth and women, were not mature enough to yield demonstrable effects. EQ has exceeded program targets on all but 2 indicators-See Table 11 under Section 3.2: Performance against EQ Program Performance Indicators for more details. Each of these activities and their results have been detailed in previous sections of this report. Conclusions. Program was well-designed and implemented to address program goals and objectives in light of the core challenges the target beneficiaries. EQ program exceeded life of program (LOP) targets for all but two indicators, which displays effective technical and operational management by program staff. EE.3 Which activity or combination of activities proved to be the most effective approach to achieve the project’s higher-level results, namely increased access to improved market information; improved capacity of key groups in the agriculture production sector; and increased leverage of private sector resources? 87 Findings and Conclusion. Based on findings from various data sources, review of program documentation, and contribution analysis, the main intervention that provided the most benefit to farmers and firms alike are EQ’s various training programs. Multiple survey respondents and key informants reported that they would like to receive more training due to the positive effects it has had on their operations. Table 26 additionally details a disaggregated view of the top activities and/or interventions based on beneficiary responses across various data sources. Table 26 also outlines the key outcomes of those activities, and the relevant results framework pathway taken by those activities to achieve FFPr strategic objectives (SOs) 1 and 2. See Annex C: Results Framework for more details. 88 25 Table 26: Top EQ Activities/Interventions, Outcomes, and Relation to FFPr Strategic Objectives Beneficiary Top EQ Activities/Interventions Most Positive Outcomes FFPr Strategic Objective Results64 FFPr Strategic Objective (SO) 1: Increased Agricultural Productivity Farmers/ Association Groups 1) Trainings: 1.a) Farmer Field Schools 1.b) Integrated Pest Management Training Increased knowledge of improved techniques for farm productivity FFPr 1.2.4: Increased knowledge by farmers of improved techniques & technologies 2) In-kind Grants: Equipment Increased grants (equipment and tools) to facilitate adoption of improved techniques FFPr 1.2: Increased use of agricultural techniques & technologies Firms 1) In-Kind Equipment Increased knowledge of improved techniques for firm productivity FFPr 1.2: Increased use of agricultural techniques & technologies 2) SPS, FSMA Training Increased knowledge of improved techniques for firm productivity FFPr 1.2: Increased use of agricultural techniques & technologies 3) Cold Chain Improvement Increased knowledge of improved techniques for firm productivity FFPr 1.2: Increased use of agricultural techniques & technologies FFPr Strategic Objective 2: Expanded trade of agricultural products (Domestic, Regional, and International) Farmers/ Associations Groups 1) Trainings: 1.a) Farmer Field Schools 1.b) Integrated Pest Management Training Improved pest management on farms; Improved market presence for products FFPr 2.1.1: Improved Quality of Post￾production ag products Firms 1) Trainings: FSMA, SPS Improved SPS and related standards FFPr 2.1.2: Increased Efficiency of Post￾Production Processes 2) Trade Shows & Trade Missions Increased knowledge of markets, market access, and exports FFPr 2.2.2: Improved linkages between buyers and sellers FFPr 2.2.1: Improved marketing of agricultural products 3) Market-Driven Certifications Improved knowledge of exports standards FFPr 2.1.1: Improved Quality of Post￾production ag products 4) Cold Chain Improvement Increased exports FFPr 2.1.1: Improved Quality of Post￾production ag products 64 See Annex C: Results Framework for more details 89 EE.4 To what extent has training and certification had an influence on improving the quality of post￾production agricultural products? How effective has participation in trade shows and buyer missions been to improve linkages between buyers and sellers? Findings. Data collected across different sources report improved quality of agricultural products from on-farm and post-harvest suite of trainings. Similarly, various certifications resulted in market-driven improvements in export product quality. For those firms and individuals who participated in trade shows and buyers mission proved effective in improving sellers’ knowledge of market requirements and resulted in improved linkages between buyers and sellers. See responses to evaluation questions A.2.1 and A.1.1 above for more details. Conclusions. EQ program was effective at transferring market-useful information across EQ value chains and to all actors at each stage of every EQ value chain. That information transfer resulted in improved export product quality. See A.2.1 and A.1.1 for more details. Given the wide scope and reach of interventions under EQ’s five activities, some interventions may have been perceived by beneficiaries as being too broad and not in-depth enough. Some key informants indicated that the program was trying to do too much for too many beneficiaries. A more strategic approach to allocating limited program resources would be to focus on interventions where most positive effects and outcomes have been reported and providing additional support and/or resources towards those demand-driven initiatives-see related point below. Efficiency EI.1 To what extent did the level of project resources lead to achievement of results? and EI.2 Could the same results be achieved with fewer resources? Findings. The responses from several different key informant interviews indicate a consensus that this program performed well with the resources allocated to it. For Activity 2, for example, taking the total amount spent on market access (Activity 2) against the Result Indicator 16, value of business transactions facilitated by the project by year 6 yielded 1:7.8 return, i.e. for every $1.00 of UDSA investment in market access, the beneficiaries experienced $7.80 in new transactions directly attributable to the intervention (e.g. trade show preparation and attendance). At the end of the project, Results Indicator had documented more than $14,4891,000 in direct transactions. Table 27 below provides a breakdown of the direct costs for activity implementation. Table 26: EQ Activity Budget (Direct Costs) EQ Activity/Expense Type Activity Cost (Direct) % Of Overall Activity Cost (Direct) Total Activities Direct Cost $ 8,073,289 - Activity 1: Capacity Building Producer Groups/Cooperatives $ 4,522,953 56% - Activity 2: Capacity Building: Trade Associations $ 1,863,470 23% 90 EQ Activity/Expense Type Activity Cost (Direct) % Of Overall Activity Cost (Direct) - Activity 3: Cold Chain Improvement $ 681,784 8% - Activity 4: In-Kind Grants: Equipment $ 573,009 7% - Activity 5: Public Information Campaign: Disperse improved market information $ 428,066 5% As shown above, more than half of the activity implementation direct costs went into Activity 1, followed by a little less than quarter of the activity budget towards implementing Activity 2 tasks and sub-activities. Regarding whether the same results could be achieved with fewer resources, none of the respondents across data sources suggested this, without adversely affecting the impact and reach of the program. Sustainability S.1 To what extent has Exporting Quality developed a) local ownership and b) sustainable partners? and S.2 What is the likelihood that the project benefits will endure over time after Exporting Quality ends? Findings. Regarding local ownership, this evaluation found strong local ownership among farm and firm beneficiaries (private sector) across the board as evidenced by high adoption rates of practices recommended by EQ interventions as well as the high percentage of beneficiaries that reported their commitment to continue those practices, where there is no need for an institutional partner for sustainability. Regarding the development of sustainable partners, a few public sector partners have been identified to sustain some activities for example, ADOEXPO’s role in sustaining the women and youth networks; CEDAF’s commitment to sustain the work of the farmer field schools and supporting firms with traceability systems; certified avocado production through Eurofresh Dominicana, Min of Ag’s extensionists, and the AGROFAMILIA65 Sierra de Neyba project; IPM activities through Pre-inspecciona and IPM Min of Ag’s programs and the TraSa Project. For other interventions, no clear sustainability partners were identified through the evaluation process. For the actions that they personally control, such as continuing the improved practices learnt from EQ support, beneficiary data sources indicate sustained action and associated benefits. For various interventions, the evaluation found that some had been internalized by government ministries, private sector, trade associations suggesting their possible continuation. However, a large proportion of government personnel indicated concerns about the government’s ability to carry on farmer training, pest surveillance, produce inspections, and monitoring, due to resource limitations. In addition, the regular activities of women and youth networks and member interest in these activities, as evidenced by mini-survey and KII responses, suggest their 65 AGROFAMILIA is an Agricultural Productivity program implemented by the Aliade Global Corporation with headquarters in Spain and global scope. Since April 2021 the program has assisted 15 years’ avocado producers in San Juan, Azua and Elías Piña provinces. 91 continuation. The evaluation team received a sustainability action plan from the EQ program team, that identified initial sustainability partners for Activities 1, 2, and 3. Conclusions. From data gathered by the evaluation, the team concluded that finding sustainable partners in the government is less likely, despite expressions of interest, because of uncertainty of available government resources. Recommendations. For interventions that require a sustainability partner, the process for identifying and solidifying partners needs to begin early in the program with the objective of institutionalizing activities, sub-activities and program elements to the extent they are feasible and appropriate. To better understand and foster prospects for sustainability, USDA should incorporate a requirement for activity￾level sustainability plans and exit strategies, and mid-term reviews of progress in relation to each of them, inclusive of modifications that may be needed to account for contextual changes. To facilitate a thorough review of sustainability plans and processes, USDA might consider organizing a sustainability/program close-out conference in the last six months of future programs. The objective of such an event would be to review sustainability planning, share lessons learned from program implementation, and engage Government and other local institutions in future program planning. Impact The term impact is used in multiple ways in evaluations. In this evaluation, impact is considered in terms of the types of results and effects the EQ program yielded; it is explored from a causality perspective and in the context of COVID 19, the possible effects of that factor on program outcomes are also examined. I.1 What are the immediate-, medium-, and long-term effects, intended and unintended, positive and negative, of the project after nearly six years of implementation? Findings. Some of the key effects that emerged from the findings of this evaluation are outlined below: Immediate-term effects: Medfly surveillance and eradication re-opened the US market; in-kind equipment grants quickly led to improvements in productivity, product quality and process efficiencies Medium-term effects: Farmer training that increases productivity; development of new markets for tropical avocado; introduction of SPS training and IPM modules that will reduce pesticide use & residues; introduction to trade fair participation leading to increased exports; certifications, FSMA￾readiness and traceability systems all contribute to increased market access and exports Long-term effects: Cold chain training and private sector infrastructure have changed the cold chain situation in the DR; fine chocolates industry will grow in volume and sophistication; increased networking among women and youth will lead to their increased participation in the ag economy. Regarding attribution of results, the term impact is also used in evaluations with reference to study approaches that seek to establish evidence of causality between programs and observed results. From the outset, the evaluation team described this evaluation as one in which questions about causality or attribution were not a focus. Nevertheless, the evaluation team did document and report occasions on which key informants and program beneficiaries causally linked the EQ program and changes or results they were describing verbally or in open ended survey questions. Key informants and beneficiaries both asserted that results they were familiar with stemmed from EQ program assistance. The evaluation 92 team also captured information about factors besides the EQ Program that may have contributed to results reported by this evaluation. Notably, in talking about topics of interest to the evaluation, improvement in productivity and exports, key informants did not on their own describe other programs or factors that might have affected outcomes of interest to EQ managers and supporters. No direct questions of this nature were included in the study’s KII protocol. Evaluation surveys included a question about sources of support beneficiaries may have received from sources other than the EQ Program. Forty-six percent of farm survey respondents and 12 percent of firm survey respondents indicated they had received support of some capacity from the DR Government. With respect to assistance from other foreign governments, beyond the US, 19 percent of farm survey respondents had received some assistance from another foreign government, while 25 percent of firm survey respondents said the same. These responses suggest that including questions of this sort in other project evaluations may provide information that is useful for making programming decisions. I.2 What was the overall impact of COVID-19 restrictions/protocols on project results? and I.3 How did the program adapt to the pandemic and to what extent were adaptations of activities successful in reaching beneficiaries?Findings. The DR economy shrank, and the tourist industry disappeared for a time, closing markets and severely restricting ag product sales for a while; this led to a significant slow￾down of EQ implementation; in a period of 3-4 months, economic activity began to rebound, and the EQ program was able to adapt its activities to new safety protocols. Not surprisingly, COVID-19 had significant negative impacts on farmers and firms in the DR. See figure 48 below which details the responses of farmers and people in firms regarding COVID’s impact on their operations. Some activities (such as training outdoors) were carried on but under strict safety protocols. Some activities were delayed, while others (trade fairs) were adapted to a virtual environment. Figure 46: Impact of COVID on Farm and Firm Operations I.5 The Exporting Quality Final Evaluation must also determine whether recommendations from the Exporting Quality Midterm Evaluation were incorporated into the project and if not, why not? 5% 31% 30% 29% 3% 2% 70% 26% 1% 3% 0% 10% 20% 30% 40% 50% 60% 70% 80% Extremely adversely Adversely No effect Positive Extremely positive Don't know Impact of COVID-19 on Operations: Farmers & Firms in Sample Firms (n=92 [excludes govt]) Farmers (n=332) 93 Findings. The final evaluation recommendations below in column 2 of Table 28, were guided by evaluation findings as well as EQ program documentation. Table 27: Final Evaluation Determination of Status of Midterm Recommendations Midterm Evaluator Recommendations Final Evaluation Determination The evaluator recommends continuing and following up with (i) the current training approach, (ii) support for getting internationally recognized certifications, (iii) the provision of grants to farmers, packers, and processors, as well as the enlarging of their scope, and (iv) the facilitation of study tours and exporter and buyer visits; The evaluator also recommends focusing more on (i) post-harvest handling techniques and methods; (ii) Integrated Pest Management (IPM) and sanitary￾related activities; and (iii) end-market activities; Based on I) EQ’s increasing of targets related to results and activity indicators for Activities 1 and 2, II) EQ having subsequently exceeded those targets (see Table 11: EQ Performance Indicator List, LOP Target, and Actuals Achieved), and III) the findings in Section 3.4 related to Activities 1 and 2, the evaluation team has determined that this midterm recommendation has been actioned upon by EQ For a few (mainly activity) indicators, including activities with members of trade associations, the evaluator recommends either (i) scaling down end of year 4 (Y4) cumulative targets (due to initial overestimation of the number of trade associations operating in the Dominican Republic [DR]); or, preferably, (ii) enlarging the purpose to include producer associations, cooperatives, and clusters, all of which would be able to trade produce among members; In lieu of scaling down targets, EQ Program adapted some activity indicators definitions: Some activity indicators were combined (such as AI#2.3 and #2.4) so as to consolidate the figures. AI#2.6’s indicator definition was updated to include produce association management, and private businesses. For a significant number of performance indicators (related primarily to training activities) the evaluator recommends scaling up the Y4 targets, as the program showed higher than expected results during the second half of Y2. This scaling up would more than compensate for the above scaling down of result indicator targets; All of these indicators have been scaled up based on the midterm recommendations and subsequently scaled up further as well. The evaluator recommends that the program perform some assessments specifically requested by beneficiaries participating during focus group discussions, including (i) expanding export of produce to the Caribbean islands, (ii) availability and cost of water treatment facilities, and (iii) facilitating resolution of business disputes over shipment overdue payments; The evaluation team received documentation that details the following actions taken on the midterm evaluation recommendation: i) IESC discussed the expanding exports assessment and gave it low priority. ii) IESC conducted an informal assessment about water chlorination, which informed EQ’s water chlorination donation activity iii) IESC conducted an informal assessment about facilitating resolution of business 94 Midterm Evaluator Recommendations Final Evaluation Determination disputes and developed a series of workshops around it. For four of the results indicators, the evaluator recommends lowering the end of Y4 cumulative targets, as the set targets appear unattainable based on current action delivery trends and achievements and types of beneficiaries potentially reachable RI#4, RI#5, RI#6 were scaled down, and subsequently scaled up again. RI#18 was not scaled down, and EQ has achieved the original LOP target of 90 percent (See table 11 on page 35-36) Evaluator recommends the program prepare an exit strategy assuring sustainability of activities after program completion. EQ has prepared a sustainability action plan document that identified sustainability partners for Activities 1,2,3, with some fully executed collaboration agreements in place to see to the sustainability of activities. Recommendations. EQ should focus on securing sustainability partnerships for activities that have been reported as having key benefits to EQ beneficiaries and stakeholders and contributing directly to EQ program goals and objectives, due to time constraints presented by an end of year closeout date. See Table 26 under Section 3.5 above for more details. 3.6 Broad Conclusions Overall, the EQ Program yielded mostly positive results based on multiple data sources. Strategic Objective Results on the EQ Program’s Results Framework. Gains in productivity and access to markets are important steppingstones for Dominican farmers and firms toward increased business revenues, which in turn, it is hoped, will increase farm and firm earnings from exports as well as domestic markets, and, in turn, improved incomes for owners and workers in these value chains. While the evaluation did not collect empirical data on all these transitions, evidence from the EQ performance indicators show sales gains for project beneficiaries in both value (Result indicator #7) where the EQ’s life of program (LOP) target was exceeded by at least $14 million and volume (Result indicator #8), where the LOP target was exceeded by at least 5,800 MT. Tracing gains from this program to the household income level or increases in firm and farm employment levels was beyond the scope of this study, but the strong positive picture drawn of this program from KIIs and open-ended survey responses suggest sustainable and enlarging benefits, with farm and firm respondents resolute in their commitment to maintain the certifications they have gained and the new production and business practices they have adopted. Section 3.3 of this report provides a detailed look at the findings and overall conclusions associated with the high-level results on EQ Program’s results framework as it relates to SO1 and SO2. Results for Key High-Level EQ Objectives. For farms this is apparent from evaluation findings on productivity, market access and exports, all of which were deemed to have realized gains by farm survey respondents: yields (82 percent), production volume (81 percent) and production quality (82 percent). Importantly, the EQ program’s performance indicators for farms demonstrate achievement of higher￾level objectives on the program’s Results Framework. Similar gains were seen for firms with high numbers of firm survey respondents reporting gains in market access (84 percent), production quality 95 (91 percent), and production volume (87 percent). These reported gains are impressive by themselves, showing as they do that the EQ program succeeded in the types of assistance and support envisioned across the five targeted value chains. In addition to reporting these gains, the evaluation team observed and reported that farmers and firms that had adopted new practices and standards as a function of their exposure to and work with EQ activities reported through survey questions that in addition realizing productivity gains, they experienced higher costs, mostly related to increased use of inputs. This is not unusual in business. To further understand this finding, the evaluation team used crosstabulations. These showed, for both farms and firms, an association between increased production volume and increased production costs – two hallmarks of a growing business. In this context, the evaluation team interpreted its findings about cost increases as more of a positive comment on the EQ program than negative one. Also, as might be expected, the evaluation team found that program benefits accrued most noticeably to those members of the EQ target groups who participated in the various EQ activities. Different levels of participation were observed across value chains and types of activities. This suggests that in multi￾activity projects, like EQ, close attention to participation and adoption rates and being flexible enough to try new approaches for ensuring both occur, are not only warranted but can serve as leading indicators for achieving the full measure of a program’s potential success. 3.7 General Recommendations • Close-out Workshops. USDA would benefit from organizing (with IESC) a series of EQ close-out workshops with beneficiary groups, aimed at gathering lessons learned, identifying sustainability challenges, and helping both Government and private sector actors in planning their paths forward; such an activity could also help to strengthen the foundation laid for the USDA Trade Safe (TraSa) project. • Action and Follow-Up on EQ Sustainability Plans. EQ has already identified sustainability partners in the private and public sector to carry forward program work after EQ close out. for several included in their programming the development of sustainability and/or contingency plans with each group of beneficiaries in order to map out how EQ program activities could be sustained after program close-out. • Collaboration. Active collaboration with EQ partners has helped the program identify long-term sustainability partners that may be able to carry forward EQ program work beyond program closeout. This collaborative model can be replicated in other USDA programming to increase local capacity and ownership of products, practices, and techniques. Other opportunities for collaboration with donors such as the Inter-American Development Bank (IDB/BID), should also be leveraged in order to maximize the impact of USDA and donor programming in the DR. • EQ Activities Informing Future USDA Programs/Projects. Programs that are informed by Exporting Quality Program may wish to experiment with approaches for achieving outcomes in various ways. An activity could be conducted with different commodities, or the same activity could be tried for the same commodity, using different aggregators or firms along the value chain, or a single combination could be conducted in different locations around the country. Thus, while these may not be formal experiments, there are myriad opportunities for USDA programming for innovative initiatives and a broad mechanism for supporting experimentation within a project structure. Table 26: Top EQ Activities/Interventions, Outcomes, and Relation to FFPr Strategic Objectives under Section 3.5 of this report provides a list of the top EQ activities and interventions, rated by beneficiaries across various data sources, and their outcomes. 96 Findings in Table 26 indicate, that any combination of top-rated activities can render positive outcomes for beneficiaries and ultimately support meeting FFPr strategic objectives (SOs) 1 and 2. See Question EE.3 in Section 3.5 for more details. 97 Annex A: Bibliography ▪ Cómo Mantener el Dinamismo Exportador en la República Dominicana Un Diagnóstico del Banco Mundial sobre Competitividad Comercial ▪ DR Ministry of Agricultural Statistics ▪ DR National Statistics Office (ONE) ▪ Harvard Atlas of Economic Complexity ▪ International Trade Center (ITC) Trade Map ▪ Knoema Data Services, New York, NY (for exports and imports of the Dominican Republic) ▪ OECD/DAC Network on Development Evaluation (2019.) Better Criteria for Better Evaluation: Revised Evaluation Criteria Definitions and Principles for Use ▪ United Nations (UN) Comtrade database ▪ USDA FFPR Exporting Quality (EQ) Monitoring and Evaluation Plan (January 22, 2021) ▪ USDA FFPR Exporting Quality (EQ) Performance Management Plan (Updated as of November 15, 2015) ▪ USDA Foreign Agricultural Service Food Assistance Division, (2014.) Food Assistance Indicators and Definitions ▪ USDA Foreign Agricultural Service Food Assistance Division, (2014.) Monitoring and Evaluation Policy ▪ USDA-Foreign Agricultural Service’s Global Agricultural Trade System (GATS.) ▪ USDA Food for Progress Learning Agenda: Trade Expansion and Agricultural Market Development ▪ World Trade Organization Data 98 Annex B: EQ Questions & Indicators for the Final Evaluation Performance (Results & Activity) Indicators66 to be Updated (as amended) N=35 Results indicators Result Indicator #1 (FFPr 17) # of individuals benefitting directly as a result of USDA assistance Result Indicator #2 (FFPr 18) # of individuals benefitting indirectly as a result of USDA assistance Result Indicator #3 (FFPr 16) # of individuals who have received short-term agricultural sector productivity or food security training Result Indicator #4 (FFPr 1) # of hectares under improved techniques or technologies Result Indicator #5 (FFPr 2) # of individuals who have applied new techniques or technologies Result Indicator #6 (FFPr 3) # of individuals who have applied improved farm management practices Result Indicator #7 (FFPr 13) Value of sales by project beneficiaries Result Indicator #8 (FFPr 14) Volume of commodities (metric tons) sold by project beneficiaries Result Indicator #9 Percentage of tests of target post-production agricultural products that come back clean Result Indicator #10 # of direct beneficiaries achieving internationally recognized certifications Result Indicator #11 Average percent reduction in time a product takes to get through problem points in a value chain Result Indicator #12 # of partner packinghouses using a traceability system Result Indicator #13 Quantity of post-harvest equipment and facilities improved Result Indicator #14 # of food safety/standards-ready target commodities with improved marking and branding Result Indicator #15 (FFPr 9) Value of new public and private sector investment leveraged Result Indicator #16 Value of buyer/seller contracts facilitated Result Indicator #17 Average OCI score of target producer groups Result Indicator #18 % of target value chain groups that have a systematic way of disseminating and updating market information Result Indicator #19 (FFPr 8) # of public–private partnerships formed Result Indicator #20 (FFPr 7) # of enterprises and organizations that applied improved techniques and technologies Activity Indicators Activity Indicator #1.1 # of trainings delivered to increase capacity of fruit and vegetable producers, processors ans packhouses 66 Source: EQ PMP, Midterm Evaluation, and EQ amendments 99 Activity Indicator #1.2 # of producers benefitting from training and technical assistance Activity Indicator #1.3 # of processors benefitting from training and technical assistance Activity Indicator #1.4 # of packinghouse staff benefitting from training and technical assistance Activity Indicator #1.5 Value of grants provided to producers Activity Indicator #2.1 # of processors/exporters trained in marketing and branding techniques Activity Indicator #2.2 # of direct beneficiaries participating in trade shows, buyer visits to DR, and exporter visits to United States and Europe Activity Indicator #2.3 # of producer association and trade association members benefitting from end-market assessments Activity Indicator #2.5 # of individuals from organizations trading target commodities benefitting capacity building activities Activity Indicator #2.6 # of women in trade and producer organizations management who participate in program-sponsored leadership training and events Activity Indicator #2.7 Value of grants provided to trade associations or producer associations that promote exports Activity Indicator #3.1 # of individuals attending cold chain knowledge sharing events Activity Indicator #4.1 # of fruit and vegetable processors benefitting from in-kind equipment to create value added products Activity Indicator #4.2 Value of in-kind grants provided to processors Activity Indicator #5.1 # of individuals reached by public information campaigns Activity Specific Questions67 N=29 Activity 1: Capacity Building: Producer Groups/Cooperatives A1.1 What impact has market-driven certification programs had on beneficiaries' access to markets and exports? A1.2 What effect has the increased presence of traceability systems had on beneficiaries' market access and exports? A1.3 What impact has improved compliance with sanitary and phytosanitary standards had on market access and exports? A1.4 What effect have the use of farmer field schools had on organizational effectiveness, production costs and yields? A1.5 Has the revitalization of cocoa and avocado farms led to significant increases in overall production and yields? A1.6 Has the implementation of recommendations by the organizational capacity index led to significant performance improvements in the organizations? A1.7 What impact has integrated pest management had on beneficiary costs, yields, access to markets? A1.8 What impact has the FSMA readiness food safety strategy for pack-houses and related interventions had on market access and exports? 67 Source: RFP Final Evaluation Services for IESC’s USDA-funded Exporting Quality Program, Table 2 100 A1.9 Did improved post-harvest infrastructure contribute to an increase in the value added to postproduction agricultural products? A1.10 Has the support for the Ministry of Agriculture in monitoring and trapping Medfly and Tuta Absoluta led to notable commercial impacts for the private sector (e.g., sales/exports) and/or the ministry's disposition to continue to apply such mitigation/preventive measures after program close-out (sustainability)? Activity 2: Capacity Building: Trade Associations A2.1 What impact has establishing market linkages through participation in trade shows, trade missions, market promotion, and market studies had on expanding market access and increasing sales? A2.2 Due to the pandemic some trade fairs were virtual. What was the impact of participation on virtual trade fairs? A2.3 Did program interventions have an impact in increasing the value and volume of exports? Has increasing the value and volume of exports altered the scale, composition or structure of value chains? A2.4 Has the introduction of new export products altered the scale, composition or structure of value chains? A2.5 Has the development of new markets altered the scale, composition or structure of value chains? A2.7 How has the training of beneficiaries to identify market trends and requirements impacted their participation in export markets and their sales/exports? A2.8 Has training beneficiaries in financial and accounting management and ecommerce led to organizational improvements and improved commercial performance (sales, exports)? A2.9 To what extent did improved marketing and branding of agricultural products contribute to an increase in access to markets to sell agricultural products? A2.10 Has the implementation of initiatives to ensure inclusive growth and diversity within the export sector, e.g., Young Exporters Network and Networking for Women in Trade Leadership led to increased partnerships among membership that yielded greater lobbying power? A2.11 Has the promotion of cocoa and chocolate through activities like the National Chocolate Competition and Festival lead to a discernible impact on the chocolate industry and exports? Activity 3: Cold Chain Improvement A3.1 To what extent did key players – logistics and export operations actors who transit cargo, like Cold Chain Working Group members - recognize and comply with the program’s cold chain pilot program guidelines and use the related manual? A3.2 What was the impact of the World Food Logistics Cold Chain Institute training? A3.3 Did cold chain interventions have an effect on other sectors? If so, to what extent? Activity 4: In-Kind Grants: Equipment 101 A4.1 What impact did in-kind equipment grants have on productivity, e.g. enhanced production volumes? A4.2 What impact did in-kind equipment grants have on product quality as experienced by the program beneficiaries who used equipment donated by the program? A4.3 Did increased productivity from improved equipment increase domestic demand for raw materials? If so, what was that estimated increase? Activity 5: Public Information Campaign: Disperse Improved Market Information A5.1 What degree of coverage and outreach was achieved through the Consumer Food Safety Campaign? A5.2 What were the discernable behavioral changes that may have occurred in particular population segments as a result? Questions Linked to OECD/DAC Criteria68 (N=15) Relevance To what extent did the Exporting Quality project design address the core issues of target beneficiaries? How were existing relevant USDA and U.S. government activities leveraged? To what extent did the program align with the Dominican Republic’s agriculture and/or development investment strategy? With USDA and USG’s development goals? Was the project designed taking into account economic, cultural, and political contexts? Effectiveness What were the major results of the project in achieving the goal and intended objectives? To what extent did Exporting Quality achieve the specific targets and results established? Which activity or combination of activities proved to be the most effective approach to achieve the project’s higher-level results, namely increased access to improved market information; improved capacity of key groups in the agriculture production sector; and increased leverage of private sector resources? To what extent has training and certification had an influence on improving the quality of post-production agricultural products? How effective has participation in trade shows and buyer missions been to improve linkages between buyers and sellers? Efficiency To what extent did the level of project resources lead to achievement of results? Could the same results be achieved with fewer resources? Sustainability What is the likelihood that the project benefits will endure over time after Exporting Quality ends? To what extent has Exporting Quality developed local ownership and sustainable partners? 68 Source: RFP Final Evaluation Services for IESC’s USDA-funded Exporting Quality Program 102 Impact What are the immediate-, medium-, and long-term effects, intended and unintended, positive and negative, of the project after nearly six years of implementation? Other What was the overall impact of COVID-19 restrictions/protocols on project results? How did the program adapt to the pandemic and to what extent were adaptations of activities successful in reaching beneficiaries? The Exporting Quality Final Evaluation must also determine whether recommendations from the Exporting Quality Midterm Evaluation were incorporated into the project and if not, why not? 103 Annex C: Food for Progress Results Frameworks 104 105 Annex D: Datasets- Submitted Separately Annex E-1: USDA EQ Final Evaluation Work Plan and Instruments-Submitted Separately 106 Annex E-2: Additional Sampling Data This annex provides some additional context and information for the quantitative data collected and supplements Section 2.1: Sampling Methods from this EQ final evaluation. Three quantitative data collection instruments developed and deployed. 1) Producer/ Farmer Survey 2) People in Firms Survey 3) People in Firms Online Mini-Survey The table below provides information regarding the database of EQ beneficiaries provided by EQ, as well as the overall response rates across beneficiary groups and data collection methods. Producers/Farmers (In-Person Survey) Valid Records 4,856 Eliminated Records 211 1. No Province listed 139 2. No value chain listed/ non-EQ value chain listed/More than 1 value chain listed 72 Total EQ Beneficiary (Producer) Population usable for Farmer/Producer Sampling Plan (Valid-Eliminated Records) 4,645 Producer Survey Response Rates 1. Total Producer Contacts Made 1,227 2. Number of Refusals 74 3. Number Not Available for Survey 10 4. Number of ‘No answer’/’No response’ 811 5. Surveys Planned 330 6. Surveys Completed (100 percent of sample) 332 People in Firms (Phone Survey) Valid Records 684 Total EQ Beneficiary (Firms) Population Usable for People in Firms Sampling Plan (Valid Records) 684 People in Firms Survey Response Rates 1. Number of Refusals 127 2. Confirmed Wrong Numbers 26 3. Number of ‘No answer’ & Unconfirmed Wrong Numbers 407 4. Surveys Planned 171 5. Surveys Completed (72 percent of sample) 124 People in Firms (Leadership) (Online Survey) Valid Records 111 Eliminated Records (with no email address listed) 11 Total EQ Beneficiary (Firm Leaders) Population Usable for People in Firms (Leadership) Sampling Plan (Valid- Eliminated Records) 100 107 People in Firms (Leadership) Survey Response Rates 1. Surveys Planned As many as possible within data collection timeframe 2. Surveys Completed (20 percent of census) 19 108 Annex G: Final Evaluation Terms of Reference Exporting Quality Final Evaluation Objectives and Scope Statement The purpose of the Exporting Quality Final Evaluation is to impartially assess whether the project achieved the expected results as outlined in the results framework. The scope of the evaluation will comprise of - Exporting Quality project design, implementation, management, and replicability; - lessons learned and recommendations for USDA, Exporting Quality participants, and other key stakeholders for future projects; and - follow-up on midterm evaluation questions, including assessing direct and indirect, intended and unintended, positive or negative impacts; and the extent to which Exporting Quality addressed midterm evaluation recommendations and incorporated them into the program’s activities and operations. The contractor will conduct the Exporting Quality Final Evaluation in compliance with USDA’s monitoring and evaluation (M&E) policy and the approved Exporting Quality Evaluation Plan. The evaluation will examine both administrative and programmatic aspects of Exporting Quality related to data capture, measurement, and intervention impact. The analysis of progress against indicator targets will define areas of shortfalls and areas of success that may highlight opportunities for IESC and USDA to design future successful interventions leading to greater impact. The contractor’s evaluation team will include various positions and the Offeror should present a detailed scope of work for each proposed role on the team. Evaluation Key Audience. The key audience for the Exporting Quality Final Evaluation includes the IESC Exporting Quality program staff, including the Exporting Quality Steering Committee comprised of key U.S. government and Dominican government stakeholders, USDA staff, and other USDA implementers. The audience also includes Exporting Quality’s international partners Global Cold Chain Alliance-World Food Logistics Organization (GCCA-WFLO) and Florida Agricultural and Mechanical University (FAMU), local partner Centro para el Desarrollo Agropecuario y Forestal (CEDAF), Exporting Quality participants and beneficiaries, and trade associations and agencies within the government of the Dominican Republic. These evaluations also intend to benefit other USDA implementers and the development community in general. The Exporting Quality Final Evaluation report will be made publicly available. IESC and the offeror will ensure public copies of the evaluation reports are free of personally identifiable information (PII) and proprietary information. Additionally, final versions of the evaluation reports will also be made accessible to persons with disabilities. Methodology for Exporting Quality Final Evaluation. The independent third-party evaluation contractor will be encouraged to use a mixed methods approach using a variety of qualitative and quantitative methods including quantitative surveys, focus group discussions with and direct observation of target beneficiaries, and key informant interviews with government officials and relevant public/private stakeholders including IESC Exporting Quality program staff and USDA representatives. For assessment purposes, evaluations may consider trade data that is available from official Dominican government sources and/or other reliable sources as appropriate. IESC will confirm details around survey design, interview questions, and sampling in conjunction with the contractor. IESC will also provide project documents to inform a preliminary desk review, e.g., the baseline and midterm evaluations, program work plans, and semi-annual reports. Information on the relevant value chains and geographic regions of activity implementation may be found in Section 2. Background. The contractor will assess the situation and include any COVID-19 considerations in its data collection methodology as needed. Due to the likelihood that COVID-19 will still be prevalent during the Exporting Quality Final Evaluation period, all in- 109 person interactions and data collection with pre-identified beneficiaries will follow DR public health protocols to protect the beneficiaries and the evaluation team. Virtual data collection methods successfully tested and used by the Exporting Quality program will be utilized when possible, including phone calls or video conference interviews, surveys conducted by emails, and online surveys using various platforms. The final data collection methods will be discussed and confirmed with the selected contractor. Some beneficiaries are counted in multiple beneficiary groups depending on the activity and indicator. As of September 30, 2020 the Exporting Quality program registered an estimated 3,931 unique, direct beneficiaries by the following types: Agriculture Producers – 3,002; Agriculture Producer Organizations and Trade Associations – 39; People in Firms i.e., individuals working in private enterprises – 582; Private Enterprises, i.e., packinghouses, processors, exporters – 267; People in Government – 40 IESC will update this count once program activities conclude. The Exporting Quality Final Evaluation will not attempt to confirm direct attribution of impact which is not possible. As such, the evaluation is not expected to include a counterfactual or comparison group. Exporting Quality does not anticipate conducting an experimental or quasi-experimental impact evaluation for it will be logistically unreasonable to find and maintain control or comparison groups due to the reach of the project throughout multiple agricultural value chains. This is a performance evaluation and is intended to verify contributions and plausible links between impact and Exporting Quality activities through contribution analysis rather than scientific attribution through randomized impact evaluations. The concept of plausible association does not mean X input equal Y output. Rather, it means a case can be made that X input, e.g., farm field school trainings, has materially affected Y output, e.g., increased number of hectares under improved techniques or technologies as a result of USDA assistance. Contribution analysis will involve 1) gathering existing evidence on the various results and the links/assumptions that lead to these results; and 2) assembling and assessing the contribution story and the challenges to it. The challenges or weaknesses in the story point to where additional data or information is needed. The contribution analysis methodology is a useful approach when working in complex areas where assessment of sole attribution is difficult, such as in the space of agriculture and food safety in the context of local, regional, and international trade. The essential value of contribution analysis is that it offers an approach designed to reduce uncertainty about the contribution that Exporting Quality is making to the observed results through an increased understanding of why the observed results have occurred (or not) and the roles played by Exporting Quality and other internal and external factors. The conclusion from a contribution analysis is not definitive proof, but rather provides evidence and a line of reasoning from which Exporting Quality can draw a plausible conclusion that, at some level of confidence, the project has made an important contribution to the documented results. The evaluation contractor will apply the contribution analysis methodology using existing quantitative data from the program’s M&E system, qualitative information from interviews, and other contextual factors that help validate or clarify the contribution story. Any missing ‘piece’ of the story is where additional data or information is needed, which the evaluation contractor will collect. Quantitative data will be analyzed using the appropriate descriptive statistics rather than inferential statistics given there are no control or comparison groups. Certain relationships may be analyzed by calculating the correlation between two variables (e.g., the relationship between the number of trainings delivered to increase capacity of fruit and vegetable producers and processors and the number of project-supported packinghouses that meet FSMA requirements). Details will be discussed and confirmed with the 110 evaluation contractor. IESC has also conducted post-assessments for equipment donation activities and an internal beneficiary survey. Exporting Quality will share these assessments with the evaluation contractor. Relevance. IESC is the prime implementer of the Food for Progress Exporting Quality project (September 2015 to November 2021) in in the Dominican Republic funded by the U.S. Department of Agriculture (USDA). IESC requires a third-party contractor (independent consultant or firm) to conduct the Final Evaluation for the Exporting Quality project. Contractor Tasks. In consideration of IESC’s Request for Proposals: Final Evaluation Services for IESC’s USDA-funded Exporting Quality Program / RFP-DREQ-004-2021 and the contents of the Contractor’s offer, the Contractor will undertake the following tasks. 1. Work Plan a. Participate in kick off coordination meetings with IESC and any IESC invitees, such as but not limited to representatives of USDA; b. Develop a final evaluation work plan describing an understanding of the project based on desk review and the kickoff meeting; methodology; planned quality control measures; communications protocol; and a final timeline. 2. Pre-Assignment a. Provide IESC with draft data collection tools for review and comment; b. Develop final versions of the data collection tools incorporating IESC feedback; c. Develop an in-country travel schedule including, but not limited to, local travel dates and locations. d. Desk review of previous evaluation reports and other relevant program documents. 3. Conduct Primary Data Collection a. Travel to the Dominican Republic to conduct primary data collection according to the agreed upon travel schedule. 4. Post-Assignment a. Create clean and final versions of all quantitative datasets and qualitative transcripts in an agreed upon format in English; b. Develop a two- to three-page brief describing the evaluation design, key findings, other relevant considerations, and including graphics and tables, in layperson language to inform final evaluation stakeholders; c. Capture 15 – 20 high-quality photographs documenting data collection process; d. Deliver presentation of brief to IESC and any IESC invitee such as, but not limited to, USDA on the evaluation findings and materials; e. Develop a trip report following international travel to the Dominican Republic. 5. Final Evaluation Report a. Create a draft of the Exporting Quality Final Evaluation report in English; as an editable Word document; and addressing all evaluation objectives and questions; b. Create a thirty- to forty-page electronic final English version of the Exporting Quality Final Evaluation report in PDF and Word for internal IESC use. These estimated page limits do not 111 apply to relevant annexes. Annexes within this version may contain proprietary information owned by third parties; information that could put individual safety at risk; and personally identifiable information. c. Create a thirty- to forty-page electronic final English version of the Exporting Quality Final Evaluation report in PDF and Word public distribution without any proprietary information owned by third parties; information that could put individual safety at risk; and personally identifiable information in the report itself or annexes. The report should include, but is not limited to, the same elements as the final report for IESC internal use.