Midterm Evaluation Prepared by Antonio Cordella | Last Updated January 19, 2018 Dominican Republic – Exporting Quality Program Midterm Evaluation ii List of Acronyms iii Executive Summary 1 1. Background 3 2. Detailed Evaluation Methodology 6 3. Auditing the Program’s Database of Beneficiaries 10 4. Findings 11 5. Suggestions and Requests from Beneficiaries 31 6. Recommendations for the Remainder of the Program 34 Annexes: Annex 1: Scope of Work for Midterm Performance Evaluation of EQ Program Annex 2: Program Database and the Evaluation Sample Annex 3: Overview of Indicator Performance Annex 4: Export Statistics Annex 5: Evaluation Questionnaire Annex 6: EQ Organizational Chart Annex 7: Evaluator’s CV [REMOVED] Annex 8: Evaluation Photographs [REMOVED] Dominican Republic – Exporting Quality Program Midterm Evaluation iii ADEXVO Association of Dominican Exporters of Asian Vegetables ADOEXPO Association of Dominican Exporters AMS Annual Program Statement APHIS Animal and Plant Health Inspection Service CAFTA Central America Free Trade Agreement CARICOM Caribbean Community CEDAF Centro para el Desarrollo Agropecuario y Forestal CEI-RD Dominican Republic Export and Investment Center CNC Centro Nacional Competitividad CONACADO Confederación Nacional de Cacao-Cultores Dominicanos DR Dominican Republic EAN Emergency Action Notification EQ Exporting Quality Program EU European Union FAMU Florida Agricultural and Mechanical University FAST Food Safety Agro Sustainability Training FSMA Food Safety Modernization Act GAP Good Agricultural Practices GHP Good Handling Practices GMP Good Manufacturing Practices HACCP Hazard Analysis Critical Control Point HARPC Hazard Analysis and Risk-Based Preventive Controls for Human Food IESC International Executive Service Corps IPM Integrated Pest Management Medfly Mediterranean fruit fly Min-Ag Ministry of Agriculture of the Dominican Republic M&E Monitoring and Evaluation MT Metric Ton OCI Organizational Capacity Index PACA Perishable Agricultural Commodity Act PCQI Preventive Controls Qualified Individual PMP Performance Management Plan SoW Scope of Work SPS Sanitary & Phytosanitary USDA United States Department of Agriculture VC Value Chain Dominican Republic – Exporting Quality Program Midterm Evaluation 1 The Exporting Quality Program Midterm Evaluation was carried out between November and December 2017 with the scope of evaluating the USDA-funded four-year program’s performance. The evaluation was conducted by an independent, third-party consultant who worked with a local enumerator. The evaluation encompassed various activities including: desk review of technical documents and statistics; interviews and on-site visits with producers, processors, association and cluster management, packinghouse owners and staff, domestic traders, exporters, extension officers, and sector experts; and direct market observations. The evaluation was executed according to the USDA-approved midterm evaluation scope of work (SoW). The main findings included the following: • The program is mostly demand-driven and has shown sufficient flexibility to accommodate beneficiary requests for more training than planned on farming methods and Food Safety Modernization Act (FSMA) issues; • A large majority of interviewed beneficiaries have great appreciation for the program’s approach to training (practical and in response to beneficiaries’ actual needs) on farming techniques and the relevance of information provided on both sanitary and phytosanitary (SPS) issues and marketing issues; • Beneficiaries believe program interventions have been efficient, relevant, and effective, and that they will continue to apply what they have learned after its completion; • The Performance Management Plan (PMP) contains 36 indicators to monitor the program’s achievements over its implementation period. The program has exceeded its midpoint target (end of year two [Y2]) for almost a third of these indicators; for two￾thirds of them it has attained less than 50 percent of target; and for one-fifth of the performance indicators less than 10 percent of target has been achieved; and one indicator (Result Indicator #17) does not have a midpoint target; Image 1: Indicators Grouped in Sets According to Their Progress Against Year Two Targets Dominican Republic – Exporting Quality Program Midterm Evaluation 2 • For most performance indicators that are below the end of Y2 cumulative target, the evaluator considers that the program is on track for year four targets, provided it sustains the pace of action and achievement demonstrated during the most recent reporting period (April–September 2017) of implementation; and, • The program had a slow start due to setup and activity fine-tuning, but has shown remarkable dynamism during the second half of Y2 that assures the meeting of targets if maintained for the remaining period of implementation. Recommendations from the midterm evaluation include the following: • The evaluator recommends continuing and following up with (i) the current training approach, (ii) support for getting internationally recognized certifications, (iii) the provision of grants to farmers, packers, and processors, as well as the enlarging of their scope, and (iv) the facilitation of study tours and exporter and buyer visits; • The evaluator also recommends focusing more on (i) post-harvest handling techniques and methods; (ii) Integrated Pest Management (IPM) and sanitary-related activities; and (iii) end-market activities; • For a few (mainly activity) indicators, including activities with members of trade associations, the evaluator recommends either (i) scaling down end of year 4 (Y4) cumulative targets (due to initial overestimation of the number of trade associations operating in the Dominican Republic [DR]); or, preferably, (ii) enlarging the purpose to include producer associations, cooperatives, and clusters, all of which would be able to trade produce among members; • For a significant number of performance indicators (related primarily to training activities) the evaluator recommends scaling up the Y4 targets, as the program showed higher than expected results during the second half of Y2. This scaling up would more than compensate for the above scaling down of result indicator targets; • The evaluator recommends that the program perform some assessments specifically requested by beneficiaries participating during focus group discussions, including (i) expanding export of produce to the Caribbean islands, (ii) availability and cost of water treatment facilities, and (iii) facilitating resolution of business disputes over shipment overdue payments; • For four of the results indicators, the evaluator recommends lowering the end of Y4 cumulative targets, as the set targets appear unattainable based on current action delivery trends and achievements and types of beneficiaries potentially reachable; and, • Finally, the evaluator recommends the program prepare an exit strategy assuring sustainability of activities after program completion. Dominican Republic – Exporting Quality Program Midterm Evaluation 3 The Exporting Quality Program is a US$15 million program that runs from October 2015 to September 2019, implemented by the International Executive Service Corps (IESC) in the DR. The program aims to increase productivity and sales for domestic and export markets of high-value fruit and vegetable global value chains: avocado, cocoa, pineapple, and greenhouse and Asian vegetables. The program is expected to improve product quality, increase production and efficiency, increase the value of post-harvest products, and improve market and marketing linkages. The program is organized into the following five activity areas: A. Capacity Building: Producer Groups and Cooperatives; B. Capacity Building: Trade Associations; C. Cold Chain Improvement; D. In-Kind Grants: Equipment; and, E. Public Information Campaigns: Disperse Improved Market Information. The program activities are designed to work primarily with value chain aggregators, such as producer organizations (clusters and associations), cooperatives, packinghouses, processors, and trade associations, to reach middlemen—traders and producers. The program looks at end markets to determine growth-related demand and requirements, in turn focusing on technical solutions related to productivity losses (e.g., orchard underperformance, technology gaps) and product losses (e.g., spoilage, cold chain mismanagement, point of entry refusal, and food safety). The program’s interventions have focused on productivity and product losses of 30 percent or higher in the first two years to achieve the end-of-year-two target results related to volume (a 25 percent increase to 55,934 metric tons from a baseline of 42,505 metric tons) and sales (a 25 percent increase to US$53.2 million from a baseline of US$44.7 million). Losses can also be defined as not having access to key markets due to certain product bans (e.g., the Mediterranean fruit fly, Ceratitis capitata [medfly], thrips, and other contaminations) or refusals based on import inspections. The program has endeavored to work with a significant number of beneficiaries in each value chain to achieve the end-of￾year-two targets (3,440 direct and 17,200 indirect beneficiaries), which should then allow the transformation needed for continued and sustained growth as additional market-driven scaling and sector-wide replication occur. The program is monitoring a total of 36 result and activity indicators, of which 32 had a baseline of zero and four had baseline data established within six months of program start￾up. The evaluation focused on the program’s performance against the end-of-year-two targets for these indicators, as well as on quantitative and qualitative information obtained through survey results, focus group discussions, and interviews with main stakeholders. Dominican Republic – Exporting Quality Program Midterm Evaluation 4 Geographic coverage. The program is carrying out activities in the areas of Santo Domingo, La Vega, San Juan, Santiago, San Cristóbal, San José de Ocoa, Monte Plata, Monseñor Nouel, Duarte, Puerto Plata, Peravia, Sánchez Ramírez, and Azua provinces. The program has recently expanded its training activities to avocado producers in the province of Pedernales, located in the most southern part of the country bordering Haiti (see Figure 1). Trade Trends for Target Commodities Recent trade patterns and trends of target commodities show exports have largely returned to pre-U.S. export ban 2014 levels in terms of value. Resulting from medfly infestations, this ban was in place between March 18, 2015 to January 9, 2016. Figure 2 below displays the Figure 2: Quarterly Export Trends of Target Commodities (excluding Cocoa) Figure 1: Map of Program Activities Dominican Republic – Exporting Quality Program Midterm Evaluation 5 value of products from target value chains traded between late 2014 and late 2016, except for cocoa products. The trade trend for cocoa products is separated in Figure 3 due to the large difference in magnitude between their values and those of the others. Avocado exports have almost returned to pre-ban export value levels, providing necessary relief to producers, some of whom preferred not to harvest the crop due to low farm gate prices. As for greenhouse production, 2016 total output is back to 2014 levels after the 2015 drop (again, due to the medfly ban), but there has been a partial shift of production from peppers to tomatoes and cucumbers.1 This production shift is confirmed by the export trend, with 2016 tomato export value higher than the 2014 level, while pepper export value is still well below that amount. Producers of bell peppers in Jarabacoa were the most affected by the medfly ban, leaving them with huge losses and high debts; producers in Rancho Arriba and Ocoa were also significantly affected. Producers of chilies (an Asian vegetable) were also negatively affected by the medfly ban. Cultivation of Asian vegetables was then hit by huge and persistent rains during the fall of 2016 and, more recently (August 2017), by the effects of Hurricane Harvey. Nonetheless, the export value of okra more than doubled between 2014 and 2016 (though from a small basis), mainly thanks to increased sales to France. As for pineapples, their export value increased between 2014 and 2016, but at a lower rate than expected. Pineapple production continues to supply the domestic market, which is still at least as remunerative as the export market. Finally, the export value of cocoa products has continued to grow steadily. 1 Export of green tomatoes was allowed as of April 2015, while export of cucumbers was never banned. Figure 3: Quarterly Export Trends of Cocoa Dominican Republic – Exporting Quality Program Midterm Evaluation 6 The evaluation team, Antonio Cordella, lead evaluator, and Christian Baez, enumerator, used a combination of methods to perform the evaluation, including a quantitative survey through a questionnaire, focus group discussions with target beneficiaries, and interviews with key public and private stakeholders including program staff, local implementing partner Center for Agriculture and Forestry Development (CEDAF), the Ministry of Agriculture, and USDA. In addition, the evaluation team used the data collected by the program’s M&E staff to execute the performance analysis of the 36 PMP indicators; the data was validated by direct observations and field visit checks. The team also collected and analyzed production and trade data from official statistical sources, such as those used during the baseline study. 2 The data collection tools used for the evaluation were similar to those used during routine monitoring by the M&E staff, thus results may be compared. The evaluation team also executed an informal audit of the data collected by the program’s M&E staff.3 During fieldwork, the evaluation team undertook a survey of program beneficiaries (producers, extension officers, packinghouse owners and staff, processors, domestic traders, exporters, and service providers) through the submission and completion of an ad hoc questionnaire designed by the team and refined with the assistance of the program’s technical staff (see Annex 5). The beneficiaries interviewed were randomly selected from some 2,617 beneficiaries (whose names and contact information were provided by the program) and the questionnaires were filled-in either at program events or over the phone.4 Toward the end of fieldwork activities, five focus groups were organized in which the main stakeholders of each value chain provided valuable additional information on program activities and results, as well as their expectations for the remainder of the program. Photographs of these groups and other meetings are attached as Annex 8. Finally, the team met with relevant public and private partners to discuss program achievements to date, expectations for the remaining period of implementation, and sustainability of efforts.5 Completion of the survey took a relatively long time due to the difficulty of contacting many beneficiaries located in remote parts of the country and often working in the field, where telecommunication networks are weak or nonexistent. Once the questionnaire survey was completed, the evaluation team conducted data entry, then processed and analyzed the 2 Trade data from ITC Trade Map web site (http://www.trademap.org/); domestic production data from Oficina Nacional de Estadìstica (ONE) (http://www.one.gob.do/#). 3 The evaluation team thanks the EQ program staff for the invaluable support provided during the execution of the evaluation. 4 Additional information on the program database and an evaluation sample can be found in Annex 2 of this report. 5 Partners met include; Pedro Pablo Peña (CEDAF’s Executive Director), Janina Segura (planning manager at CEDAF); Emigdio Gomez (SPS Director, Min-Ag); Raul Peralta (Food Safety Director, Min-Ag), and Virgilio A. Mayol (USDA agricultural specialist). Dominican Republic – Exporting Quality Program Midterm Evaluation 7 collected evaluation data. The team organized all the collected information into a database that ensures anonymity of evaluation participants by using identification numbers and separating names from participants’ data. The team performed quality assurance at all stages: design; testing; fieldwork; data entry; data analysis; and, reporting. Survey Sample Screening The evaluation team carried out the survey between November 15 and December 21, 2017. The questionnaire was tested at a program training event on Organizational Capacity Index (OCI) methodology, where a dozen pineapple beneficiaries were asked to complete the questionnaire. The testing went well, and the survey continued over the following five weeks. The evaluation team administered 337 questionnaires but, after a quality assurance test, 11 were rejected, leaving the analyzed sample at 326. A further nine beneficiaries were then contacted and the questionnaires administered to reach the target of 335 completed questionnaires. This sample size ensures a 95 percent confidence level with a 5 percent margin of error and a 50 percent distribution (see Annex 2). The majority of surveyed beneficiaries are producers (46 percent of the sample), but there is also a significant share of extension officers and consultants (30 percent), most on the ministry of agriculture payroll even if working with specific producer associations; exporters represent 11 percent of the sample (many of them are also producers and, in certain cases, packers); packinghouse staff accounts for just six percent of the sample, mainly dealing with both greenhouse and Asian vegetables; cocoa processors add an additional four percent; and domestic traders a three percent share; the sample also includes two students and one air transporter (see Figure 4). Figure 4: Survey Sample by Profession Dominican Republic – Exporting Quality Program Midterm Evaluation 8 As illustrated in Figure 5, the survey sample breakdown by value chain includes operators working mainly with cocoa (24 percent of the sample), avocados (21 percent), Asian vegetables (20 percent), greenhouse vegetables (eight percent), and pineapple (six percent), while beneficiaries dealing with all target value chains account for a 16 percent share. This group primarily consists of extension officers with some exporters and traders. The remaining five percent of the sample includes would-be beneficiaries (e.g. students) and beneficiaries of other value chains (bananas, mangoes, coffee, citrus, tobacco, cheeses, meats, and sausages) as well as non-food goods (color additives, furniture). Men represent the largest share of survey beneficiaries, 75 percent. Women have a slightly higher representation in the cocoa value chain where they account for almost 33 percent of interviewed beneficiaries, as well as in the group of beneficiaries dealing with all value chains, i.e., extension officers, traders, and exporters. Women are less present in value chains such as Asian vegetables (where they account for only 19 percent of the total), greenhouse vegetables (7 percent), avocados (4 percent), and pineapples (3 percent), as displayed in Figure 6. Figure 5: Survey Sample by Value Chain Dominican Republic – Exporting Quality Program Midterm Evaluation 9 Producers are the majority of interviewed beneficiaries in the avocado, cocoa, and pineapple value chains, but less so in the Asian and greenhouse vegetable value chains where, in fact, most exporters are also producers, as shown in Table 1. All interviewed exporters of Asian vegetables have packinghouses. The number of extension officers interviewed is significant for both greenhouse and Asian vegetables, as well as for cocoa. Surveyed processors are only represented in the cocoa value chain. Table 1: Number of surveyed beneficiaries by profession and value chain Producers Packinghouse staff Processors Traders Exporters Extension officers Avocados 63 7 Pineapples 11 4 5 Greenhouse Vegetables 3 5 1 5 13 Asian Vegetables 22 13 16 12 Cocoa 52 14 1 14 Figure 6: Female Participants by VC Dominican Republic – Exporting Quality Program Midterm Evaluation 10 In accordance with a specific request from USDA made during a teleconference, an informal audit of the quality of the database supplied by the program was undertaken. The analysis of the performance of both Result and Activity Indicators also permitted an indirect audit of the database. The volume of data collected by the program’s M&E team is impressive, as is its organization, with separate information for each of the 36 PMP indicators. For each program activity a list of participants is provided, though unfortunately not all of them contain valid contact information (phone or e-mail) nor profession and value chain to which they belong. It is worth noting that some beneficiaries (especially producers) do not have email or phone. The database contains a few participants who were mistakenly keyed in twice or even more often, due to misspelled or incomplete names. It also contains some would-be beneficiaries who should not have been included because they were merely contacted or involved in an activity through brief attendance (non-recurring participation). 6 An estimation of the total number of the beneficiaries mistakenly included in the database is set at about three to five percent of the audit sample. There are also a few beneficiaries attending program activities who may not have been included in the database, or at least their names could not be found in it (these are estimated at between two and four percent of the audit sample). All in all, by balancing the two estimates, the evaluation team assesses the number of beneficiaries over￾estimated in the database at around one to two percent, thus within an acceptable statistical margin of error. In Result Indicators #7 and #8, on value and volume of sales by program beneficiaries, the program has calculated the sales attributable to USDA support by interviewing a representative sample of beneficiaries and applying the simple average of their estimation to the framing sample. It is recommended that in the future the program applies the weighted average instead. In addition, at least one third of volume sales of cocoa products in the sample seems to have been computed incorrectly. This should not have a significant impact on the positive performance of this Result Indicator; nonetheless it is recommended that the program M&E checks the relevant data. 6 The typical example of this is high school students attending a conference who are included in the database as direct beneficiaries rather than indirect beneficiaries. Most of them refused to fill in questionnaires when contacted during the survey. Dominican Republic – Exporting Quality Program Midterm Evaluation 11 As for Activity Indicator #2.6 (which counts the number of women in trade association management who participate in program-sponsored leadership training and events), the evaluation team considers that the program’s M&E team has mistakenly computed as beneficiaries all women participating at the three events organized by the program rather than only the women in trade association management as indicated by the indicator definition.7 It is recommended that the program M&E team recalculate this indicator. This is further discussed at the end of the report as recommendation #14, which goes further in suggesting the change of “trade organizations” definition. It is recommended that beneficiaries attending program activities include more precise information including their full name, profession, and related value chain of interest, as well as a valid contact number or email address when completing the attendance sheet. The information on profession will be useful when performing sampling segmentation during the next evaluation at the end of the program, and will provide additional information for routine monitoring. A valid beneficiary contact number/address will facilitate performance of the final evaluation. Findings of the program results evaluation performed by the evaluation team are described in the following sections. The main results suggested by the survey analysis are given first, then the evaluation of program activities as in the SoW scheme (see Annex 1) is provided. Most of the basic information on program activities was provided by the program M&E team. Main Survey Results The results of the survey are shown in Table 2 below. For each question/statement, the table provides the percentage of answers given by the interviewed beneficiaries and the average score calculated based on the points (from 1–5) given for each statement. A score of 5 indicates that the interviewed beneficiaries strongly agree with the relevant sentence, while a score of 1 indicates they strongly disagree. An average score above 3 indicates a positive answer; the higher the score, the more the respondents agree with the relevant question. For each section the highest score is highlighted. As shown in Table 2, the average score is in almost all cases above 4, the only exception being the questions concerning challenges faced by the beneficiary business during the first two years of program implementation, challenges that were not directly related to program 7 The M&E team counted 80 women beneficiaries for this activity indicator, while the audit suggests they number just three, or five percent of the Y2 target. Dominican Republic – Exporting Quality Program Midterm Evaluation 12 activities (e.g., the medfly ban, heavy rains, hurricanes, and so on) and that the program had no ability to influence. Table 2: Survey Results Percentage of sampled beneficiaries providing an answer Average score SECTION 1 PROGRAM EFFICIENCY 1.A.1 The quality of my business has improved thanks to program activities 90% 4.24 1.A.2 The value of my business has increased thanks to program activities 75% 4.14 1.A.3 My technical skills have improved thanks to program activities 97% 4.62 1.A.4 My commercial skills have improved thanks to program activities 50% 4.35 1.A.5 Please quantify the improvements due to program activities 60% 41–60%8 1.B.1 The program has helped me to achieve more than I expected 90% 4.34 1.B.2 Please provide more details 9% - 1.C.1 My business has faced major challenges during the last 24 months 30% 3.46 1.C.2 Please name some of these challenges, if any 4% - 1.C.3 The medfly ban has significantly impacted my activity 20% 3.00 1.C.4 The heavy rains in the Fall of 2016 negatively affected my production/trade/exports 30% 3.89 1.C.5 The 2017 hurricanes (heavy rains and strong winds) negatively affected my production/trade/exports 25% 3.70 1.C.6 The program has been able to help me to counter these challenges 27% 3.75 1.C.7 How can the program further support you in meeting your business challenges? 7% - 1.D.1 How can the program help you in unexpected situations? 5% - SECTION 2 RELEVANCE OF INTERVENTION 2.A.1 The program has been able to address the core issues of my activity 82% 4.08 2.A.2 The program has helped strengthen my organization 81% 4.40 2.A.3 How can the program improve the relevance of its assistance to better support you? 7% - SECTION 3 EFFECTIVENESS OF INTERVENTION 3.A.1 The program activity on export promotion has been very effective in improving my business 30% 4.09 3.A.2 The program activity on cold chain awareness has been very effective in improving my business 19% 4.29 3.A.3 The program activity on branding has been very effective in improving my business 19% 4.20 3.A.4 The program activity on safety measures has been very effective for improving my business 87% 4.54 3.A.5 The program activity on phytosanitary measures has been very effective for improving my business 86% 4.50 3.A.6 The program activity on traceability has been very effective in improving my business 48% 4.35 3.A.7 The program activity on compliance measures has been very effective in improving my business 51% 4.35 3.A.8 The program activity on certification has been very effective in improving my business 50% 4.30 3.A.9 The program activity on GAP/GHM/GMP has been very effective in improving my business 80% 4.47 3.A.1 The program activity on improving/expanding farming productivity has been very 64% 4.35 8 On average the interviewed beneficiaries estimated at between 41-60% the improvement due to Program activities. Dominican Republic – Exporting Quality Program Midterm Evaluation 13 0 effective for improving my business 3.A.1 1 The program activity on improving/expanding processing capacity has been very effective for improving my business 23% 4.22 3.A.1 2 The program activity on improving management capacity of my farm has been very effective for improving my business 56% 4.24 3.A.1 3 How can the program improve the effectiveness of its assistance? 5% - 3.B.1 The program has encouraged participation of women in its activities 91% 4.05 3.B.2 The program has been very effective at reaching women beneficiaries 88% 4.11 3.B.3 What could be done to improve women’s participation in target value chains? 6% - 3.C.1 The training approach has been very good at enhancing my productivity 81% 4.27 3.C.2 The training approach has been very good at enhancing my processing capacity 26% 4.31 3.C.3 The training approach has been very good at enhancing my managerial skills 54% 4.32 3.C.4 How can we improve the training approach to better support you? 5% - SECTION 4 SUSTAINABILITY EFFORTS TO DATE 4.1 At the end of the program I will be able to continue to apply what I learned with no problems 98% 4.64 4.2 I would like this or a similar program of technical assistance to continue after its scheduled end 98% 4.90 4.3 How can the program ensure that this type of assistance is available after their program ends? 4% - 4.4 I would prefer that a similar program of assistance is continued by the Dominican government/public organizations 97% 4.05 4.5 I would prefer that a similar program of assistance is continued by the Dominican private sector 97% 4.10 Section 1 – Program Efficiency Interviewed beneficiaries believe that their technical (score of 4.62) and commercial (score of 4.35) skills have much improved thanks to program activities and that the program has been able to outperform their expectations (score of 4.34). These high scores confirm that beneficiaries believe the program has been very effective in providing them with training and relevant information. Section 2 – Relevance of Intervention Survey results suggest that beneficiaries feel the program has been able to address the core issues of their activity and that has helped strengthen their organizations (with both a high percentage of answers, 4.08, and of high scores, 4.40). Section 3 – Effectiveness of Intervention To evaluate the effectiveness of the program, the evaluation team designed a few statements directed to specific groups of beneficiaries according to the training received and/or activity/event in which they had participated. This means that the percentage of answers is lower than that for other statements, since only the beneficiaries involved with that specific activity were asked to provide an answer. The results are very specific for each type of professional and training received, and provide much useful and high-value information. Dominican Republic – Exporting Quality Program Midterm Evaluation 14 Survey results suggest that activities related to sanitary and phytosanitary measures were the ones in which most beneficiaries participated (more than 86 percent of beneficiaries answered) and one of the most appreciated, since beneficiaries believe that those activities have been very effective in improving their business (scores on these statements averaged 4.54 and 4.50, respectively). The activities related to good practices (GAP, GHP and GMP) also had significant participation (80 percent) and a high appreciation score (4.47). The training approach and its ability to enhance beneficiaries’ productivity has also been considerably discussed (81 percent of answers) and significantly appreciated (a score of 4.27). Among the most highly discussed issues are those related to women’s participation in program activities (91 percent of answers) and whether the program has been very effective at reaching women beneficiaries (88 percent of answers); most beneficiaries (and especially women) considered these activities very successful (scores are above 4 for both statements). All other issues in this section, program effectiveness, received answers ranging from low to middle percentage shares, but their score is on average above 4, again confirming that beneficiaries appreciated the work of the program. Section 4 – Sustainability Efforts to Date The interviewed beneficiaries suggest that they would like this or a similar program to continue after completion (a score of 4.90) with a slightly less favorable opinion for similar programs run either by the Dominican government (a score of 4.05) or by the private sector (4.10). Finally, as for sustainability, the interviewed beneficiaries believe they will be able to continue to apply what they have learned after program completion (a score of 4.64), thus confirming their appreciation for the quality and approach of the training received and the usefulness of the events/study tours in which they participated. Program Design, Implementation, and Efficiency A. Is the program on track to reach specified targets at midpoint and by the end of the program? The program has 36 indicators, including 20 result indicators and 16 activity indicators. The program is on track to reach one-third of specified targets at midpoint (12 indicators), and has already accomplished at least 100 percent of the end-of-program targets for three of these indicators. The program has achieved more than 50 percent of midpoint targets for an additional one-fifth (seven indicators) of total indicators, while for some two-fifths of indicators (or 14 indicators) the program has achieved less than 30 percent of target. For about one-fifth of indicators, or 7 indicators, the achievement rate is lower than 10 percent of target (see Tables 3 and 4). Dominican Republic – Exporting Quality Program Midterm Evaluation 15 • Specifically, what percent is on track and what percent is not on track? Below is a summary of performance given for each indicator, with different traffic light colors to highlight the percentage of accomplishment for each specific target. Table 3: Performance of result indicators at program midpoint Indicator ID Indicator Summary Score Against Target Result Indicator #1 (FFPr 17) # of individuals benefitting directly as a result of USDA assistance 77% Result Indicator #2 (FFPr 18) # of individuals benefitting indirectly as a result of USDA assistance 77% Result Indicator #3 (FFPr 16) # of individuals who have received short-term agricultural sector productivity or food security training 131% Result Indicator #4 (FFPr 1) # of hectares under improved techniques or technologies 28% Result Indicator #5 (FFPr 2) # of individuals who have applied new techniques or technologies 19% Result Indicator #6 (FFPr 3) # of individuals who have applied improved farm management practices 0% Result Indicator #7 (FFPr 13) Value of sales by project beneficiaries 105% Result Indicator #8 (FFPr 14) Volume of commodities (metric tons) sold by project beneficiaries 193% Result Indicator #9 Percentage of tests of target post-production agricultural products that come back clean 200% Result Indicator #10 # of direct beneficiaries achieving internationally recognized certifications 0% Result Indicator #11 Average percent reduction in time a product takes to get through problem points in a value chain 0% Result Indicator #12 # of partner packinghouses using a traceability system 0% Result Indicator #13 Quantity of post-harvest equipment and facilities improved 10% Result Indicator #14 # of food safety/standards-ready target commodities with improved marketing and branding 20% Result Indicator #15 (FFPr 9) Value of new public and private sector investment leveraged 46% Result Indicator #16 Value of buyer/seller contracts facilitated 37% Result Indicator #17 Average OCI score of target producer groups No end of Y2 target Result Indicator #18 % of target value chain groups that have a systematic way of disseminating and updating market information 0% Result Indicator #19 (FFPr 8) # of public–private partnerships formed 79% Result Indicator #20 (FFPr 7) # of enterprises and organizations that applied improved techniques and technologies 53% Color Legend: Result Indicator #1 is probably the indicator that best gauges the overall accomplishment of the program. This indicator aims at measuring the number of individuals directly participating in the USDA-funded interventions. The program has achieved 77 percent of this midpoint Green highlight Achieved or surpassed target Yellow highlight On track to achieve target Red highlight Has not met target Dominican Republic – Exporting Quality Program Midterm Evaluation 16 target. The performance analysis of this indicator also shows how the program, after a slow start, has taken off during the second half of Y2 implementation, as seen in Figure 7. The results of a few other performance indicators suggest that the program reacted positively to beneficiary demand for practical training on upgraded cultivation methods. The program provided several trainings on FSMA-related issues in response to increasing demand for information on that subject from beneficiaries, packers and exporters, and even producers. For instance, Result Indicator #3, a round-up of all activity training indicators (counting the number of individuals who have received short-term agricultural sector productivity or food security training) has exceeded its target by almost one-third. Most other activity indicators gauging training activities also show a very high level of performance achievement much higher than targets. Performance indicators on sales (Result Indicators #7 and #8) also show positive results. The actual performance of Result Indicator #9, which counts the percentage of tests of target post-production agricultural products that come back clean, is more difficult to evaluate. The problem here is that the program has only collected tests on the presence of salmonella in chilies, all of which came back 100 percent clean (an achievement in itself). However, no test has been carried out for the presence of other diseases and/or pest infestations that are known to be present in the value chains. Tests/measures showing declines of such infestations (and/or pesticide residue levels) over the period of implementation would in fact suggest a more robust positive result of program activities. Certain program activities—that have already been initiated but will count as results only when finalized—have low or even zero percentages of accomplishment. But they are Figure 7: Result Indicator #1: Achievement by Reporting Period Dominican Republic – Exporting Quality Program Midterm Evaluation 17 expected to be on track (or nearly so) with a certain delay due to late/slow start of program core activities (see the following section). They include indicators such as (i) beneficiaries achieving internationally recognized certifications (Result Indicator #10); (ii) the average percent reduction in time a product takes to get through problem points in a value chain (Result Indicator #11); (iii) packinghouses using a traceability system (Result Indicator #12); (iv) number of post-harvest equipment and facilities improved (Result Indicator #13); (v) number of food safety/standards-ready target commodities with improved marketing and branding (Result Indicator #14); (vi) value of new public and private sector investment leveraged (Result Indicator #15); (vii) value of buyer/seller contracts facilitated (Result Indicator #16); (viii) number of public–private partnerships formed as a result of USDA assistance (Result Indicator #19); and (ix), organizations/groups that applied improved techniques and technologies (Result Indicator #20). For these result indicators to be on track at program completion, the performance pace achieved during the second half of Y2 must be maintained for the remaining period of implementation. Less positive is the performance evaluation of certain other indicators—again, with low or even zero percent target achievement—that not only are not quite on track, but are not expected to be on track by the end of the program. It is therefore advisable to scale back expectations for completion of these objectives. These indicators include: (i) surveying beneficiaries who have applied new techniques or technologies (Result Indicators #4 and #5); (ii) assessing the success rate among beneficiaries who have applied improved farm management practices (Result Indicator #6); and (iii) identify the target value chain groups that have a systematic way of disseminating and updating market information (Result Indicator #18). A complete evaluation for each indicator is provided in Annex 3 of this report. • For targets not on track to be met, specify changes (if any) to the initial assumptions or within the operating context that warrant a review and recommended revision of the targets. A few changes have been proposed to modify the initial assumptions for Result Indicator #11, measuring the average percent reduction in time a product takes to get through problem points in a value chain. Certain activities of Result Indicator #11 have been regrouped to include two value chains, 9 while other activities have been refocused to take into consideration actual beneficiary demand (e.g., cocoa fermentation and processing facilities), changed regulatory environment (repeal of the medfly ban), and changing climate 9 For instance, (a) internationally recognized certification and (b) improved processing capacity now apply for both avocados and pineapples, while (c) IPM and (d) post-harvest treatment now apply for both Asian and greenhouse vegetables. Dominican Republic – Exporting Quality Program Midterm Evaluation 18 conditions (more, longer rainy seasons than expected). An alternative metric for the indicator result has been proposed when time-related measurement proved difficult to apply. As for the indicators that are expected to be not quite on track by the end of the program, it is advisable to revise and scale down the life-of-project targets. In particular it is recommended to (i) scale down by one-quarter the end of Y4 cumulative target of Result Indicator #4, setting the new end of Y4 cumulative target at 3,750 hectares under improved techniques or technologies as a result of USDA assistance; (ii) scale down by one-quarter the end of Y4 cumulative target of Result Indicator #5, with a new end of Y4 cumulative target of 4,500 producers who have introduced improved techniques or technologies as a result of USDA assistance; (iii) scale down by at least one-third the end of Y4 cumulative target of Result Indicator #6, setting the new end of Y4 cumulative target at 830 producers who have introduced improved farm management practices such as governance, administration, or financial management as a result of USDA assistance; (iv) scale down the end of Y4 cumulative target of Result Indicator #18 by at least half and reduce the denominator from 10 organizations to just five (either trade or producer associations, cooperatives, or clusters), with the new end of Y4 cumulative target set at 50 percent of target value chain groups that have a systematic way of disseminating and updating market information. For a few Activity Indicators with low performance scores, the main problem is that there was an initial misunderstanding regarding the number of trade organizations operating in the target value chains, thought to be seven but in fact only two. This has resulted in setting targets that the evaluation team considers unattainable, whereas they could be achieved by resetting them at the level of the initial rationale, that is, by reducing them by a ratio of 7 to 2 (i.e., by 3.5 times); or by enlarging the scope of the indicator to include not only trade associations but also producer associations, cooperatives, and clusters. For instance, (i) for Activity Indicator #2.4—counting the number of trade association members benefitting from the end-market assessments—it is recommended to reduce the end of Y4 cumulative target to 40 members (rather than 140); (ii) for Activity Indicator #2.5—counting the number of trade association individuals benefitting from trade association capacity building activities—it is recommended to maintain the end of Y4 cumulative target but also include in the computation of this indicator producer associations, cooperatives, and clusters; (iii) for Activity Indicator #2.6—counting the number of women in trade association management who participate in program-sponsored leadership training and events—the end of Y4 cumulative target could be reset at 30 (rather than 107) women or, as an alternative, enlarge the goal to include other types of organizations; and, (iv) for Activity Indicator #2.7—counting the value of grants provided to trade associations—it is recommended that the scope of the indicator be enlarged to include other types of organizations as above. Dominican Republic – Exporting Quality Program Midterm Evaluation 19 Table 4: Performance of Activity Indicators at Program Midpoint Indicator ID Indicator Summary Score Against Target Activity Indicator #1.1 # of trainings delivered to increase capacity of fruit and vegetable operators 700% Activity Indicator #1.2 # of producers benefitting from training and technical assistance 214% Activity Indicator #1.3 # of processors benefitting from training and technical assistance 181% Activity Indicator #1.4 # of packinghouse staff benefitting from training and technical assistance 61% Activity Indicator #1.5 Value of grants provided to producers 14% Activity Indicator #2.1 # of processors/exporters trained in marketing and branding techniques 123% Activity Indicator #2.2 # of direct beneficiaries participating in trade shows, buyer visits to the DR, and exporter visits to the United States and Europe 613% Activity Indicator #2.3 # of producer organization members benefitting from end-market assessments 215% Activity Indicator #2.4 # of trade association members benefitting from the end-market assessments 18% Activity Indicator #2.5 # of trade association individuals benefitting from trade association capacity building activities 0% Activity Indicator #2.6 # of women in trade association management who participate in program-sponsored leadership training and events 143% / 5% Activity Indicator #2.7 Value of grants provided to trade associations 5% Activity Indicator #3.1 # of individuals attending cold chain knowledge sharing events 370% Activity Indicator #4.1 # of fruit processors benefitting from in-kind equipment to create value added products 50% Activity Indicator #4.2 Value of in-kind grants provided to processors 42% Activity Indicator #5.1 # of individuals reached by public information campaigns 60% Color Legend: B. What have been the primary reasons for current achievements compared to result and activity indicator targets? The program had a slow start due to setup, staffing, and establishing suitable long-term accommodations. In practical terms, most, if not all, activities in the first six months of program implementation focused on setup activities; core activities only started during the second half of year one. Core activities carried out during the second part of year one and first part of Y2 include: (i) carrying out value chain analysis for each target commodity and a cold chain assessment; (ii) performing a few study tours abroad; (iii) organizing several training sessions, workshops, and buyer visits; and (iv) participating in various events. It was in the second part of Y2 that program activities began to flourish, with plenty of training on farming methods, farmer field schools, workshops on FSMA issues, IPM-related activities, Green highlight Achieved or surpassed target Yellow highlight On track to achieve target Red highlight Has not met target Dominican Republic – Exporting Quality Program Midterm Evaluation 20 OCI, and so on. The training approach, focusing on practical demonstrations, has been a driver of success and much appreciated, especially by farmers. • Specifically, what should be maintained and/or scaled up? Analysis of program activities and performance suggests that the program is a demand￾driven program that has provided more training than expected and thus has exceeded Y4 targets for a few result indicators. It is advisable to scale up a few target indicators related to training activities. This scaling up would de facto compensate for the scaling down recommended for four result indicators and the modifications suggested for an additional four activity indicators (concerning trade organization definitions). Specifically, the following: • The end of Y4 cumulative target for Result Indicator #3—counting the number of individuals who have received short-term agricultural sector productivity or food security training—could be increased by one-quarter, therefore training a cumulative 8,500 individuals; • The end of Y4 cumulative target for Activity Indicator #1.1—counting the number of trainings delivered to increase capacity of fruit and vegetable producers, processors, and packinghouses—could be expanded by seven times, so the training target would be set at 120; • The end of Y4 cumulative targets for Activity Indicators #1.2 and #1.3—counting the number of producers and processors benefitting from training and technical assistance, respectively—could go up by 50 percent and 100 percent, setting the new targets at 4,650 producers and 150 processors, respectively; • The end of Y4 cumulative target for Activity Indicator #2.1—counting the number of processors/exporters trained in marketing and branding techniques—could be increased by one-quarter, which would set the target at 600 individuals; • The end of Y4 cumulative target for Activity Indicator #2.2—counting the number of direct beneficiaries participating in trade shows, buyer visits to the DR, and exporter visits to the United States and Europe—could be doubled, thus set at 200 individuals; • The end of Y4 cumulative target for Activity Indicator #2.3—counting the number of producer organization members benefitting from end-market assessments—could be tripled, reset at 180 individuals;10 • The end of Y4 cumulative target for Activity Indicator #3.1—counting the number of individuals attending cold chain knowledge sharing events—could also double, with the new target set at 750. 10 This indicator could in fact be merged with Activity Indicator #2.4 if the trade organization definition is enlarged to include other types of organizations. Dominican Republic – Exporting Quality Program Midterm Evaluation 21 C. What Have Been the Major Challenges to Program Implementation and How Has the Program Responded to These Challenges? A major challenge to program implementation has been the recruitment of suitable candidates for the key position of value chain facilitator. The recruitment process has taken longer than expected and this has caused a delay for the start of the program’s core activities. In practical terms, no beneficiary related activities were recorded during the first six months. In addition, it took longer than expected to fine-tune the program activity delivery machine and the program only saw progress in the second half of Y2 in particular thanks to (i) the organization and delivery of a series of practical trainings focusing on farming practices and methods (pruning, fertilization, irrigation, GAP, etc.) and targeting producers; (ii) workshops providing useful information on FSMA-related issues targeting all actors in the target value chains; (iii) practical training on good handling practices targeting packinghouse staff and good manufacturing practices targeting processors; and (iv) seminars on strengthening organizational capacity for a selected number of private partners, including producer and trade organizations, cooperatives, and clusters. An additional challenge has been the organization and delivery of program activities via a public information campaign, the results of which have been largely below expectations for at least the first 18 months of program implementation. This resulted in a change of personnel in charge of such activities in the latter part of Y2. Implementation of activities related to the dispersal of improved market information was also problematic. The program has started directly disseminating the weekly Emergency Action Notifications (EAN) on pest border detentions provided by USDA Animal and Plant Health Inspection Service (APHIS) and has included market information in weekly emails regarding prices for selected Dominican products in different terminal markets in the United States. However, the program is still looking for suitable means or an interested group/organization to continue distributing this information. The program also expects to publish relevant information through a webpage of the ministry of agriculture’s food safety department; that webpage, however, is still in the design phase. • Specifically, to what extent has the Mediterranean fruit fly ban affected program implementation? On March 18, 2015, USDA APHIS issued import restrictions on “host commodities from the Dominican Republic into or through the United States,” upon detection of the medfly in the Punta Cana area on the eastern coast of the DR. Among the banned produce were all avocado varieties produced in the country, though in late April 2015 shipments of Hass variety avocados were allowed, as APHIS does not consider Hass avocados a fruit fly host. Peppers and tomatoes were banned too, though in late April shipments of green tomatoes Dominican Republic – Exporting Quality Program Midterm Evaluation 22 were allowed. The export ban was repealed on January 7, 2016 for produce originating from 23 provinces and, in December 2017, for the remaining seven provinces of the DR.11 The medfly ban was the main subject of awareness for most potential beneficiaries during the first months of program implementation. It has continued to negatively affect a significant number of beneficiaries dealing with avocados, tomatoes, and both bell and chili peppers. The vegetable producers of Jarabacoa, in particular, most of whom incurred large, unsustainable debts, had to cease or significantly scale down production.12 The program has been providing support to greenhouse producers in Rancho Arriba (in the province of San José de Ocoa) as well as chili producers in the city of La Vega, to assist them in increasing exports and export value, though it has been less successful in reaching out to greenhouse producers in Jarabacoa and Constanza (both located in La Vega province). • To what extent did unexpected heavy rains in the fall of 2016 affect Asian vegetable exports? The program has helped Asian vegetable producers and exporters through training on IPM, GAP, GHP, the strengthening of organizational capacity, and the upgrading of existing traceability systems to conform to FSMA requirements. In the fall of 2016, heavy rains hit the DR and several locations in the country were inundated. This has particularly affected producers of Asian vegetables, whose production has been partly damaged. The 2017 hurricanes, three of which had significant landfalls in the DR, have had a similar effect. Hurricane Harvey caused heavy, persistent rains that negatively affected Asian vegetable output and quality. D. What, if any, unintended but important direct results (positive or negative) have occurred? The program has been working closely with other USDA-financed programs, namely Food Safety Agro Sustainability Training (FAST) and the Participatory Agency Program Agreement for Sanitary and Phytosanitary Standards in the DR and Central America, to prepare the target commodity industry for FSMA compliance, a synergy appreciated by USDA officials. Another unintended but important direct result has been preparation of the national interpretation guide for GLOBALG.A.P. certification. A primary objective of program activities 11 Beneficiaries of the EQ program are located in just one of these nine provinces, Monte Plata, and mainly produce pineapples, a commodity not affected by the ban. 12 Greenhouse vegetables are mainly produced in three areas of the DR: (i) Rancho Arriba and Ocoa; (ii) Jarabacoa; and (iii) Constanza. The first two areas specialize in the production of bell peppers mainly for the U.S. market, while the third is more oriented toward production of cucumbers, while all three produce tomatoes, mainly for the domestic market. The medfly ban principally affected producers of U.S.-bound bell peppers. Dominican Republic – Exporting Quality Program Midterm Evaluation 23 is to obtain internationally recognized certifications. One of these certifications is GLOBALG.A.P., increasingly needed if target beneficiaries want to export avocados and pineapples to European (as well as other) markets. GLOBALG.A.P. universal standards generally apply to all countries, but it is possible to adapt them to the characteristics of a specific country (e.g., the DR) through a process of localized interpretation that provides guidance to certification bodies and producers on how best to implement GLOBALG.A.P. control points and compliance criteria at the national level. This is done through the drafting of a national interpretation guide that must be approved by the GLOBALG.A.P. secretariat and the technical committee for crops. • How should these results be assessed in the overall program context? Training on FSMA-related issues is among the most appreciated activities that the program delivers to producers, packinghouse owners, and exporters. Approval of the GLOBALG.A.P. national interpretation guide would be a significant direct result of program activities that are expected to facilitate increased GLOBALG.A.P. certification in the DR and increase exports (and export value) to European markets. • How has the program responded to these guidelines? The program has facilitated adoption of this GLOBALG.A.P. national interpretation guide by providing an international consultant to support the national technical working group (based at the local partner, CEDAF) responsible for drafting such a guide. Approval of the guide is expected soon. E. To what extent are staffing, management, and oversight costs suitable given the number/scope of activities carried out? The program has a total staff of 20 employees plus two collaborators employed by the local partner, CEDAF. Program chief of staff is Brian Rudert, the only U.S. citizen on the team—the others include one French national and the remainder are Dominican. Half of the employees are women; one of the program’s four senior positions is filled by a woman. About half of the positions are administrative, with the remaining half being mainly operational, including four value chain facilitators, two thematic specialists, and a training analyst who covers most of the program’s field activities. A second training analyst has been employed since the beginning of year three. The important task of program communication is covered by one media specialist and one assistant; both these positions were re-staffed during the second half of Y2. Over the first two years of program implementation, there has been limited staff turnover, with changes occurring in a total of only four positions. Staffing of the program has seen a Dominican Republic – Exporting Quality Program Midterm Evaluation 24 slow start with just one-third of the positions filled during the first six months of program activities. Key value chain facilitator positions were only filled during the second half of year one. The number of program employees appears adequate; employment of a second training supervisor seems reasonable taking into consideration the escalated number of activities during the second year that are envisaged to continue at least at the same, improved, pace during the remaining program life. Program management also seems adequate to satisfy program needs; after the above-mentioned slow start the program seems to have found a suitable pace that, if maintained, will result in fulfillment of most of the program’s target results. Viewed this way, the minor change in the management organization during the second year is a move to consolidate the accelerated pace of program implementation.13 The program changed its “brand” name to Exporting Quality; in Spanish “Exporta Calidad” seemed easier to remember and recognize than the original name (“Exportando Calidad e Inocuidad”). The name change, together with the adoption of a new logo resembling a quality seal, are intended to improve the understanding of program objectives and create a more immediate visual identification of the program. No significant oversight costs were identified during this program midterm evaluation. The program currently occupies the entire second floor of the building hosting CEDAF. Relevance of Interventions F. To what extent do program activities address the core issues of target beneficiaries? The issue of the relevance of interventions has been addressed through interviews with key stakeholders, focus groups, and a survey in the form of a questionnaire. There is general acknowledgement among beneficiaries that program activities addressed their core concerns.14 Small-scale producers15 showed deep appreciation of the program’s training 13 The training and communication units now report directly to the M&E manager; the market linkage facilitator is to report directly to the chief of party. 14 The program has carried out the following core activities: Avocados: low small farmer productivity, diversify market opportunities by expanding exports to European markets for green skinned avocados, help expand Hass and other plantings along border with Haiti, expand certifications, strengthen producer organizations through OCI, traceability; Pineapples: support continued expansion of plantings, expand access to European premium markets, invest in increased processing capacity, certifications, strengthen SPS system to prevent entry of pests and diseases and promote rational pesticide use, country and company branding, strengthen producer organizations through OCI; Asian vegetables: reduce expensive European and U.S.-border detentions because of pests, prepare for FSMA compliance, strengthen producer organizations through OCI, improve post-harvest handling, traceability; Greenhouse vegetables: reduce expensive European and U.S.-border detentions because of pests, prepare for FSMA compliance, strengthen producer organizations through OCI post-harvest; Cocoa: low small farmer productivity, ensure that new plantings use grafted plants and preserve specialty genetic base, create local market value added opportunities. Dominican Republic – Exporting Quality Program Midterm Evaluation 25 activities dealing with good agricultural practices (GAP) and applied farming methods and technologies. Small-holder avocado producers were satisfied with their participation in practical training activities dealing with pruning, fertilization, land drainage, irrigation, and GAP; pineapple producers were more than pleased to have learned the up-to-date cultivation methods and handling technologies applied in Costa Rica (during the study tour and subsequent briefing workshops presenting the visit results); exporters of Asian vegetables appreciated that the program has focused on their needs and not only those of the producers; the producers and exporters of greenhouse vegetables acknowledged the importance of the study tour in the United States, where they were able to acquire knowledge of new production techniques and packing technologies, as well as meet with potential buyers; processors of cocoa products were mostly satisfied with the practical training received on good manufacturing practices that enlarged their understanding of issues ranging from contamination and cross-contamination of residues and the importance of constant low temperature for the quality of chocolate products. Effectiveness of Interventions G. Which interventions/activities have the highest potential/are most effective for achieving the program’s higher-level results (increased agricultural productivity and market transactions)? Analysis of the survey results and focus group discussions suggest that beneficiaries believe that the most effective approach for achieving the program’s higher level results is to: (i) continue with the practical training and ad hoc study tours focusing on improved agricultural methods and practices (pruning, fertilization, land drainage, irrigation, IPM) for achieving greater agricultural productivity; (ii) continue workshops on FSMA-related issues to get needed, relevant information for continuing to export to the U.S. market, and the provision of assistance for installing traceability systems; (iii) continue to provide assistance to get internationally recognized certifications (GLOBALG.A.P., fair trade) and grants to improve handling and processing of the target commodities; and, (iv) continue with buyer visits for direct contact with importers, ensuring more market transactions, and ad hoc study tours providing up-to-date information on applied technologies in each of the target value chains. 15 Small-scale producers are producers operating at a small scale, used to distinguish from industrialized producers. The distinction between small- and large- scale producers is arbitrary. What is considered small-scale in one country may be considered large-scale in another. In DR small-scale producers can be considered those cultivating less than 3 hectares of land, but less than 5 hectares for cocoa producers and less than 0.5 hectare for greenhouses. Dominican Republic – Exporting Quality Program Midterm Evaluation 26 H. How effective is the program at reaching women beneficiaries? What could be done to improve women’s participation? This gender issue has been investigated through interviews with key actors, focus groups, and a survey in the form of a questionnaire. In general, beneficiaries have reported that the program has been very effective at reaching women beneficiaries, as shown by the great success of the three leadership workshops held during the first two years of program implementation. Among beneficiaries, there is a relatively high number of cocoa, Asian, and greenhouse vegetable women owners and operators. Most appreciation for program activity on gender issues comes, as expected, from women beneficiaries, while not all male beneficiaries agree on the positive aspects of this achievement. As for what can be done to improve women’s participation, there is consensus on letting women take more responsibility in the various value chain activities and disseminating relevant information on women’s roles and achievements in specific value chains, as well as supporting women’s associations working on target value chains. I. How effective has the training approach been to build and/or enhance capacity of target beneficiaries? The interviewed beneficiaries have generally described most of the practical activities in which they took part as very effective. Most farmers have shown great appreciation for training activities such as pruning, fertilization, land drainage and irrigation methods, good agricultural practices, and IPM. Beneficiaries also mentioned the great utility of the training they received, praising their practical aspects especially compared with training provided by other programs in which they had participated in the past. Direct observations of farming methods used in Costa Rican pineapple plantations and the efficiency of packinghouse staff there in handling pineapples were considered very successful. Packers expressed the greatest appreciation for the training received in good handling practices and the various trainings and workshops on FSMA issues. The six packinghouses that have applied for installation of the traceability system are eager to get the software installed and become ready to continue exporting to the United States by complying with the new FSMA requirements. Cocoa processors found practical activities on good manufacturing practices (e.g., contamination and cross contamination and constant cold temperature for chocolate products) very useful. As for cold chain awareness, beneficiaries particularly appreciated the study tour visits and ad hoc training on issues such as sliding doors, energy savings resulting from using suitable flooring, and advice on the avoidance of handling of produce at night when the risk of insect infestation is higher. Exporters acknowledged the utility of study tours and buyer visits, as well as the importance for farmers and packinghouses to get internationally recognized certifications. Dominican Republic – Exporting Quality Program Midterm Evaluation 27 J. To what extent do public authorities and private sector partners have an influence (positive or negative) on the achievement of program objectives? The program has actively engaged with both public and private partners. The main public partners were the phytosanitary and food safety departments of the ministry of agriculture. When interviewed, both directors expressed appreciation for the fruitful collaboration with the program, though they think more should have been done by the program. The phytosanitary director said that beneficiaries were unsatisfied with program activities/achievements and that they expected much more from the program. Indeed, he believes that not enough has been done to fight diseases and pests such as thrips, an insect that continue to infest both Asian and greenhouse vegetable crops. 16 He greatly appreciated the Asian vegetable IPM demonstration farm activity in La Vega but expected this activity to be extended to other farms. 17 He would also like the program to assist in keeping non-native pests and diseases out of the DR, in particular for cocoa and pineapple crops. In addition, he would like the program to reorient the activities of the last two years of implementation to make available more physical facilities while continuing to provide practical training activities. As for the director of food safety, he acknowledged that the synergy with the program has improved during its second year of implementation. The program has entered an alliance with the ministry of agriculture’s food safety directorate on various issues such as training on post-harvest handling and food safety. The most important of these issues is, specifically, salmonella testing and a related GAP/GHP training program that has focused on a few DR firms on the FDA automatic detention list because of salmonella contamination. The number of border detentions for residual pesticides has been steadily reduced over the last few years, for instance, there were 18 in 2015, but just four so far in 2017. The food safety director would like to see its department officials more involved in program activities and would appreciate program assistance in obtaining an international recognition/certification of low cadmium content in DR cocoa products. Other public partners include the National Council for Competitiveness (CNC), a partner for cold chain activities, the Dominican Center for Export and Investment (CEI-RD), a partner for the Agri-Food Fair (“Agroalimentaria”) in Santo Domingo, and “Pro-Consumidor,” a consumer association partner for the program’s public campaign. CEDAF, the main local implementing partner of the program, has been actively participating at most program activities and is already preparing to continue most of the program activities 16 Some 70 border detention notifications were received during the first 11 months of 2017 (approximately as many as in previous years), more than half of them related to thrips infestation. 17 The program plans to expand support to five additional IPM demonstration farms in the upcoming reporting period. Dominican Republic – Exporting Quality Program Midterm Evaluation 28 at the end of the fourth year of implementation. The CEDAF director acknowledged full cooperation and synergy with the program and its activities. CEDAF is very active in performing the organizational capacity strengthening of some of the program’s private partner organizations as well as carrying out farmer field school activities for various target commodities and training on post-harvest handling of produce (FSNA, food safety, GHP, etc.). In addition, CEDAF is already the DR reference organization for GLOBALG.A.P. certification (it hosts the relevant working group for drafting the national interpretation guide), and it is upgrading an existing (but simpler) traceability system to be FSMA-compliant that is expected to be ready for installation in six packinghouses by March 2018. Other private partners include the Association of Dominican Exporters (ADOEXPO), the Dominican Business Association (JAD), the Agriculture University in Santiago (ISA), a partner for postharvest activities, the cocoa cooperative CONACADO and the Foundation for Organic Cocoa, FUPAROCA, partners for cocoa farmer field schools, and seven value chain organizations where the OCI methodology is being applied. All these partners, both public and private, had a positive influence on the achievement of program objectives. Some of them are part of the program steering committee, while all of them have participated in various program activities and events. Sustainability Efforts to Date K. What factors contribute to sustainability of program results and how has the program focused on these? A few factors contributing to the sustainability of program results are compared with the program activities and achievements to date. The program responds to clear needs. There is a clear need for the provision of technical assistance to operators of the target commodities, ranging from the need for improved farming techniques and practices for all value chain producers, to getting GLOBALG.A.P. certification for avocados, a fruit for which supply largely exceed demand, and pineapples, for which future supply is expected to exceed demand; to the introduction of IPM for Asian and greenhouse vegetables; to preparations for taking on commercial challenges faced by all target value chain actors; to the need to add value through wet fermentation to that of improving processing knowledge and practices for small cocoa associations. Presence of the “train the trainers” component. This has been one of the most appreciated training components to date, particularly for farming techniques and practices. It includes training by consultants (both foreign and local) and farmer field schools, where experienced farmers are expected to train other farmers. The program has also supported the training of Preventive Controls Qualified Individuals (PCQIs) who must prepare food Dominican Republic – Exporting Quality Program Midterm Evaluation 29 safety plans needed by most food processors and packinghouses for export to the United States.18 There has been general satisfaction among beneficiaries regarding the utility of such practical activities and recurring demand to continue and expand them. Flexibility and adaptability. The program has shown sufficient flexibility and adaptability to respond to the changing environment and meet beneficiary demands. This is suggested by the change of focus toward activities most demanded by beneficiaries, such as hands-on training in farming practices and workshops on FSMA regulations and cold chain awareness, as well as study tours to learn more advanced techniques, practices, and applied technologies. Collaborative partners. Local partners have shown considerable collaboration in program implementation and taken appropriate actions to ensure sustainability for most of the activities started by the program, in particular farmer field schools, traceability systems, certification (GLOBALG.A.P.), strengthening organizational capacity of relevant groups, organization of women in trade seminars, and dissemination of bulletins on border detections that provide details about shipments not allowed to enter the United States because of pests/diseases found in the produce. Results orientation and program evaluation. The program has 36 indicators to gauge its performance both at its midpoint and at the end of program implementation. This assures that the program follows, as much as possible, an approach oriented toward the achievement of results. In addition, the program evaluation aims at keeping the project’s goals and outcomes on track and helps the chief of party identify if something is not working as expected and then reorient the program toward achievable results. Strong Leadership. The program has strong leadership from both the chief of party and senior staff as well as appropriate and helpful background support from IESC in Washington, DC. The program uses local staff. Most staff is local, with foreigners mainly providing short￾term input on specific issues. The program has also made significant use of local consultancies for implementing specific activities, such as farming techniques, farmer field schools, and strengthening the commercial capacity of groups. All these aspects favor sustainability in the medium- to long-term. Support from local government, policy makers, and business. To date there has been significant support from local government, policy makers, and businesses that are among the 18 FSMA requires that facilities have a food safety plan implemented by the deadline date; the plan must be prepared and applied by a properly trained PCQI. Dominican Republic – Exporting Quality Program Midterm Evaluation 30 major beneficiaries of the program, as demonstrated by the large number of public extension officers trained by the program. High visibility in communities. This aspect is probably the weakest of the factors listed here. Beneficiaries sometimes struggle to understand the role of the program in delivering the training or other activities in which they have participated. This is partly since many activities have taken place in collaboration with other institutions or programs, sometimes even under their names. This weakness has been already considered by the program with changes in the communication team and a strategy to improve program visibility, but more must be done in the remaining two years of program implementation to improve the effectiveness of the communication activities. L. Is there a well-developed exit strategy? A program exit strategy is a plan describing how the program intends to withdraw its resources while ensuring that achievement of the program’s goals is not jeopardized and that progress toward these goals will continue. The goal of an exit strategy is to ensure the sustainability of a program’s objectives after its completion. Ongoing and timely monitoring of benchmarks is critical to the successful implementation of exit strategies and the monitoring of exit strategy benchmarks should be integrated into the program’s overall monitoring and evaluation plan. This can prevent duplication of monitoring efforts and maximize use of existing data. To date, no exit strategy plan has been prepared by the program. The chief of party expects that the long-term sustainability of the program will be assured by continuation of program activities by CEDAF, the local partner. In this respect, the preparation and approval of the GLOBALG.A.P. national interpretation guide is expected to ensure more internationally recognized certifications in the next few years and after program completion, while implementation of the organizational capacity methodology should result in strengthening the commercial capacity of an extended number of organizations after the program’s end. Also expected is continuation of the activities of farmer field schools for all target commodities; these have been a significant success among producers to date and are easy to replicate with relatively few additional funds. Installation of the improved traceability system is also expected to be extended to many more exporters who are eager to get the help of a modern instrument to comply with FSMA requirements. Private sector partner ADOEXPO is expected to continue activities for women in trade by organizing at least one additional workshop. Value chain aggregators who have incentives to adopt new practices are expected to continue to apply IPM and treatment with hot water to eliminate post-harvest pests and diseases. The food safety department of the ministry of Dominican Republic – Exporting Quality Program Midterm Evaluation 31 agriculture is, on its side, eager to continue disseminating the weekly EAN on pest border detentions provided by USDA APHIS. During the first two years of implementation, the program has shown itself to be both flexible and demand-driven. Most activities for the remaining two years of the program are already planned and they are expected to be performed as planned. Nonetheless, the evaluation exercise has permitted the beneficiaries to identify what activities are most needed from their points of view. In addition, they have requested that the program facilitate a few additional activities that could be carried out in the remaining time and that would benefit most of the beneficiaries. Producers were particularly eager that the program continue to provide training activities on farming methods (pruning, fertilization, irrigation, land preparation, drainage) and practices (such as good agricultural practices and IPM) and maintain the same practical approach that has been very much appreciated. Study tours in Costa Rica were greatly appreciated by pineapple producers, where they learned farming practices that are now applied in their farms and witnessed the high efficiency of Costa Rican packinghouses in handling the produce. They requested more study tour opportunities for other producers. Packers requested that the program continue to provide training on GAP to producers, as well organize more workshops providing practical information on FSMA-related issues. They requested more training on post-harvest handling practices for their staff as well, and anticipated that installation of the traceability system upgraded with program support be completed soon. Processors of cocoa products requested more ad hoc practical training on good manufacturing practices, as well as assistance on organizational issues and commercial practices. All of them were eager to receive grants to procure needed processing equipment. Exporters were for the most part satisfied with study tours that permitted them to be in contact with importers and observe new technologies applied abroad. Buyer visits were also appreciated, though up to now they resulted in fewer commercial deals than expected and hoped for. Exporters also believed their training in the impact and usefulness of cold chain awareness and related activities was significant, including new information on floor insulation (which can reduce energy costs by up to 20%), pre-cooling and cooling storage within 1–2 hours of harvesting, cooling facilities, screen doors, equivalence of regulations, and improved knowledge of temperature and humidity management. Dominican Republic – Exporting Quality Program Midterm Evaluation 32 During the focus group discussions, a few relatively new issues and related requests for program assistance were put forward by participants. These might become part of the new activities to be carried out by the program in its remaining period of implementation. These issues are discussed in the following section, followed by the summarized list of recommendations for the remainder of the program. The DR is part of the Dominican Republic–Central America–United States Free Trade Agreement (DR-CAFTA) between the United States, five Central American countries (Costa Rica, El Salvador, Guatemala, Honduras, and Nicaragua) and the DR. The DR is also part of CARIFORUM, a free trade agreement between the fourteen countries of the Caribbean Community, or CARICOM, and the DR. These two agreements aim at creating new economic opportunities by eliminating tariffs, opening markets, reducing barriers to services, promoting transparency, facilitating trade and investment among the signatory countries, and furthering regional integration. As many as 40 million individuals reside in a total of 28 different island nations in the Caribbean, each of which possesses its own unique economic characteristics. However, trade among them is limited; for instance, the DR exports only US$180 million in goods to other Caribbean countries in an average year. One possible explanation for this limited trade flow might be a presumed lack of adequate transport services, but studies have found that, in fact, there is no lack of direct services among the sub-regions. In addition, due to the increase in transshipment services, almost all countries in the Caribbean area have different options to trade with each other. The main explanation for low trade levels seems instead to be found in production patterns, history, and language barriers. Transport services between the Caribbean islands are, however, costlier and far less frequent than those to and from North America (e.g., Miami). This difference should become less relevant as transshipment within the Caribbean area, between countries such as the DR, Jamaica, and Panama, gains market share vis-à-vis transshipment to Miami. Such changes, as well as improvements in DR port efficiency and information dissemination about available transport options, as well as non-transport-related initiatives, could potentially help increase trade between the DR, CARICOM, and CAFTA’s Central American members. It is therefore recommended that the program assess the possibility of expanding exports of fruit and vegetables to the Caribbean islands, study which are the main obstacles to trade and challenges for exporters, and propose solutions for expanding trade. Further support activities could then be proposed based on the study’s results and recommendations. Stakeholders asked that the program further support small-holder producers in getting internationally recognized certification by providing financial support for building/refurnishing facilities needed for their farms to conform to GAP and HACCP/HARPC requirements. These Dominican Republic – Exporting Quality Program Midterm Evaluation 33 facilities include restrooms, storage depots, signs, etc. The program already provides producers with grants covering consulting expenditures for obtaining GLOBALG.A.P. certification; the relevant annual program statements for grant opportunities could be therefore modified to include the building/refurbishing of needed facilities in the expenditures covered by the program’s financial support in the form of grants. Packers and producers have asked the program to assist them in finding a solution to the increasingly pressing problem of water sanitation. Using untreated water can cause rejection of the exported produce due to sanitary problems. This is a problem that must be taken in account because of increasing water pollution as well as the introduction of more stringent requirements for allowing exports of produce (e.g., FSMA regulations). The program could help in assessing this water sanitation problem and verifying the availability of water treatment systems in the country, their effectiveness, their installation, and procurement costs. For instance, it is expected that a water treatment system based on reverse osmosis should be easily available at reasonable costs in the DR. The program may also consider using available grants for covering part of the expenditure to install similar water treatment systems in beneficiary facilities (packinghouses, farms, and processing units) as part of the process of gaining internationally recognized certification. It is recommended that the program carry out such an assessment on water treatment systems. A few beneficiaries have requested that the program design an activity to facilitate the recovery of overdue payments for shipments of fruit and vegetables to the United States. Beneficiaries are aware of the existence of the Perishable Agricultural Commodities Act (PACA) system in the United States, but believe that they need a faster solution to solve this overdue payment issue.19 The problem is that by the time a solution is found through PACA assistance, the typical DR exporter may have already gone bankrupt. Leading beneficiaries have asked the program for help in finding a possible solution to such overdue payment issues that takes into consideration specific characteristics of Dominican exporters. It is recommended that the program take steps to verify whether and how it can help beneficiaries in finding a viable solution to overcome such a problem. 19 PACA was enacted at the request of the fruit and vegetable industry to promote fair trade in the industry. PACA protects businesses dealing in fresh and frozen fruits and vegetables by establishing and enforcing a code of fair business practices and by helping companies resolve business disputes. PACA usually receives requests from companies for assistance with problems unique to the industry, such as interpretation of inspection certificates, advice on contract disputes, and bankruptcy payments. Working in partnership with the fruit and vegetable industry, PACA facilitates fair trade practices through education, mediation, arbitration, licensing, and enforcement. Dominican Republic – Exporting Quality Program Midterm Evaluation 34 Implementation of the following activities is recommended, along with the planned activities of the program: 1) Continue current training activities focusing on practical demonstrations, which are especially useful for low productivity small producers. 2) Maintain support for producers willing to get internationally recognized certification with grants, possibly being more flexible in setting the application requirements. The evaluator recommends that the program also provide grants for the building/refurbishing of basic facilities such as toilets, storage depots, etc., to facilitate acquisition of the certification. 3) Continue workshops dealing with FSMA issues, as well as facilitate installation of the improved traceability system for as many beneficiaries as possible. 4) Retain support for processors willing to invest in upgraded equipment to add value to DR produce, possibly being more accommodating in setting the application requirements, taking into consideration the financial and patrimonial situation of small associations/processors. 5) Continue the facilitation of study tours even if the Y4 target has been already achieved, because they are very useful for beneficiaries in terms of updating/upgrading their knowledge through practical, direct observation, and because they often come at no extra cost for the program (most beneficiaries being willing to pay for the tour themselves). 6) Focus on developing a strategy for getting beneficiaries involved in the application of improved farm management practices. 7) Focus on post-harvest handling techniques and processes because DR vegetables have more quality problems and a shorter shelf-life than those produced in other countries. 8) Focus on IPM activities, because the sanitary conditions of most vegetables produced in the DR continue to be poor and hence unsuitable for export. 9) Focus on end-market activities to establish increased linkages with importers and attain increased export sales of target commodities. 10) Focus on strengthening the organizational and commercial capacity of newly formed associations and of associations that never received assistance from international funded programs. The evaluator believes that assistance provided to these types of organizations will be more successful and long-lasting than assistance provided to other groups. Dominican Republic – Exporting Quality Program Midterm Evaluation 35 11) Follow up on the above activities benefitting beneficiaries during the remaining period and take the necessary exit strategy steps to ensure they continue to be carried out once the program is over. 12) As for performance indicators, a few program activities with low or zero achievement score have already been started and will bear fruit in the next reporting period, while other planned activities are not as easily carried out in a satisfactory way and need to be scaled down. A few indicators can be scaled up, instead. 13) Scale down the end of Y4 cumulative targets of a few Result Indicators: Result Indicator #4 to be set at 3,750 hectares under improved techniques or technologies; Result Indicator #5 at 4,500 producers who have introduced improved techniques or technologies; Result Indicator #6 at 830 producers who have introduced improved farm management practices; Result Indicator #18 at 50 percent of target value chain groups that have a systematic way of disseminating and updating market information. 14) Rephrase “trade organizations” as “organizations trading target commodities” for a few Activity Indicators to include producer organizations. As an alternative, (i) reduce substantially (i.e., by a ratio of 7 to 2) the end of Y4 cumulative target of Activity Indicator #2.4 (setting the new target at 40 members rather than 140); and that of Activity Indicator #2.6 at 30 (rather than 107) women or, as an option, enlarge the purpose to other organizations; or (ii) enlarge the scope of the indicator to include not only trade associations but also producer associations, cooperatives, and clusters, for both Activity Indicators #2.5 and #2.7. 15) Scale up the end of Y4 cumulative target of a few performance indicators: (i) Result Indicator #3 to train a cumulative number of 8,500 individuals; (ii) Activity Indicator #1.1 set at 120 trainings; (iii) Activity Indicators #1.2 and # 1.3 at, respectively, 4,650 producers and 150 processors; (iv) Activity Indicator #2.1 at 600 individuals; (v) Activity Indicator #2.2 at 200 individuals; (vi) Activity Indicator #2.3 at 180 individuals; and (vii) Activity Indicator #3.1 at 750 individuals. 16) Perform a market assessment on the possibility of expanding exports of target fruit and vegetables to other Caribbean islands. 17) Make an assessment of how to procure water treatment systems for packinghouses and processors as well as producers. 18) Verify the possibility and viability of assessing how to facilitate the resolution of business disputes related to the overdue payments for fruit and vegetable shipments to the United States as well as to other countries. 19) Prepare an exit strategy assuring sustainability of activities after program completion. Dominican Republic – Exporting Quality Program Midterm Evaluation 36 In addition, it is recommended to amend Annex E of the agreement signed between IESC and USDA to reflect the actual OCI scoring system that uses a scale of 1 to 100; and to set the end of Y4 target of Result Indicator #17 at 85 rather than 3 as in Annex E, with no intermediate targets. Finally, it is recommended that USDA authorize the obtaining of a FSMA Preparedness Assessment facilitated by the program as a certification valid for being counted in Result Indicator #10. Annex 1: Scope of Work for Midterm Performance Evaluation of EQ Program 1 The Exporting Quality (EQ) Program is a program in the Dominican Republic that aims to increase productivity and sales for domestic and export markets of high-value fruit and vegetable global value chains: avocado, cacao, pineapple, and greenhouse and oriental vegetables. The program will improve product quality, increase production and efficiency, increase the value of post-harvest products, and improve market and marketing linkages. Purpose of Midterm Performance Evaluation Per USDA’s Food Aid Division (FAD) M&E Policy, the purpose is to critically and objectively review and take stock of EQ’s program implementation, assess relevance of interventions, provide early signal of effectiveness of interventions (or lack of), document lessons learned, assess sustainability efforts to date, and discuss/recommend mid-course corrections as necessary. Consultant Tasks: Objectives and Evaluation Questions Specifically, the evaluator/evaluation team will assess the following: 1. Program design, implementation, and efficiency a. Is the program on track to reach specified targets at midpoint and by end of the program? • Specifically, what percent is on track and what percent is not on track? • For targets not on track to be met, specify changes (if any) to the initial assumptions or within the operating context that warrant a review and recommended revision of the targets. b. What have been the main reasons behind current achievement against result and activity indicator targets? • Specifically, what should be maintained and/or scaled up? c. What have been the major challenges to program implementation and how has EQ responded to these challenges? • Specifically, to what extent has the Med-fly ban affected program implementation? • To what extent did unexpected heavy rains in the Fall of 2016 affect oriental vegetable exports? Annex 1: Scope of Work for Midterm Performance Evaluation of EQ Program 2 d. What, if any, unintended but important direct results (positive or negative) have occurred? • How should these be assessed in the overall program context? • How has EQ responded to these? e. To what extent is staffing, management, and oversight costs suitable given the number/scope of activities carried out? 2. Relevance of interventions a. To what extent do EQ activities address the core issues of target beneficiaries? 3. Effectiveness of interventions a. Which interventions/activities have the highest potential in being the most effective approach to achieve the program’s higher-level results (increased agricultural productivity and more market transactions) b. How effective is EQ at reaching women beneficiaries? What could be done to improve women’s participation? c. How effective has the training approach been to build and/or enhance capacity of target beneficiaries? d. To what extent do public authorities and private sector partners have an influence (positive or negative) on the achievement of program objectives? 4. Sustainability efforts to date a. What factors contribute towards sustainability of program results and how has EQ focused on these? b. Is there a well-developed exit strategy? Methodology Guideline The evaluator/evaluation team will use a mixed methods approach, including but not limited to quantitative surveys, focus group discussions with target beneficiaries, and key informant interviews with government officials and relevant public/private stakeholders including EQ staff and USDA. The data collection tools for collection of key data should be similar to the tools used during the baseline study or during routine monitoring (to the extent possible) so that results may be comparable. The quantitative sample size should ensure a 95% confidence level with a 5% margin of error. The total direct beneficiaries as of March2017 is 698, which could be used for the sampling frame. Annex 1: Scope of Work for Midterm Performance Evaluation of EQ Program 3 The evaluator/evaluation team will meet with IESC and USDA to discuss proposed methodology prior to initiating evaluation activities. Key Activities The evaluator/evaluation team will undertake the following key activities: 1. Desk review a. Review program-related documents including: • EQ agreement and any modifications • Approved M&E plan • Baseline study report • Semi-annual reports 1, 2, 3, and 4 • Approved work plan • USDA M&E policy • Value chain assessment reports • Beneficiary mapping report • Any other relevant program documents provided by IESC b. Review other relevant documents to understand the operating environment in which EQ operates: • Biweekly communiques • Program Facebook Page 2. Develop and finalize evaluation methodology a. In collaboration with IESC and USDA, • Confirm sampling frame and finalize sampling technique and sample size; • Develop and propose survey design and data collection tools for quantitative survey, follow-up focus group discussions, and key informant interviews. Refine or create any tools necessary to answer the evaluation questions; and • Based on the desk review, propose any additional topics or issues for analysis in the final evaluation at the end of the program. 3. Field data collection • Plan and coordinate all necessary logistics to conduct evaluation. In terms of office space, the evaluator/evaluation team will co-locate with the EQ program in Santo Domingo. The evaluator/evaluation team will also receive assistance from EQ in locating relevant beneficiaries. Other than this, all resources (HR and material) should be arranged by the evaluator/evaluation team. • Translate, pre-test, and finalize data collection tools including survey instruments and focus group questionnaire. • Train enumerators, as needed. Annex 1: Scope of Work for Midterm Performance Evaluation of EQ Program 4 • Carry out fieldwork with liaison support from EQ staff, particularly M&E manager and analyst. 4. Data analysis and reporting • Scrub, synthesize, analyze, and interpret data from survey, focus groups, and interviews. • Prepare and submit datasets with relevant documents to EQ and USDA. • Prepare a draft evaluation report addressing the objectives and questions of this midterm evaluation as well as propose course correction or scale-up recommendations for review by EQ. • Present evaluation findings in-person to EQ, and relevant stakeholders • Submit final report. Timeframe and Proposed Level of Effort This contract should take about eight weeks starting on or about October 16, 2017. An illustrative timeline is presented below. Activity Level of Effort Due Date Comments Desk review 2 days NA Does not have to be consecutive days Kick-off meeting with USDA and IESC staff to discuss expectations, protocol, methodology, and timeline 1 day TBD IESC/EQ staff to coordinate. Develop draft evaluation work plan (including methodology and draft data collection tools) 5 days TBD Present draft evaluation work plan to USDA and IESC. Refine based on comments received. 2 days TBD IESC/EQ staff to coordinate presentation date and location. Annex 1: Scope of Work for Midterm Performance Evaluation of EQ Program 5 Field work/data collection 15 days NA Includes training of enumerators, if needed. Data entry and scrubbing 3 days NA Data analysis and report writing 8 days NA Submit draft report to IESC for comments and clarifications NA Within 7 weeks of contract signing EQ will review and provide comments within 5 days of receiving draft report Prepare oral presentation of findings to USDA, IESC/EQ, and relevant stakeholders and conduct presentation. 2 days TBD IESC/EQ staff to coordinate presentation date, location, and required attendees. Finalize report and submit to IESC. 3 days Within 8 weeks of contract signing Submit to IESC and USDA Deliverables 1. Evaluation work plan that describes the following: • Understanding of the program based on desk review and kick-off meeting • Final evaluation methodology, including detailed sampling plan, field work plan, and any limitations of the proposed approach • Description of planned quality control measures • Description of communication protocol • Final timeline 2. Electronic copies of all clean and final versions of data collection tools, both in English and Spanish 3. Clean and final versions (both English and Spanish) of quantitative datasets and qualitative transcripts in agreed upon format 4. Electronic draft midterm evaluation report in English, addressing all evaluation objectives and questions 5. Oral presentation materials of evaluation findings in agreed upon format 6. 15 – 20 high quality pictures of the process Annex 1: Scope of Work for Midterm Performance Evaluation of EQ Program 6 7. Electronic English version of the final evaluation report in PDF and Word, as well as two printed copies in color (one for USDA/DR and one for the EQ office). The final report should include, but not limited to: • List of acronyms/abbreviations • Table of contents • Executive summary • Background • Detailed evaluation methodology • Findings • Recommendations for the remainder of the program • Annexed scope of work • Annexed data collection instruments Annex 2: The Program Database and the Evaluation Sample 1 The midterm evaluation scope of work requires that the quantitative sample size should ensure a 95 percent confidence level with a 5 percent margin of error. As of Sept. 30, 2017, the total number of direct beneficiaries tracked by the program monitoring and evaluation team was 2,650, which was used as a sampling frame. The sample size of the evaluation survey was 335 direct beneficiaries. The program provided the evaluation team with a database containing the names of beneficiaries and events in which they participated. The program database of direct beneficiaries had data disaggregated by value chain (VC) and gender but not a breakdown (in a specific Excel column) by profession (farmer/producer/packinghouse staff/exporter/extension officer, etc.). 1 Therefore, the stratification focused on (i) gender and (ii) VC relevance: a total of seven VC groups were identified, including the five target VCs of avocados, cocoa, pineapples, greenhouse vegetables, and oriental vegetables, plus one group including activities related to two or more VCs and an additional group including an unspecified VC. Almost three quarters of the beneficiaries are men. Women show slightly higher representation in cocoa and greenhouse vegetable VCs and are underrepresented in the avocado and, particularly, pineapple VC. The value chain is unspecified for as much as 43 percent of the beneficiaries, while an additional seven percent are active in two or more VCs. An initial screening of the beneficiaries of unspecified VCs showed that they included all professions targeted by the program, i.e., producers, packinghouse staff, processors, traders, and exporters, but mostly extension officers (from the Ministry of Agriculture and other public institutions) and private consultants. This has been confirmed by the survey. Table A1: Sampling Frame by Value Chain and Gender VCs Total Male Female Avocados 372 14% 323 87% 49 13% Pineapples 140 5% 129 92% 11 8% Greenhouse vegetables 167 6% 116 69% 51 31% Oriental vegetables 267 10% 209 78% 58 22% Cocoa 361 14% 256 71% 105 29% 2 or more target commodities 195 7% 139 71% 56 29% Unspecified VCs 1115 43% 754 68% 361 32% Total 2,617 100% 1,926 74% 691 26% 1 Disaggregation by profession was available for a number of indicators but not for the entire database of beneficiaries used for performing this evaluation. Annex 2: The Program Database and the Evaluation Sample 2 The evaluation team met with the program and IESC home-office staff to discuss and validate evaluation methodology prior to initiating the evaluation activities. A total of 2,617 beneficiaries were used as the sampling frame, therefore the quantitative sample size—to ensure a 95 percent confidence level with a 5 percent margin of error and a 50 percent distribution level, which is the most conservative—was set at about 329–335 beneficiaries,2 from whom suitable feedback was gathered in the form of questionnaire surveys and focus groups discussions. The survey sample was selected through preliminary stratification using the above proportions for each of the seven VC groups and the gender distribution in each of them. The sample breakdown by gender and VC is in Table A2 below. In most surveys it is almost impossible for all the selected individuals to be available to fill in the questionnaires, therefore the survey sample was increased by 50 percent to ensure that the target of 335 completed survey questionnaires was attained. In practical terms, since (i) the provided database does not contain contact information for some of the randomly selected beneficiaries, and (ii) the random selection in some cases selected the same beneficiary twice; in both these cases, the next beneficiary in the database list with contact information was selected to replace the missing ones (unless contact information could be provided for the randomly selected respondents). Table A2: Random Sample by VC and Gender VCs Number of individuals Male Female Avocados 48 41 6 Pineapples 18 17 1 Greenhouse vegetables 21 15 7 Oriental vegetables 34 27 7 Cocoa 46 33 13 2+ Fruits and vegetables 25 18 7 Unspecified 143 97 46 Total 335 247 88 2 Since individuals participating in USDA-sponsored trainings in multiple fiscal years may be counted once in each fiscal year, the sample size ensures a 95 percent confidence level with a 5 percent margin of error ranging between 229 (i.e., all fiscal year #1 beneficiaries also participated in fiscal year #2 activities) and 335 (i.e., no fiscal year #1 beneficiaries participated in fiscal year #2 activities) beneficiaries. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 1 Result Indicator #1 (Food for Progress (FFPr)17): Total number of individuals benefitting directly as a result of USDA assistance This performance indicator measures the number of individuals directly participating in the USDA funded interventions. The individuals must be engaged with a project activity or come into direct contact with a set of interventions (goods or services) provided by the program. This may include, for example, farmers, processors, and others receiving training, inputs, or financial services. 1 The cumulative target at the end of Y2 is 3,440 beneficiaries and that at the end of year four is 8,000 beneficiaries. They are disaggregated as follows: Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 600 3,440 2,650 23% below target 6,248 8,000 Male 0 540 3,096 1,946 37% below target 5,623 7,200 Female 0 60 344 704 105% above target 625 800 The performance indicator tracks access to services that can lead to adoption of improved agricultural techniques, technologies, practices, services, and policies that lead to greater agricultural productivity and expanded agricultural markets. The EQ program had trained 2,650 beneficiaries by the end of year two, thus remaining below the target by 23 percent. Almost three-quarters of the beneficiaries are men, with women relatively more represented in cocoa and greenhouse value chains (VCs) and underrepresented in the avocado VC. This tracks approximately the actual gender representation in the target value chains2 . Nonetheless, the EQ program seems likely to achieve the end of year four cumulative target if it can sustain the performance pace achieved between April and September 2017 (see chart below). 1 Individuals merely contacted or involved in an activity through brief attendance (nonrecurring participation) do not count. Individuals participating in USDA-sponsored trainings in multiple fiscal years may be counted once in each fiscal year. 2 Additional information on gender representation in each VCs can be found in Annex 2. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 2 The informal audit of the database suggests that a certain number of beneficiaries were counted twice while a relatively larger number of them should not be counted as direct beneficiaries because they were involved in an activity through brief attendance (e.g., high￾school students attending workshops); in addition, a similar number of would-be direct beneficiaries do not consider themselves as such3 . On the other hand, the evaluation team encountered a relatively significant number of beneficiaries who did attend program activities but whose names were not found in the database. All in all, the evaluation team estimated the number of individuals who should be excluded from the number of direct beneficiaries counted to be between 1 percent- 2 percent, thus within an allowable statistical margin of error. Result Indicator #2 (FFPr18): Total number of individuals benefitting indirectly as a result of USDA assistance This performance indicator measures the number of individuals indirectly benefitting from USDA-funded interventions. These individuals will not be directly engaged with a project 3 The total share of mistaken double entries, nonrecurring participation, and nonbeneficiaries has been estimated by the evaluating team at about 4–5 percent of the Y2 cumulative beneficiaries. Figure 1: Result Indicator #1: Number of Direct Beneficiaries (Cumulative Actual) Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 3 activity or come into direct contact with a set of interventions (goods or services) provided by the project. This may include, for example, family members of farmers who were trained. If an individual is already counted as a direct beneficiary, the individual should not also be counted as an indirect beneficiary. The indicator is calculated using a multiplier of five. The result is, therefore, an estimate. The multiplier considers grown children who help with the farm/business (not toddlers or babies). It also includes other farmers (neighbors or friends) of the direct farmer beneficiary as s/he may have passed on or demonstrated new knowledge, skill, equipment. Being directly correlated with Result Indicator #1, the performance of the EQ program is the same as above, hence by program midpoint its performance is 23 percent below target. Result Indicator #3 (FFPr16): Number of individuals who have received short-term agricultural sector productivity or food security training as a result of USDA assistance This performance indicator aims at measuring increased agricultural productivity due to USDA intervention. Its unit of measurement is the number of individuals to whom significant knowledge or skills have been imparted through formal or informal interactions that are intentional, structured, and purposed. The relevant data is collected using attendance sign-in sheets and technical assistance trackers. The indicator is a roll-up of all activity training indicators. Training can enhance human capacity and hence may result in transformational capacity development. The cumulative target at the end of year two is 1,783 trainees and that at the end of year four is 6,333. They are disaggregated as follows: Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) Total 0 365 1,783 2,336 31% above target 4,120 6,333 Male 0 329 1,604 1718 7% above target 3,708 5,699 Female 0 37 178 618 247% above target 412 633 Producer 0 - 515 1,060 106% above target 1,820 3,010 People in firms 0 290 1,182 828 30% below target 2,149 3,106 People in government 20 60 292 387% above target 100 140 The EQ program has trained 2,336 beneficiaries by the end of year two, thus exceeding the target by 31 percent. Trained men total 1,718 and represent 74 percent of trainees, women number 618 (26 percent). Producers are the largest group of these trainees (1,060, or 45 percent), then staff of firms (828, or 35 percent) and people in government (292, or 13 Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 4 percent). Both producers and, especially, people in government (i.e., extension officers) are overrepresented when compared with the target. Given the achievement so far, it is recommended to scale up the Life of Project target by at least one-quarter, i.e., the new end of year 4 cumulative target would be set at 8,500 individuals who have received short-term agricultural sector productivity or food security training. Result Indicator #4 (FFPr1): Number of hectares under improved techniques or technologies as a result of USDA assistance This performance indicator aims at measuring the improved quality of land and water resources (FFPr1.1). The indicator measures the area in hectares of land first brought under improved techniques or technologies during the reporting year. Technologies to be counted must be agricultural-related and/or land-based innovations that include climate change adaptation and mitigation. EQ program staff is to calculate data using program activity information reported in aggregate reporting forms and confirmed using direct observation and interviews during site visits as needed. They should count hectares affected by GLOBAL G.A.P., traceability, and climate smart-related activities, as well as hectares affected by traps (greenhouses) and IPM trainings. The cumulative target at the end of year two is 2,150 hectares and that at end of year four is 5,000 hectares. Indicators are disaggregated as follows: Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) Total 0 375 2,150 596.69 72% below target 3,905 5,000 Pest management 0 188 1,075 18.30 98% below target 1,953 2,500 Climate mitigation or adaptation 0 75 430 - 100% below target 781 1,000 Other 0 113 645 578.40 10% below target 1,172 1,500 Data is to be used to monitor the extent to which improved techniques or technologies and climate smart activities are used by direct program beneficiaries. Land under improved techniques or technologies because of USDA assistance amounted to 596.69 hectares at the midpoint of EQ program implementation, or just 28 percent of target. Integrated pest management (IPM) has been introduced in a small percentage (3 percent) of acreage, while no climate smart techniques have been introduced so far. Most of the beneficiaries have introduced improved techniques to increase crop productivity (fertilization, pruning, irrigation, etc.). Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 5 Given the progress so far, it is expected that the end-of-program target will not be achieved. It is therefore recommended to scale down the target by at least one-quarter, i.e., the new end of year 4 cumulative target would be set at 3,750 hectares under improved techniques or technologies because of USDA assistance. Result Indicator #5 (FFPr2): Number of individuals who have applied new techniques or technologies as a result of USDA assistance This performance indicator aims at measuring the increased use of improved agricultural techniques and technologies (FFPr1.2). The indicator measures the total number of agricultural producers who applied new techniques or technologies anywhere within the targeted VCs because of USDA assistance. This includes innovations in efficiency, value￾addition, on-farm post-harvest management, sustainable land management, and water management. Program staff is to deduce an estimate from a sample of direct program beneficiaries who submit engagement reporting forms, which will have relevant questions that collect data for this indicator. The information must be confirmed through direct observation and interviews during site visits for a random sample. The cumulative target at the end of year two is 2,580 beneficiaries applying improved techniques and/or technologies, while that at end of year four is 6,000 such beneficiaries. They are disaggregated as follows: Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 450 2,580 503 81% below target 4,686 6,000 Male 0 405 2,322 384 83% below target 4,217 5,400 Female 0 45 258 119 54% below target 469 600 Pest management 0 225 1,290 48 96% below target 2,343 3,000 Climate mitigation or adaptation 0 90 516 - 100% below target 937 1,200 Other 0 135 774 396 49% below target 1,406 1,800 Technological change and its adoption (i.e., behavior change) by different actors in the agricultural supply chain are critical to increasing agricultural productivity. Producers who have introduced improved techniques or technologies because of USDA assistance totaled 503 individuals at the midpoint of EQ program implementation, or just 19 percent of target. Three-quarters of them are men; women’s share is higher than expected, but their number is too small to achieve target. Some 68 farmers have introduced (integrated) pest management, while most of the beneficiaries have applied improved techniques to increase crop productivity (fertilization, pruning, irrigation, etc.). Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 6 Given the progress so far, it is expected that the end-of-program target will not be achieved. It is therefore recommended to scale down the target by at least one-quarter, i.e., the new end of year 4 cumulative target would be set at 4,500 producers who have introduced improved techniques or technologies because of USDA assistance. Result Indicator #6 (FFPr3): Number of individuals who have applied improved farm management practices (i.e., governance, administration, or financial management) as a result of USDA assistance This performance indicator aims at measuring improved farm management (FFPr1.3). This indicator measures the total number of beneficiaries who are applying the knowledge or skills received in USDA supported training in farm management practices. They take account of on￾farm, agricultural-related management practices including calculating input, output, and labor needs; incorporating business practices related to the management of land, equipment, facilities, transportation, and shipping; the use of IT; financial planning, cash flow, balance sheets; record-keeping; maintaining and using inventories, etc. The cumulative target at the end of year two is 425 beneficiaries applying improved farm management practices, while that at the end of year four is 1,250 such beneficiaries. They are disaggregated as follows: Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 - 425 - 100% below target 850 1,250 Male 0 - 383 - 100% below target 765 1,125 Female 0 - 43 - 100% below target 85 125 Ag producer 0 - 425 - 100% below target 850 1,250 Record-keeping is essential for all international certifications and Food Safety Modernization Act (FSMA) as well. No producer has yet introduced such improved farm management practices. The program has published a scope of work for a consultant to train farmers in the use of record-keeping. It is recommended that the program focus on developing a strategy for getting beneficiaries involved in the application of improved farm management practices. It is believed that behavior change in farm management practices by different actors in agriculture may result in increased agricultural productivity and increased income for farmers. Given the progress so far, it is expected that the end-of-program target will not be achieved. It is therefore recommended to scale down the target by at least one-third, i.e., the new end of year 4 cumulative target would be set at 830 producers who have introduced improved Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 7 farm management practices such as governance, administration, or financial management because of USDA assistance. Result indicator #7 (FFPr13): Value of sales by project beneficiaries Result Indicator #7 (FFPr13) collects the value (in US$) of sales of the five targeted VCs by all program beneficiaries, not just farm-gate sales. The actual number reported for the indicator must be the value of sales of produce by direct project beneficiaries in each reporting period. Only sales in the reporting period attributable to USDA investment are counted. Methods of data collection include forms completed and signed by program beneficiaries (producer groups and processors), as well as document review of sales reports, if available. The EQ program aims at increasing the cumulative value of sales by beneficiaries by 25 percent after two years, by 45 percent after three years, and by 65 percent at the end of the four years of program implementation. Therefore, the EQ program target sales by the beneficiaries are US$ 11.2 million by the end of year two from a baseline of zero and US$ 29.1 million by the end of year four. These sales should only include those attributable to USDA support. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 0 11,186,750 11,718,778 5% above target 20,136,150 29,085,550 Avocados 0 0 2,910,500 894,711 69% below target 5,238,900 7,567,300 Pineapples 0 0 1,619,250 2,927,446 81% above target 2,914,650 4,210,050 Asian vegetables 0 0 2,563,750 1,653,194 36% below target 4,614,750 6,665,750 Greenhouse vegetables 0 0 1,783,500 1,236,721 31% below target 3,210,300 4,637,100 Cocoa 0 0 2,309,500 5,006,703 117% above target 4,157,100 6,004,700 According to information collected by the M&E staff of the EQ program, sales of direct beneficiaries totaled US$66.5 million at program midpoint, of which US$ 11,7 million attributable to USDA support, thus exceeding target by five percent. As for value chain breakdown, the sales of pineapples and cocoa were well above the end on year two target, while those of Asian and greenhouse vegetables and avocados were all well below that target. It should be noted that: a) Only a fraction (218) of total direct beneficiaries was able to provide information on actual sales, though they are among the biggest producers and exporters4 ; 4 A total of 218 beneficiaries provided information on their sales of target commodities. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 8 b) The program has calculated the sales attributable to USDA support by interviewing a representative sample of 68 beneficiaries and applying the simple average of their estimation to the framing sample. It is recommended that in future the program applies the weighted average instead. c) the estimate refers to the sales of the last available period (i.e., the 2016 calendar year) rather than the October 2016—September 2017 period of reference of the indicator5 ; d) the analysis of available international trade statistics suggests that the export value of target commodities has not significantly grown between 2014 (baseline) and 2016 (last available year on Trade Map and it exclude the fourth quarter of that year) except for cocoa, pineapples, tomatoes, and certain Asian vegetables (e.g., okra). The medfly ban has in fact had a significant impact on exports of avocados and peppers during 2015 (see the Tables in Annex 4). Result indicator #8 (FFPr14): Volume of commodities (metric tons) sold by project beneficiaries Result Indicator #8 (FFPr14) collects the volume (in gross metric tons or MT) of sales of the five targeted VCs by all project beneficiaries, not just volume of farm-gate sales. The actual number reported for the indicator must be the gross volume of sales of a produce by direct project beneficiaries in each reporting period. Only the gross volume of sales in the reporting period attributable to USDA investment is counted. Methods of data collection include forms completed and signed by program beneficiaries (producer groups and processors) as well as document review of sales reports, if available. The EQ program aims at increasing the cumulative volume of sales by project beneficiaries by 25 percent after two years, by 45 percent after three years, and by 65 percent at the end of the four years of program implementation. Therefore, the EQ program target volume of sales by the beneficiaries is MT 55,934 by the end of year two from a baseline of MT 42,505, and MT 70,208 by the end of year four. These sales should only include those attributable to USDA support. Volume in MT of sales at the farm and enterprise level of targeted produce is a measure of competitiveness of those receiving USDA assistance. Moreover, it helps track supply, access to markets, and progress toward commercialization. 5 This is justifiable because RD beneficiaries are used to report sales on calendar year (which also corresponds to the fiscal year, contrary to USA and program reference periods) and they will be able to report 2017 sales not before early 2018. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 9 Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 0 10,626 20,491 93% above target 19,127 27,628 Avocados 0 0 2,998 4,590 53% above target 5,396 7,795 Pineapples 0 0 2,519 4,841 92% above target 4,534 6,549 Asian vegetables 0 0 2,070 1,911 8% below target 3,726 5,381 Greenhouse vegetables 0 0 2,289 1,288 44% below target 4,119 5,950 Cocoa 0 0 752 7,862 946% above target 1,353 1,954 According to information collected by the M&E staff of the EQ program, sales of direct beneficiaries totaled almost MT 286,000 at program midpoint, of which MT 10,626 million attributable to USDA support thus exceeding target by 93 percent. As for value chain breakdown, sales of avocados, pineapples and cocoa were well above the end on year 2 target, while those of Asian and greenhouse vegetables were below that target. When evaluating this performance, considerations like those above in result indicator #7 apply. In addition, at least one third of volume sales of cocoa products in the sample appears to have been calculated wrongly6 . This should not impact significantly the positive performance of this Result Indicator; nonetheless it is recommended that the Program M&E checks the relevant data. Result Indicator #9: Percentage of tests of target post-production agricultural products that come back clean This custom performance indicator aims to measure the improved quality of post-production agricultural products (part of Food for Progress 2.2.1). “Clean” means the products that undergo a test pass post-production processing and handling requirements. The percentage is calculated by dividing the tests that come back favorable (numerator) by the total number of tests (denominator). The indicator counts only lab tests from packing-house partners. The cumulative target at the end of year two is 50 percent and that at the end of year four is 80 percent. The calculation should be made from a review and tally of available lab reports. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 Zero 50% 100% 100% above target 70% 80% 6 An informal audit of the database shows that the average price of cocoa products ranges between US$ 2 per MT and US$ 1,000 per MT, thus indicating a clear overestimation of the volume sold as confirmed by the performance of the indicator. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 10 Post-production agricultural products that “pass,” or come back with no significant findings, indicate improved quality. The EQ program has achieved 100 percent of clean lab tests, thus exceeding the target foreseen for the end of year two. Produce samples were tested for the presence of salmonella in chili peppers produced in La Vega, Santiago, and Moca by farmers and packing houses. It should be noted, however, that tests should also be performed on other kinds of infestations, such as thrips (an insect whose presence has been detected in both Asian and greenhouse vegetables) or any sort of fungi, bacteria, or virus. The aim of the indicator is to show progressive improvement in the quality of post-production produce measured by the reduction of the infestation level. Testing vegetables for thrips in Y1 would have provided a baseline value of infestation to be compared with that at the end of Years 2, 3, and 4 to evaluate the progress of the program assistance. The performance of this indicator is difficult to assess even with (or because of) a result of 100 percent clean tests as an initial indicator. The salmonella infection was due to bad agricultural practices, such as using dirty, salmonella-infected sacks to collect chili peppers and the simple application of good agricultural practices (i.e., the use of clean sacks) has solved the problem (an achievement of the program, nonetheless). Result indicator #10: Number of direct beneficiaries achieving internationally recognized certifications This custom performance indicator aims to measure the improved quality of post-production agricultural products (FFPr2.1.1) through increased adoption of established standards by industry (Food for Progress 2.1.1.1). The indicator counts the total number of targeted processors and packing houses that become certified in GLOBALG.A.P., organic, fair trade, International Standardization Organization (ISO), and Hazard Analysis and Critical Control Points (HACCP). Program beneficiaries must complete, sign, and send a reporting form to program staff. Program staff will use traceability reports and conduct random site visits to verify data. The cumulative target at the end of year two is to get three internationally recognized certifications, and that at the end of year four is to obtain 13 such certifications. An increase in the number of processors being certified means there is an increase in adoption of established standards. No such certification was obtained by the midpoint of program implementation. However, two groups of producers of, respectively, avocados and pineapples linked to two packing houses have been receiving program assistance to get GLOBALG.A.P., an internationally recognized certification increasingly needed to export to the European Union and, partly, to the United States. The program issued an annual program statement for grant applications to support the technical assistance required by farmers, packing houses, and cocoa processors to prepare themselves for a certification of their choice. The program received five applications, of which two grants have been executed so far. In addition, the program has supported the activity of the working group on GLOBAL G.A.P. certification for the approval of a national interpretation guide for the application of the GLOBAL G.A.P. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 11 standards adapted to the DR’s characteristics; once approved, the guide is expected to facilitate obtaining GLOBAL G.A.P. certification for DR beneficiaries. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 Zero 3 0 100% below target 8 13 The USDA has asked the program to expand the FSMA Preparedness Assessment work they are doing under USDA CAFTA for five Dominican packing houses to include an additional five to ten. Even though there is not an official FSMA certification, the FDA is preparing to perform audits of compliance with preventive controls and produce safety regulations. Therefore, any FSMA Preparedness Assessments the program facilitates may count toward this indicator. It is recommended that the USDA authorize the counting of the obtaining of such FSMA Preparedness Assessments as a certification for this performance indicator. Result Indicator #11: Average percent reduction in time a product takes to get through problem points in a value chain This custom performance indicator measures the average percent reduction in time a product takes to get through problem points in the VC. The performance indicator tracks the change in time (expressed in percent of time reduced) of target commodities getting through pre￾identified specific problem points in the VC that the program will help to address. Such a reduction will measure the increased efficiency of post-production processes (FFPr2.1.2). The end of Y2 target was set at 10% in term of time reduction and that at the end of Y4 at 30%. There has been no achievement so far, though certain activities are in progress and are expected to bear fruit in the next reporting periods, for instance for international recognized certification. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 Zero 10% 0% 100% below target 20% 30% In the baseline report 12 problem points were identified and a baseline value established. They were intended to be assessed during the first part of program implementation to verify actual interest on the part of beneficiaries to invest in reducing/eliminating such problem points. At program midpoint, it is recommended to focus on those problem points that can realistically be dealt with by program activities, while eliminating the others. The newly selected problem points have been refocused or modified to take into consideration the actual needs and requests of the program’s direct beneficiaries, as described below. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 12 AVOCADO AND PINEAPPLE VALUE CHAIN PROBLEM POINTS Problem point 1 GLOBAL G.A.P. certification takes time to be awarded, while it could be done more quickly with adequate and ad hoc technical assistance. There is considerable potential for expanding avocado and pineapple premium exports to the European Union. In the few years, exports of both avocados and pineapples to the European Union have increased significantly (almost double for avocados, though from a low base), but to ensure a steady and sustainable flow of produce to non-U.S. markets, GLOBAL G.A.P. certification is needed. A few producers have already obtained or are close to obtaining such certification, while others are initiating the process. This takes time and has a cost (on average US$5,000 per 4-hectare farm to get a certification and an annual fee of about US$2,000) and technical knowledge is a scarce resource to meet the demand of possibly interested producers. Objective: The program may facilitate the process of obtaining GLOBAL G.A.P. certification for direct beneficiaries, reducing the time of certification attainment from 12 months (or more) to nine months (or less) for groups of 20–25 producers per organization (e.g., packing house, processor). The facilitation may take the form of contributing to the payment of consulting fees and/or contributions to the building/refurbishing of farm facilities (bathrooms, depots for inputs/outputs, etc.) needed to get/maintain certification. Problem point 2: Processing capacity is very low for both avocados and pineapples. The supply of avocados far exceeds demand and sometimes producers do not harvest their produce because the sale price does not cover harvesting costs. 7 Domestic demand for pineapples is currently stronger than supply, but this may change soon because much sugar cane and pastureland is being converted to pineapple plantations and production is expected to expand by 20–30 percent per annum. Processing capacity is therefore critical to assure VC competitiveness for produce when supply exceeds demand. Currently there is no processing of avocados (either into guacamole or oils for cosmetic use), and very little processing of pineapples (pulp, juice). The program may facilitate the investment in processing machines by providing in-kind grants for the purchase of processing equipment for pineapples (for juice, pulp, or dried slices) and avocados (for guacamole or oils for consumption or cosmetic use). The program may facilitate the process of purchasing processing equipment providing in-kind grants for up-to one quarter of capital expenditures and by so doing shortening the time for i) getting the equipment and ii) processing the produce. Objective: To reduce problem point time by at least 30 percent, in terms of shortening the period needed for purchasing the equipment from 15 months (or more) to 10 months (or less); or by reducing the processing time for obtaining high-quality manufactured produce by at least 7 The avocado supply is expected to increase significantly in the next few years thanks to a recently enacted forestation policy targeting the areas bordering Haiti and providing subsidies to farmers planting trees, including avocado trees. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 13 30 percent, from several days (if done manually) to just a few hours (if done with high￾performing equipment). GREENHOUSE AND ASIAN VEGETABLES VALUE CHAIN PROBLEM POINTS Problem point 3: Too many pest treatments are reducing the quality and sanitation of the produce as well as increasing production costs, therefore lowering the competitiveness of the produce. Improvement in the safety of exported produce can be facilitated by introducing good agricultural practices in pest management and reducing the level of pesticide residues in the exported produce. Currently, for instance, there is an intensive use of fumigation, especially for controlling thrips infestations that must be reduced to obtain high￾quality produce and lower production costs. Greenhouse and Asian vegetables production could greatly benefit by the application of IPM and the introduction of microbial agents that can control or suppress the growth and multiplication of insect pests, thus reducing the need for fumigations or pesticide use. The program could introduce a pilot project to reduce the use of fumigation and other pesticides by applying IPM in a few selected neighboring medium-sized farms to improve the quality and sanitation of the produce while lowering production costs. Objective: To reduce problem point time by at least 30 percent, in terms of reducing the time to introduce improved pest management from 24 months (or more) to 16 months (or less) by adopting biological-based IPM techniques to reduce pest levels at the farm level with reduced pesticide use. Suggested Alternative Measure: To reduce the cost of pest treatments (or the number of necessary treatments) by at least 30 percent. Problem point 4: Hot water treatments is required to cut the level of pest infestation in vegetables exported to Europe, but this treatment has reduced the shelf life of the produce. Hot water treatments were originally used to control fungi, but later the application was extended for insect disinfestations as an alternative to the more expensive vapor heat treatments. The cost of hot water immersion technology is about 10 percent that of a commercial vapor heat treatment. Hot water treatments can be used for insect disinfestations of tropical horticultural produce to control fruit flies. For insect control, temperatures range from 43–49°C but the application time is longer than for fungi because the entire vegetable, not only the surface, must be brought to a higher temperature. Treatment times range from 30 to 120 minutes and can be reduced if only insects found in the outer layers are to be killed. However, heat treatment may cause a shorter shelf life for produce if the treatment is not done with the right modalities of temperature and timing of immersion. Therefore, the program may provide technical assistance to refine the modalities of the use of hot water treatments that is required for exports bound to Europe, reducing the impact on vegetable shelf-life. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 14 Objective: To reduce problem point time by at least 30 percent, in terms of a shorter period to obtain packaged vegetables with low levels of pest infestation suitable for export to Europe. Suggested Alternative Measure: To prolong the shelf life of vegetables treated with hot water by at least 30 percent, or reduce the post-harvest losses of vegetables treated with hot water by 30 percent. COCOA BEAN VALUE CHAIN PROBLEM POINTS Problem point 5: The need to expand the process of wet fermentation to add value to small producers’ activities. The DR cocoa industry pioneered the wet fermentation that adds quality and value to exported cocoa products. This fermentation process is already performed in most of the cooperatives and associations that are member of the National Confederation of Dominican Cocoa Producers (CONACADO), as well as by most exporters, but there are a few smaller associations that deal directly with foreign buyers, that would prefer to expand the on-farm fermentation process rather than use facilities of third-party enterprises or sell the unfermented product to exporters. The program might help by providing in-kind grants for facilitating the supplying of this fermentation equipment. Objective: To reduce problem point time by at least 30 percent, in terms of reducing the period of time needed by cocoa producers to build structures and purchase fermentation equipment by providing in-kind grants, from 24 months (or more) to 15 months (or less). Suggested Alternative Measure: increase by at least 30 percent the value of produce added by the investment. Problem point 6: Improve and modernize the manufacturing of cocoa product, especially those targeting the high-value tourist sector. The sorting and packaging of processed cocoa products is often done by hand or with old equipment. There is a need to invest in new, better-performing processing and packaging equipment, able to meet good manufacturing practice requirements, but small producers’ associations/processors lack capital to invest in such equipment. The program can provide in￾kind grants to cover up to one-quarter of capital expenditures for modernizing processing and packaging tools, such as presses, refiners, evaporators, roasters, filling and weighing machines, etc. Objective: To reduce problem point time by at least 30 percent, reducing the time needed by cocoa processors to purchase processing/packaging equipment by providing in-kind grants, Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 15 from 24 months (or more) to 15 months (or less); or by reducing the manufacturing time for obtaining packaged produce. Result Indicator #12: Number of partner packing houses using a traceability system as a result of USDA assistance This custom performance indicator aims to measure the increased efficiency in post￾production processes (FFPr2.1.2) and the increased use of improved post-production processing and handling practices (FFPr2.1.2.1). The indicator counts the number of target packing houses with previously nonfunctioning or no existing traceability systems. These partners are requested to cost-share the traceability system. Program staff will track, use traceability reports, and conduct random site visits to verify data. The cumulative target at the end of year two is eight packing houses, and that at end of year four is 30 packing houses. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 Zero 8 0 100% below target 15 30 The use of market traceability data is correlated with the use of improved post-production processing and handling practices. No partner packing house has installed a traceability system as of the end of year two. However, the program has supported specific activities to update and improve the existing traceability software to track produce from farm to fork. The updated software will be able to manage packing-house operations, including the planning of supply, stock management, sale management, and post-sale assistance. The basic module could be upgraded on demand with additional components, including IPM, HACCP, GLOBAL G.A.P., and similar specific packages. This traceability system will be able to meet new FSMA export traceability requirements (that shift responsibility for applying safety measure and controls from U.S. importers to DR exporters), that will be compulsory for food exports to the United States starting in May 2018. Six packing houses are expected to install this traceability software by March 2018. Result Indicator #13: Quantity of post-harvest equipment and facilities improved after USDA assistance The indicator counts the quantity of post-harvest equipment upgraded and facilities improved after USDA assistance (this may include grants, technical assistance, and training). As with the one above, this custom performance indicator aims at measuring increased efficiency in post-production processes (FFPr2.1.2) and the improved post-harvest infrastructure (FFPr2.1.2.2). Project staff must track deals as grants and technical assistance occur. Follow￾up after trainings on SPS, FSMA, and others must be conducted on a sample of beneficiaries through a survey. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 16 The cumulative target at the end of year two is having 40 units of post-harvest equipment upgraded and packing-house facilities improved, and that at the end of year four is to have 137 units of post-harvest equipment upgraded and packing houses facilities improved. This is a direct measure of the activity involving modernizing post-harvest infrastructure in the VCs. At the midpoint of program implementation, only four units of post-harvest equipment were upgraded, just 10 percent of target. The program did not include in this Indicator three trainings provided to cocoa processors that may count as improved facilities. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 7 40 4 90% below target 99 137 Result Indicator #14: Number of food safety/standards-ready target commodities with improved marking and branding This custom performance indicator aims to measure the improved marketing of agricultural products (FFPr2.1.3, 2.2.1). The indicator counts the number of target commodities that achieved food safety and quality standards (aka, elevated as “food-safety ready”) with program assistance and improved branding and marking. Program staff is tasked to compare former packaging/branding/marking with improved ones. The cumulative target at the end of year two is five commodities to achieve food safety and quality standards, and that at the end of year four is 20 such commodities. Improved branding and marking of commodities evaluated as "food-safe ready" increases marketing potential. Only one commodity has the achieved food-safety ready standard by the end of year two, thus the program has reached just 20 percent of target. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 0 5 1 80% below target 10 20 The program Chief of Party believes that the method to get improved marking and branding for target commodities is in place but there is a need to apply it throughout; interviewed beneficiaries confirmed their interest in establishing country-wide or product-wide brands for certain premium vegetables. The program believes that the packaging improvements it facilitated for long beans, avocados, and pineapples should be included in this indicator (currently they are not). Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 17 Result Indicator #15 (FFPr9): Value of new public and private sector investment leveraged as a result of USDA assistance This performance indicator aims to measure increased access to markets to sell agricultural products (FFPr 2.2). Investment is defined as any use of public or private sector resources intended to increase future production, output, or income, to improve the sustainable use of agriculture-related natural resources (soil, water, etc.), to improve water or land management anywhere along the food chain, and natural resource management. “Leveraged as a result of USDA assistance” indicates that the new investment was directly encouraged or facilitated by activities funded by, or resources provided by, the USDA. New investment means investment made during the reporting year. The cumulative target at the end of year two is to have US$600,000 of new public and private sector investment leveraged, and that at end of year four is US$1.22 million in such investment. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL Investment 0 Zero US$ 600,000 US$278,236 54% below target US$ 1,087,500 US$ 1.222,500 Private investment US$ 219,986 Program grants US$ 58,150 The program requires beneficiaries to contribute to the investment with a 3:1 cost share, that is, the program provides a maximum of 25 percent of the investment cost. The 3:1 cost￾share requirement considers sustainability. The investment from the local beneficiary considers 5-year contribution projections (i.e., beyond the expected life of the program). Increased public and private sector investment is critical in sustainability in agricultural production and trade. The EQ program has leveraged US$278,236 of new private sector investment by midpoint implementation, thus remaining 54 percent short of the target. Program grants accounted for 21 percent of the total investment. Result Indicator #16: Value of buyer/seller contracts facilitated This custom performance indicator aims to measure improved linkages between buyers and sellers (FFPr2.2.2). The indicator counts the value in US$ of deals resulting from program facilitation activities. The number of deals will be automatically tracked to provide context for the total value. The contracts must be tracked by program staff. Beneficiaries are asked to share information when deals occur using engagement reporting forms. The cumulative target at the end of year two is to facilitate contracts for a total value of US$1 million, and that at end of year four is a total value of US$9 million of such contracts. A high value (and number) of contracts facilitated contributes to the start of improved linkages between buyers and sellers. The EQ program has facilitated contracts for a total value of US$374,584, thus remaining 63 percent short of the target. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 18 Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 0 US$ 1,000,000 US$374,584 63% below target US$ 5,000,000 US$ 9,000,000 Result Indicator #17: Average OCI score of target producer groups This performance indicator aims to measure improved capacity of key groups in the agriculture production sector (FFPr1.4.4). The EQ program is using the Organizational Capacity Index (OCI) methodology to strengthen the capacity of eight key producer organizations to deliver services to their associates. 8 Activities started in early 2017 with the visit of an international expert, and continued in the following months with the selection of the initial five of the eight targeted associations and the delivery of technical support to the first three in the first two years of program implementation9 . This activity has continued at beginning of year three of program implementation, expanding support to all five selected associations. Some of these activities were carried out during the field mission of this evaluation, when two additional organizations were selected while the program plans to engage with the remaining one in early 2018. The evaluator and the enumerator both assisted in some of these OCI related activities in November 2017. There is no mid-point target for Result Indicator #17 regarding the organizational capacity of target producer groups since it was anticipated that change in organizational capacity takes time to fully manifest. However, Annex E of the signed agreement between IESC and USDA wrongly states the annual targets for this Result Indicators at between 2 and 3 on a scale of 1 to 4. It is recommended to amend Annex E of the above agreement to reflect the actual OCI scoring system that uses a scale of 1 to 100. The end of year 4 target should be set at 85 rather than 3 as in Annex E, with no intermediate targets, as below10 . 8 The Organizational Capacity Index (OCI) is an objective measurement expressed in numbers, in a prearranged scale, of the levels of development of a producers' organization, from the point of view of the following: i) democracy in the management of the organization; ii) the provision of services to its members; iii) economic and financial performances; iv) human development of the members; and v) aspects of management, administration, and the running of the organization. As a diagnostic tool, the OCI facilitates the analysis of the performance of the organization, and organizes the provision of support for improving or adjusting organizational performance. It allows prioritizing the formulation of strategies and activities of organizational strengthening against available resources in a way that allows the organization to effectively consolidate its business management in the medium term. 9 The baseline study selected five organizations and applied the OCI methodology to all of them. Only three of them have been retained, while of the other two, one is member of CONACADO, a strong group of cooperatives operating in the cocoa VC that has been considered not in need of organizational strengthening support; the remaining one has almost ceased operations. 10 Noteworthy, the new target of 85 out of 100 (i.e. 85% of maximum score) is higher than the target in Annex E that is 3 out of 4 (i.e. 75 percent of the maximum score). Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 19 Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL No target Not applicable 85 The OCI-related organizational strengthening encompasses all three forms of producer organization currently existing in the DR: cooperatives, producer associations, and clusters. Cooperatives appear to be the strongest form of producer organization in DR, including activities such as the collective buying of inputs, processing and packaging of members’ crops, commercialization of members’ produce, and provision of credit (saving and loans); furthermore, cooperatives can be established as for-profit organizations. Associations are a weaker form of producer organizations, because they cannot provide credit to their members and can only be established as not-for-profit organizations. Clusters appear to be the weakest form of such organizations, looser than the previous two but still having juridical status (an anomaly as compared with clusters established in other Latin American and Caribbean countries). In general, the selected organizations for which the OCI methodology has been applied show relatively strong performance in member participation and human development, while being weak in issues related to commercialization of the produce. This organizational strengthening activity will be provided both to i) organizations that have already received technical assistance from donors in the past and are eager to request similar kinds of financial support (for instance, they expect the program to contribute to or directly finance the employment of a commercial director, something the program is not likely to provide), and ii) newly formed organizations with no previous experience of donor support. It would be extremely interesting to see the results of this organizational strengthening activity at the end of four years of implementation by comparing the results obtained for each of such different forms of organization with their dissimilar experiences as beneficiaries of foreign funded programs. Result Indicator #18: Percentage of target value chain groups that have a systematic way of disseminating and updating market information This custom performance indicator aims to measure increased access to improved market information (FFPr1.4.3 and 2.4.3). Target VC groups include producer groups and trade associations. “A systematic way” means that a process exists in written form or in practice. The indicator is calculated through direct observation and review of what direct beneficiaries submit. The denominator is seven trade associations and three producer groups. The cumulative target at the end of year two is to have 50 percent of target VC groups that have a systematic way of disseminating and updating market information, and the target for the end of year four is to have 100 percent. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 20 Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 Zero 50% Zero 100% below target 70% 90% The rationale behind this indicator is that if a systematic method exists, then access to improved market information increases. The EQ program has a zero percent score in this indicator. It appears that the number of existing trade associations was overestimated, with just two of them (rather than seven) operating in the DR. Consequently, the denominator for the calculation of this indicator should be reduced and the expected performance of this indicator might need to be significantly scaled down. The program has informed the evaluation team that the web design for the ministry of agriculture Food Safety Directorate (DIA) website is underway, though it is still unclear how to measure producer group and trade association use of it. The program has started directly disseminating the weekly Emergency Action Notifications (EAN) on pest border detentions - which give details about shipments not allowed to enter the USA because of pests/diseases found in the produce - provided by USDA APHIS. The EAN information is useful as it shows in what ports detentions are taking place, for what products, and for what pests/diseases. The program has included market information in weekly emails regarding prices for selected Dominican products in different terminal markets in the United States. The EQ program is looking for suitable means or an interested group/organization to continue disseminating this information. Given the progress so far, it is expected that the end-of-program target will not be achieved. It is therefore recommended to scale down the year four target by at least half, i.e., the new year four target would be set at 50 percent of target VC groups that have a systematic way to disseminate and update market information because of USDA assistance. In addition, the denominator used for the calculation of the index could be reduced from 10 to just five organizations (either trade or producer associations, cooperatives, or clusters). Result Indicator #19 (FFPr8): Number of public–private partnerships formed as a result of USDA assistance This performance indicator aims to measure increased leverage of private sector resources (FFPr1.4.5 and 2.4.5). The indicator counts the number of public–private partnerships (PPPs) in agriculture or nutrition formed during the reporting year due to USDA intervention. A PPP is considered formed when there is a clear agreement, usually written, to work together to achieve a common objective. There must be either a cash or in-kind significant contribution to the effort by both public and private entities. An agricultural activity is any activity related to the supply of agricultural inputs, production methods, agricultural processing, or transportation. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 21 The cumulative target at the end of year two is to have 19 PPPs formed, and that at the end of year four is to have 50 PPPs. More partnerships assume more investments in agriculture, which should increase agricultural productivity and trade. The EQ program has facilitated the formation of 15 PPPs at midpoint implementation, thus remaining 21 percent short of target. Most of these partnerships focused on agricultural production (80 percent of total), while the remaining share focused on postharvest transformation. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 0 19 15 21% below target 32 50 Agriculture production 0 0 15 12 20% below target 15 30 Agriculture postharvest transformation 0 0 4 3 25% below target 17 20 Result Indicator #20 (FFPr7): Number of private enterprises, producer organizations, water user associations, women's groups, trade and business associations, and community-based organizations that applied improved techniques and technologies as a result of USDA assistance This performance indicator aims to measure improved capacity of key organizations in the trade sector (FFPr2.4.4). The indicator tracks the number of private enterprises, producer organizations, etc. that applied improved techniques or technologies11 . The cumulative target at the end of year two is to have 57 enterprises/organizations that applied improved techniques and technologies and the target at the end of year four is to have 195 such enterprises/organizations. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 3 57 30 47% below target 113 195 Private enterprises 0 0 36 11 69% below target 45 77 Producer organizations 0 3 18 18 On target 63 108 Trade and business associations 0 0 3 1 67% below target 5 10 11 The entity is only counted once per reporting year, even if multiple technologies/practices are applied. Any groups applying a technology that was first applied in the previous year and continues to be applied in the reporting year should be counted under “continuing.” However, if they added a new technology or practice during the reporting year to those they continued to apply from the previous year(s), they would be counted as “new.” No entity should be counted under both “new” and “continuing” in one year. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 22 The indicator tracks private sector and civil society behavior change to increase agricultural sector productivity. Data for this indicator complements FFPr#2 indicator, i.e., Result Indicator #5. The EQ program has facilitated the application of improved techniques and technologies in 30 enterprises/organizations at midpoint implementation, thus remaining 47 percent short of target. The program performance is on target for producer organization (18 of them have applied improved techniques and technologies by the end of Y2), but below target for both private enterprises and trade/business organizations). Activity Indicator #1.1: Number of trainings delivered to increase capacity of fruit and vegetable producers, processors, and packing houses This performance indicator aims to measure the increased value added to post-production agricultural products (FFPr2.1). The indicator counts the number of trainings delivered under activity 1. Trainings include formal and informal settings. Training topics include fruit processing, Sanitary and Phytosanitary (SPS) measures, post￾harvest handling, etc. The cumulative target at the end of year two is to perform 11 trainings, and that at end of year four is to perform 17 trainings. At midpoint, the EQ program has already carried out 77 trainings, thus significantly exceeding the target by seven times over. Figure 2 provides an approximate breakdown of trainings beneficiaries by their respective activity value chain(s). Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 3 11 77 600% above target 15 17 Trainings included improved farming techniques, pest management, processing, packaging, good agricultural practices, good manufacturing practices, GLOBAL G.A.P. and fair-trade certifications, traceability, cold chain procedures and awareness, market access, border inspections, study tours abroad, organizational capacity, etc. Three of these trainings were carried out in collaboration with other USDA/USAID funded programs, mainly focusing on FSMA-related issues. Given the achievement so far, it is recommended to scale up the Life of Project target by seven times, i.e., the new end of year 4 cumulative target would be set at 120 trainings Figure 2: Breakdown of Trainings Provided Under Activity Indicator #1.1 Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 23 delivered by the program to increase capacity of fruit and vegetable producers, processors, and packinghouses. Activity Indicator #1.2: Number of producers benefitting from training and technical assistance This performance indicator aims to measure the increased value added to post-production agricultural products (FFPr2.1). The indicator counts the number of individual producers who benefit from activity 1 training and technical assistance The cumulative target at the end of year two is to train or provide technical assistance to 515 producers, and that at end of year four is to train/assist 3,100 farmers. At midpoint, the EQ program has trained and/or provided technical assistance to 1,100 farmers, thus largely exceeding the target more than two times over. Beneficiaries of this activity were mostly men (84 percent). Figure 3 provides an approximate breakdown of beneficiaries by their respective value chain(s). Trainings included improved farming techniques, pest management, good agricultural practices, certification in GLOBAL G.A.P. and fair trade, traceability, market access, border inspections, study tours abroad, organizational capacity, etc. Three of these trainings were carried out in collaboration with other USDA-funded programs, namely the Food Safety Agro Sustainability Training (FAST) and the so-called PAPA-CAFTA focusing on Sanitary and Phytosanitary Standards in the DR and Central America and related to FSMA issues. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 15 515 1.100 114% above target 1.820 3.010 Given the achievement so far, it is recommended to scale up the Life of Project target by at least 50 percent, i.e., the new end of year 4 cumulative target would be set at 4,650 producers benefitting from training and technical assistance provided by the program. Figure 3: Breakdown of Activity Indicator #1.2 Progress by VC Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 24 Activity Indicator #1.3: Number of processors benefitting from training and technical assistance This performance indicator aims to measure the increased value added to post-production agricultural products (FFPr2.1). The indicator counts the number of individual processors who benefit from post-harvest processing training and technical assistance. The cumulative target at the end of year two is to train or provide technical assistance to 47 processors, and that at the end of year four is to train/assist 74 processors. At midpoint, the EQ program has trained and/or provided technical assistance to 85 processors, thus exceeding the target almost two times over. Beneficiaries of this activity were mostly women (64 percent). Figure 4 provides an approximate breakdown of beneficiaries by their respective value chain(s). Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 32 47 85 81% above target 62 74 Trainings included improved processing and packaging techniques, good manufacturing practices, certification, GLOBAL G.A.P. and Fair Trade, traceability, cold chain procedures, market access, border inspections, study tours abroad, organizational capacity, etc. Two of these trainings were carried out in collaboration with other USDA/USAID funded programs; mainly focusing on FSMA-related issues. Given the achievement so far, it is recommended to double the Life of Project target, i.e., the new end of year 4 cumulative target would be set at 150 processors benefitting from training and technical assistance provided by the program. Figure 4: Breakdown of Activity Indicator #1.3 Progress by VC Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 25 Activity Indicator #1.4: Number of packing-house staff benefitting from training and technical assistance This performance indicator aims to measure the increased value added to post￾production agricultural products (FFPr2.1). The indicator counts the number of individual packing-house staff who benefit from post-harvest handling training and technical assistance. The cumulative target at the end of year two is to train or provide technical assistance to 600 packing-house staff, and that at the end of year four is to train/assist 1,800 packing-house staff. At midpoint, the EQ program has trained and/or provided technical assistance to 367 packing-house staff, thus remaining below the target by almost 40 percent. Beneficiaries of this activity were mostly men (76 percent). Figure 5 provides an approximate breakdown of beneficiaries by their respective value chain(s). Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 40 600 367 39% below target 1.200 1.800 Trainings included improved processing and packaging techniques, good manufacturing practices, GLOBAL G.A.P. and Fair-Trade certification, traceability, cold chain procedures, market access, border inspection, study tours abroad, organizational capacity, etc.12. Three of these trainings were carried out in collaboration with other USDA-funded programs (FAST, DR-CAFTA) and focused on FSMA-related issues, including the training of Preventive Controls Qualified Individuals (PCQI). Activity Indicator #1.5: Value of grants provided to producers This performance indicator aims to measure improved capacity of key groups in the agriculture production sector (FFPr1.4.4). The EQ program has a total budget of US$301,250 for grants provided to producers of the five VCs. The midpoint target is to provide grants of 12 Among these activities, the M&E staff also included training in farming techniques, which are not necessarily compulsory training for packing-house staff, though information on GAP may be useful. Figure 5: Breakdown of Activity Indicator #1.4 Progress by VC Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 26 US$41,250. The program has earmarked these grants to help producers get internationally recognized certification, including GLOBAL G.A.P., GHP, GMP, Fair Trade, Organic, and ISO certifications. The program publishes an Annual Program Statement (APS) for certification and then selects participants based on their application. Grants provided to producers in the first two years of program implementation totaled US$5,678.14, or 14 percent of target. All these grants were provided during the second year of program implementation. Following the APS applications, six packing houses from the avocado and pineapple VCs applied for assistance in obtaining internationally recognized certifications. The program signed two contracts for implementation of the GLOBAL G.A.P. quality standards during the April – September 2017 period. The two recipient companies have already selected the consultants who will advise and prepare them for the certification process. They are exporters of fruit and vegetables to the United States and Europe who procure produce from small and medium-sized producers and required the GLOBAL G.A.P. certification for groups of between 20 and 50 producers of avocados and pineapples. The program will help the beneficiaries cover the cost of these consultants. The other four applicants are still in the process of complying with APS requirements. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 US$ 13,750 US$ 41,250 US$ 5,678 86% below target US$ 201,250 US$ 301,250 During focus group discussions, some exporters requested that the EQ program provide financial help to producers getting GLOBAL G.A.P. certification for the building/refurbishing of facilities such as bathrooms, depots for inputs/outputs, etc. These projects can represent a significant cost for small-holder producers who might be unable to get the certification for lack of funds to build these basic facilities. It is recommended that the program provide grants for this type of intervention at farm level to facilitate acquisition of the certification. Activity Indicator #2.1: Number of processors/exporters trained in marketing and branding techniques This performance indicator aims to measure the improved marketing of agricultural products (FFPr2.1.3, 2.2.1). The indicator counts the number of individual processors and exporters who receive training to improve approaches to marketing and branding. Training topics could include fundamentals of marketing or specific topics including packaging, labeling, etc. The cumulative target at the end of year two is to have 240 processors/exporters trained in marketing and branding techniques, and that at the end of year four is 480 such processors/exporters. This is a direct measure of the activity on delivering training on marketing and branding techniques. The EQ program has trained 294 processors/exporters in marketing and branding Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 27 techniques by the end of year two, thus exceeding the target by 23 percent. Men represented 68 percent of this activity’s total beneficiaries. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 120 240 294 23% above target 360 480 Given the achievement so far, it is recommended to scale up the Life of Project target by at least one quarter, i.e., the new end of year 4 cumulative target would be set at 600 processors and /or exporters trained in marketing and branding techniques. Activity Indicator #2.2: Number of direct beneficiaries participating in trade shows, buyer visits to the DR, and exporter visits to the United States and Europe This performance indicator aims to measure improved linkages between buyers and sellers (FFPr2.2.2). The indicator counts the number of direct beneficiaries participating in trade shows, buyer visits to the DR, and exporter visits to the United States and Europe. A direct beneficiary is an individual and can be counted multiple times if benefitting from attending multiple program-sponsored events. The cumulative target at the end of year two is to have 16 individuals participating in trade shows, buyer visits to the DR, and exporter visits to the United States and Europe, and that at the end of year four is to have 100 such beneficiaries. This is a direct measure of the activity of facilitating buyer-seller relationships. The EQ program has facilitated participation in trade shows, buyer visits to the DR, and exporter visits to the United States and Europe for 98 beneficiaries, thus exceeding the target six times over and almost achieving the year four target two years ahead of schedule. Three￾quarters of these beneficiaries were men, while the breakdown by VC shows the dominance of Asian vegetables, 13 followed by pineapples, avocados, greenhouse vegetables, and cocoa. 13 These are defined as horticulture in the M&E database. Figure 6: Breakdown of Activity Indicator #2.2 Progress by VC Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 28 Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 0 16 98 513% above target 48 100 Given the achievement so far, it is recommended to double the Life of Project target, i.e., the new end of year 4 cumulative target would be set at 200 direct beneficiaries participating in trade shows, buyer visits to the DR, and exporter visits to the United States and Europe. Activity Indicator #2.3: Number of producer organization members benefitting from end-market assessments This performance indicator aims to measure mproved linkages between buyers and sellers (FFPr2.2.2). The indicator counts producer organization members who sell directly to the international and domestic markets, i.e., the number of individuals benefitting from end-market assessments. Program staff is to track this indicator using standard trackers and engagement reporting forms. The cumulative target at the end of year two is to have 40 members of producer organizations benefitting from the end￾market assessments, while that at the end of year four is to have 60 such beneficiaries. The EQ program has facilitated direct sales to international markets by 86 members of producer organizations, thus exceeding not only year two target by more than twice over, but already exceeding the year four target by 43 percent. Men are by far the largest share of beneficiaries (91 percent of the total). The program has facilitated a trip to Europe for avocado and pineapple exporters and disseminated the results. There are also upcoming trips to the United States and Canada for exporters of Asian vegetables. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 0 40 86 115% above target 60 60 Given the achievement so far, it is recommended to triple the Life of Project target, i.e., the new end of year 4 cumulative target would be set at 200 Given the achievement so far, it is Figure 7: Breakdown of Activity Indicator #2.3 Progress by VC Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 29 recommended to double the Life of Project target, i.e., the new end of year 4 cumulative target would be set at 180 direct beneficiaries participating in trade shows, buyer visits to the DR, and exporter visits to the United States and Europe. Activity Indicator #2.4: Number of trade association members benefitting from the end-market assessments This performance indicator aims to measure improved linkages between buyers and sellers (FFPr2.2.2). The indicator counts the number of trade association members who sell directly to the international and domestic markets, i.e., the number of individuals benefitting from end-market assessments. The cumulative target at the end of year two is to have 60 members of trade organizations benefitting from the end-market assessments, while that at the end of year four is to have 140 of these beneficiaries. The EQ program has facilitated direct sales to international markets of 11 members of trade organizations, thus scoring just 18 percent of target. Men are by far the largest share of these beneficiaries (82 percent of total). As for VC disaggregation, Asian vegetables accounted for almost two-thirds of the total, with avocados adding the remaining third. The problem here is that the number of trade associations operating in the DR was mistakenly thought to be seven, while only two are currently in operation14. It is recommended therefore to reduce the target by 3.5 times (i.e. applying a 7 to 3 ratio), thus reducing the year four target to 40 members, in line with the initial rationale for this indicator. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 0 60 11 82% below target 120 140 14 This should be compared with Activity Indicator #2.4, in which the number of active producer organization was probably underestimated and the number of producer organization members benefitting from the end-market assessments should more correctly be set at 140 as the Y4 target rather than 40. Figure 8: Breakdown of Activity Indicator #2.4 Progress by VC Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 30 Activity Indicator #2.5: Number of trade association individuals benefitting from trade association capacity building activities This performance indicator aims to measure improved capacity of key organizations in the trade sector (FFPr2.4.4). Trade associations include those involved in domestic and international trade. This indicator counts the number of individual trade association staff and members who benefit from trade association capacity building activities. The cumulative target at the end of year two is to have 196 trade association individuals benefitting from trade association capacity building activities and that at end of year four is to have 504 such beneficiaries. No individuals have benefited from trade association capacity building activity yet. One of the problems here is that the PMP foresaw the existence of seven trade associations active in the trade of targeted fruit and vegetables in the DR, when in fact only two exist. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 28 196 0 100% below target 364 504 The program believes that it is a mistake to maintain an artificial distinction between producer and trade associations, since in the DR many associations function as both producer and trade organizations. For instance, this distinction has delayed the starting of the OCI exercise with ADEXVO (a trade association that sells Asian vegetables). The program is also supporting COOPARRIBA (a cooperative that exports greenhouse vegetables) in the preparation of a business plan and the identification of a commercial director to expand their export business. Newly formed ASOPROMIPOPLA in Monte Plata wants to engage in direct marketing of membership production of pineapple with program assistance. In addition, much of the OCI training activities would contribute to this indicator. It is recommended to maintain the same target for year four but to include in the computation of this indicator not only trade associations but also producer associations, cooperatives, and clusters. Activity Indicator #2.6: Number of women in trade association management who participate in program-sponsored leadership training and events This performance indicator aims to measure improved capacity of key organizations in the trade sector (FFPr2.4.4). The indicator counts the number of mid-level to senior-level women who are active in the targeted trade association’s management or overall cluster. The cumulative target at the end of year two is to have 56 women involved in trade association management who participate in program-sponsored leadership training and events, and the target at the end of year four is to have 107 of these beneficiaries. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 31 This activity contributes to building trade association capacity, which increases access to markets for women. According to data provided by the M&E team, the EQ program has facilitated the participation in program-sponsored leadership training and events of 80 women involved in trade association management, thus exceeding the target by 43 percent. Beneficiaries, particularly women, demonstrated great appreciation for these activities during the quantitative survey. The evaluation team considers that the program’s M&E team has mistakenly computed as beneficiaries not only women in trade association management (the evaluation found just three, or 5 percent of the year two target), but all women who participated in the three seminars on women’s leadership organized by the program. Again, here the problem is the number of active trade organization operating in the DR. The end of year four cumulative target could be reduced to just 30 women by applying a ratio of 7 to 2 (would-be organizations vs. existing ones)), or the purpose enlarged to include other types of organization. In any case, it is recommended that the program M&E team recalculate the result of this indicator. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL (as calculated by Program) 0 30 56 80 43% above target 81 107 TOTAL (as calculated by Evaluator) 56 3 95% below target Activity Indicator #2.7: Value of grants provided to trade associations This performance indicator aims to measure improved capacity of key organizations in the trade sector (FFPr2.4.4), and counts the total value of grants provided to trade associations. The cumulative target at the end of year two is to provide grants totaling US$16,000 to trade associations, and that at end of year four is to provide US$34,000 of such grants. This indicator is a direct measure of the grant activity involving trade associations. At midpoint the EQ program has provided grants in the amount of US$816, thus remaining 95 percent short of target. This activity is somewhat associated with Result Indicator #17, i.e. the strengthening of the organizational capacity that has been initiated only in the Oct 16- Mar 17 period of program implementation. In addition, it faces the same problems as in Activity Indicator #2.5 above, i.e., just two trade organizations are dealing with target commodities, though many more producer associations/cooperatives export produce and could be also included as beneficiaries of this indicator. It is recommended that the of this indicator be enlarged to include additional types of organizations, such as producer associations, cooperatives, and clusters. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 32 Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 0 US$ 16,000 US$ 816 95% below target US$ 26,000 US$ 34,000 Activity Indicator #3.1: Number of individuals attending cold chain knowledge sharing events This indicator aims at measuring increased efficiency in post-production processes (FFPr2.1.2) and increased use of improved post-harvest infrastructure (FFPr2.1.2.2). The indicator counts the number of individuals attending educational visit to Panama, cold chain workshops in the Agroalimentaria Fair in Santo Domingo, WFLO-led in-country demonstrations of cold chain facilities, and the Global Cold Chain Alliance annual Latin America Conference. An individual can be counted more than once if attending different demonstrations of various aspects of a cold chain facility. The cumulative target at the end of year two is 129 individuals attending cold chain sharing events, and that at the end of year four is 358 individuals attending such events. This is a direct measure of activities aiming at strengthening the cold chain through private–private and public–private partnerships. At midpoint of the EQ program implementation, as many as 477 individuals attended cold chain sharing events, thus exceeding the year two target by almost four times, and the year four target by one-third. Men represented two thirds of the total beneficiaries of this activity. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 100 129 477 270% above target 233 358 Given the achievement so far, it is recommended to double the Life of Project target, i.e., the new end of year 4 cumulative target would be set at 750 individuals attending cold chain￾knowledge sharing events. Activity Indicator #4.1: Number of fruit and vegetable processors 15 benefitting from in-kind equipment to create value added products This performance indicator aims to measure the increased value added to post-production agricultural products (FFPr2.1). Fruit and vegetable processors include cocoa, pineapple, avocado and Asian and greenhouse vegetable producers who receive equipment and/or assistance from the program and who will have more opportunities to create value added 15 The program was amended in August 2017. One of the changes in the program description was to include vegetable processors to activity four. Therefore, the Evaluator recommends that the indicator definition should be changed to include vegetable processors as well. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 33 products and improve HACCP and HARPC conditions that would address recent rejections from the U.S. market. The cumulative target at the end of year two is four fruit processors benefitting from in-kind equipment to create value added products, and that at the end of year four is 20 such processors. At midpoint of the EQ program implementation, two processors have benefitted from in-kind equipment to create value added products, or one half of the target. More relevant information is provided below (Activity Indicator #4.2). Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 0 4 2 50% below target 15 20 Activity Indicator #4.2: Value of in-kind grants provided to processors This performance indicator aims to measure increased value added to post-production agricultural products (FFPr2.1). The EQ program has a total budget of US$350,000 for in-kind grants provided to processors of one or more of the five VCs. The midpoint target is to provide in-kind grants of US$140,000. The program has earmarked these grants to help processors purchase equipment to improve processing capacity. A maximum amount is established for each beneficiary, ranging from US$20,000 (for processors of cocoa and greenhouse and Asian vegetables) to US$30,000 (for avocado and pineapple processors). The program publishes a RFA and subsequently verifies the actual need for deploying of the equipment and the applicant’s financial situation to evaluate long-term sustainability of the procurement. The program provides a maximum of 25 percent of the equipment’s procurement cost (i.e., a cost share ratio of at least 3:1 by the grant recipient). Total procurement costs may also include installation and operational costs, but exclude border taxes and shipment/freight costs. The beneficiaries must advance most of the equipment’s capital cost, after which the program provides the balance for purchase of the machine. In-kind grants provided to processors in the first two years of program implementation totaled US$58,150, or 42 percent of the midpoint target. All these grants were provided during the second year of program implementation. Beneficiaries are two processors of pineapple who received, respectively, a pineapple peeler /corer from Italy and a pulper to make pineapple pulp and juice from California, in the United States. Three applications for cocoa processing equipment were also approved during the Apr 17 – Sep 17 period and those grants are expected to be finalized during the Oct 17 – Mar 18 period. Annex 3: Overview of Performance for Each Result Indicator and Activity Indicator 34 Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 0 US$ 140,000 US$ 58,150 58% below target US$ 315,000 US$ 350,000 Local processors of fruit and vegetables are usually small and lack the financial resources to modernize processing equipment. By providing up to a quarter of the procurement cost, the program speeds up purchasing of equipment that, without the in-kind grants, may be delayed by months or years, or perhaps never finalized. The program requires recipients to do the market search for equipment they need, then verifies that the applicant has financial capacity to sustain the 3:1 cost share ratio. This ensures buy-in and demand-driven ownership but has created delays in grant realization. The program is expected to employ a short-term consultant in early 2018 to speed up the process of evaluation and approval of the in-kind grants. Activity Indicator #5.1: Number of individuals reached by public information campaigns This performance indicator aims to measure increased access to improved market information (FFPr1.4.3 and 2.4.3). The indicator measures the estimated number of beneficiaries (farmers, producers, middlemen, etc.) who receive messaging from the campaign, which includes radio, newspaper, and TV audiences. Technical staff collects data from radio, newspaper, and media viewership and listenership reports and submits it to the M&E manager. The cumulative target at the end of year two is to have 500,000 individuals reached by public information campaign, and that at end of year four is to have 1.5 million such individuals (this assumes that about 12 percent of the DR's total population of 10.4m will be reached by the campaign). The indicator is therefore a direct measure of the public information campaign activity. The public information campaign has reached almost 300,000 individuals at midpoint of program implementation, thus remaining 40 percent below target. Disaggregation Baseline Value (Year) End of Y1 Target End of Y2 Target (Cumulative) Achievement End of Y2 (Cumulative) Percent met End of Y2 (Cumulative) End of Y3 Target (Cumulative) End of Y4 Target (Cumulative) TOTAL 0 0 500,000 298,629 40% below target 1,000,000 1,500,000 Annex 4: Export Statistics 1 Table A4.1: DR global exports of selected targeted commodities, thousands of US$, 2014–2016 Value Chain Exported value in 2014 Exported value in 2015 Exported value in 2016–Q1 to Q3 Avocados 22,530 16,585 18,664 Pineapples 3,690 4,979 4,123 Tomatoes 9,230 7,993 10,143 Peppers 27,887 10,044 11,449 Cucumbers 7,309 9,728 6,466 Eggplants 7,255 6,299 4,345 Okra 583 488 1,141 Green Beans/Long Beans 1,721 1,600 1,160 Cocoa Beans 212,116 250,786 202,037 Cocoa and Cocoa Preparations 227,076 264,185 212,319 Source: ITC Trade Map, www.trademap.org Table A4.2: DR total global exports of selected targeted commodities, thousands of US$, by quarters,1 2014–2016 Value Chain Exported value in 2014-Q4 Exported value in 2015-Q1 Exported value in 2015-Q2 Exported value in 2015-Q3 Exported value in 2015-Q4 Exported value in 2016-Q1 Exported value in 2016-Q2 Exported value in 2016-Q3 Avocados 8,328 9,987 942 1,011 4,645 10,825 3,233 4,606 Pineapples 1,029 1,515 1,282 794 1,388 1,540 1,522 1,061 Tomatoes 2,757 2,813 1,279 1,283 2,618 5,060 2,671 2,412 Peppers 6,850 7,381 1,402 394 867 3,705 4,504 3,240 Cucumbers 2,295 2,609 2,330 2,137 2,652 2,653 1,707 2106 Eggplants 1,970 1,796 1,271 1,161 2,071 1,517 1,154 1674 Okra 129 63 26 14 385 390 573 178 Green Beans/Long Beans 580 438 422 387 353 346 429 385 Cocoa Beans 27,877 47,503 103,849 76,757 22,677 21,827 100,816 79,394 Cocoa and Cocoa Preparations 31,689 50,535 106,433 80,897 26,320 24,345 103,781 84,193 Source: ITC Trade Map, www.trademap.org 1 Calendar year quarters. Annex 5: Evaluation Questionnaire 1 EXPORTING QUALITY AND SAFETY MIDTERM EVALUATION QUESTIONNAIRE Instructions: This is an anonymous questionnaire. The facilitator will provide you the anonymous identification number. The match between this number and your name is only known to the facilitator. Please provide information regarding your activity, kind of production, and gender. Please give your answers or comments in writing, or indicate the extent to which you evaluate the activities/performances of the Exporting Quality and Safety (EQS) Program on the topics on a scale of 1 (least) to 5 (highest). If you believe a question does not apply to you, please leave it blank. Code number _________ Male Female Date _____________ Producer Trader Processor Exporter Extension Officer _________ Avocado Pineapple Cocoa Greenhouse Oriental Vegetables ____________ SECTION 1 Strongly Disagree Disagree Neutral Agree Strongly Agree 1.A.1 The quality of my business has improved thanks to program activities 1 2 3 4 5 1.A.2 The value of my business has increased thanks to program activities 1 2 3 4 5 1.A.3 My technical skills have improved thanks to program activities 1 2 3 4 5 1.A.4 My commercial skills have improved thanks to program activities 1 2 3 4 5 1.A.5 Please quantify the improvements due to program activities 0-20% 21-40% 41-60% 61-80% 81-100% 1.B.1 The program has helped me to achieve more than I expected 1 2 3 4 5 1.B.2 Please provide more details 1.C.1 My business has faced major challenges during the last 24 months 1 2 3 4 5 1.C.2 Please name some of these challenges, if any 1.C.3 The medfly ban has significantly impacted my activity 1 2 3 4 5 1.C.4 The heavy rains in the fall of 2016 negatively affected my production/trade/exports 1 2 3 4 5 Annex 5: Evaluation Questionnaire 2 1.C.5 The 2017 hurricanes (heavy rains and strong winds) negatively affected my production/trade/exports 1 2 3 4 5 1.C.6 The program has been able to help me to counter these challenges 1 2 3 4 5 1.C.7 How can the program further support you in your business challenges? 1.D1 How can the program help you in unexpected situations? SECTION 2 Strongly Disagree Disagree Neutral Agree Strongly Agree 2.A.1 The program has been able to address the core issues of my activity 1 2 3 4 5 2.A.2 The program has helped strengthen my organization 1 2 3 4 5 2.A.3 How can the program improve the relevance of its assistance to better support you? SECTION 3 Strongly Disagree Disagree Neutral Agree Strongly Agree 3.A.1 The program activity on export promotion has been very effective in improving my business 1 2 3 4 5 3.A2 The program activity on cold-chain awareness has been very effective in improving my business 1 2 3 4 5 3.A.3 The program activity on branding has been very effective in improving my business 1 2 3 4 5 3.A.4 The program activity on safety measures has been very effective in improving my business 1 2 3 4 5 3.A.5 The program activity on phytosanitary measures has been very effective in improving my business 1 2 3 4 5 3.A6 The program activity on traceability has been very effective in improving my business 1 2 3 4 5 3.A.7 The program activity on compliance measures has been very effective in improving my business 1 2 3 4 5 3.A.8 The program activity on certification has been very effective in improving my business 1 2 3 4 5 3.A.9 The program activity on GAP/GHM/GMP has been very effective in improving my business 1 2 3 4 5 3.A.10 The program activity on improving/expanding farming productivity has been very effective in improving my business 1 2 3 4 5 Annex 5: Evaluation Questionnaire 3 3.A.11 The program activity on improving/expanding processing capacity has been very effective inr improving my business 1 2 3 4 5 3.A.12 The program activity on improving management capacity of my farm has been very effective in improving my business 1 2 3 4 5 3.A.13 How can the program improve the effectiveness of its assistance? 3.B.1 The program has encouraged participation of women in its activities 1 2 3 4 5 3.B.2 The program has been very effective at reaching women beneficiaries. 1 2 3 4 5 3.B.3 What could be done to improve women’s participation in target value chains? 3.C.1 The training approach has been very good at enhancing my productivity 1 2 3 4 5 3.C.2 The training approach has been very good at enhancing my processing capacity 1 2 3 4 5 3.C.3 The training approach has been very good at enhancing my managerial skills 1 2 3 4 5 3.C.4 How can we improve the training approach to better support you? SECTION 4 Strongly Disagree Disagree Neutral Agree Strongly Agree 4.1 At the end of the program I will be able to continue to apply what I learned with no problem. 1 2 3 4 5 4.2 I would like this or a similar program of technical assistance to continue after its scheduled end 1 2 3 4 5 4.3 How can the program ensure that this type of assistance is available after it ends? 1 2 3 4 5 4.4 I would prefer that a similar program of assistance is continued by the Dominican Government/Public Organizations 1 2 3 4 5 4.5 How can the program ensure that this type of assistance is available after this program ends? CHIEF OF PARTY * ORGANIZATIONAL CHART EXPORTING QUALITY PROGRAM EXECUTIVE ASSISTANT FINANCE & ADMIN MANAGER * PROCUREMENT / GRANTS SPECIALIST HR/OFFICE MANAGER PINEAPPLE & AVOCADO VC FACILITATOR CACAO VC FACILITATOR & GENDER INTEGRATION GREENHOUSE & VEGETABLES VC FACILITATOR TRACEABILITY SPECIALIST (CEDAF) OCI SPECIALIST (CEDAF) ACCOUNTANT RECEPCIONIST AND HR ASSISTANT DRIVERS 1 DRIVERS 2 TRAINING ANALYST 1 M & E MANAGER * DEPUTY CHIEF OF PARTY * COMMUNICATIONS ASSISTANT M & E ANALYST COMMUNICATIONS SPECIALIST MARKET LINKAGES EXPORT TEAM FACILITATOR TRAINING ANALYST 2 CARLOS PUELLO LANDO REYES NILSY DELGADO PEGGY AVIOTTI EMMANUEL GUZMÁN SANTIAGO RIVAS JULISSA ALMÁNZAR VÍCTOR DE OLEO ANA TAVÁREZ TEÓFILO SURIEL BRIAN RUDERT MARÍA DEL VILLAR MARGARITA DE LOS SANTOS JEANNATY SOLÍS VLADIMIR RAMÍREZ RAQUEL TRAVIESO DANIEL RIVAS MICHELLE DE LOS SANTOS GERMAN JIMÉNEZ CARLODYS ALMEYDA PAULETTE BONSEÑOR PATRICIA NÚÑEZ IESC Home Office Associate Vice President, Chad Ford Senior Program Manager, Gabriela Kliewer* Senior Program Associate, David Stein * Denotes Key Personnel